WorksComputer Science
What Makes a Good Move? Learning to Act from Experience in Visual Worlds
Abstract
Learning visual policies requires turning interaction experience into guidance for individual decisions. Trajectory outcomes obscure action quality, while unguided exploration can revisit familiar situations. We present LightAct, an experience-learning framework that connects skill-guided exploration with policy internalization. Interaction segments are summarized into experience entries and consolidated into a game-specific skill that guides root collection toward broader state coverage. At each root, candidate actions are compared through short rollouts under a shared frozen policy. Environment-reported score gains are averaged across continuations and normalized into relative action rewards, producing reusable supervision for reinforcement learning. Policy training uses a compact dataset of just 1,024 decision roots collected in a single source game and paired with their action rewards. Across 170 GameWorld tasks spanning 34 games, LightAct improves overall progress by 23.6, 29.2, and 13.0 percentage points at the 2B, 4B, and 9B scales, respectively, with our strongest model competitive with reported proprietary-model results. Ablations show broader root-state coverage with skill guidance and consistent gains across three policy optimizers. Case studies show how rollouts distinguish actions with identical immediate gains.
Figure 1. Learning from source experience with LIGHTACT.
Introduction
Vision-language models (VLMs) increasingly serve as agents that connect visual perception and language reasoning to action in interactive environments [14, 25, 31, 32]. Video games expose the demands of this process: an agent must recognize the current situation, choose an executable move, and adapt as its actions change the environment [9, 16, 17]. Interaction also provides experience from which to improve. Prior work retains such experience as contextual guidance [21, 29] or uses it to update policy parameters through reinforcement learning [1, 8].
The central challenge is to turn this experience into guidance about individual decisions. A trajectory records the actions taken and their realized outcomes, leaving alternatives at the same decision point unobserved [24]. Its final outcome combines many choices: a useful move can be followed by a mistake, while a poor move can be rescued later. Environment scores summarize these outcomes, but do not directly reveal which action would have been preferable at a particular state [18]. The states encountered during exploration introduce a second limitation. Repeatedly visiting familiar situations concentrates experience on a narrow range of decisions [5, 15].
These observations suggest organizing experience around both the states reached and the choices available there. Summaries of prior interaction can guide exploration toward a broader set of decision states. At a selected state, executing alternative actions under a shared continuation policy makes their subsequent outcomes comparable. Averaging score gains over short rollouts then yields an estimate of each action’s value, which can be converted into supervision for policy learning.
We present LIGHTACT, an experience-learning framework that combines skill-guided exploration with policy internalization. We partition preliminary trajectories collected by a frozen pretrained VLM into short interaction segments. The model summarizes each segment into an experience entry describing the observed conditions, behavior, and outcome, then consolidates these entries into a game-specific textual skill. The skill guides subsequent exploration by providing reusable advice on effective behavior and recurring failures. We retain distinct visited states and their visual decision contexts as a bank of decision roots.
At each root, we compare the available actions through repeated short continuations under the same frozen policy. The resulting environment-reported score gains are averaged and normalized within the root to produce relative action rewards. Pairing these rewards with the collected decision contexts forms a fixed training set. Reinforcement learning then updates the policy by sampling responses to these contexts and assigning rewards to their parsed actions.
We instantiate LIGHTACT at the 2B, 4B, and 9B scales using Qwen3.5 [19] and evaluate on all 170 GameWorld tasks across 34 games and five categories [16]. With experience collected in 2048, overall progress improves by 23.6, 29.2, and 13.0 percentage points at the three scales, respectively. LIGHTACT-2B reaches 36.9% progress and 13.9% success, exceeding the pretrained 9B baseline on both metrics, while LIGHTACT-9B achieves 41.8% progress, competitive with reported proprietary-model results. Further experiments show broader root-state coverage with skill guidance, gains across three RL optimizers, and benefits from source-game experience across game categories. Qualitative examples illustrate how rollout comparisons reveal action preferences that immediate gains alone cannot distinguish.
Our contributions are threefold:
We introduce an experience-learning framework that connects skill-guided source exploration with reinforcement learning from action-level feedback.
We construct reusable action-reward tables by comparing continuations from shared decision roots under a frozen policy and normalizing their environment-reported score gains.
We demonstrate improvements across three model scales on GameWorld and examine how skill guidance, source environments, and policy optimizers contribute to learning from interaction.

Figure 2. A 2048 decision by LIGHTACT-2B. When repeated downward moves no longer change the board, the policy switches to moving right and gains 24 points.
Related Work
Vision-Language Game Agents
Video games test whether visual understanding supports sequential decisions [36]. Cradle [25] combines perception, planning, and memory to operate through screenshots and keyboard or mouse controls. BALROG [17] evaluates language and vision-language agents across games and reveals weaknesses in visually grounded interaction. VideoGameBench [34] examines play from raw visual input under real-time and paused interaction, separating decision quality from the effects of inference latency. GameWorld [16] provides browser games with standardized interfaces and state-verifiable task evaluation. Complementing these benchmarks, lmgame-Bench [8] studies perception and memory scaffolds and reports that reinforcement learning on individual games can transfer beyond the training environment. Our work examines how source-game interaction supports such policy learning.
Experience Learning of VLMs
VLM agents learn from accumulated interaction through contextual guidance and parameter updates. External memory retains experience for retrieval at decision time [13], while parameter updates encode it in the policy [6]. JARVIS-1 [29] retrieves multimodal Minecraft experience to support planning, while ELITE [30] extracts reusable strategies from trajectories and retrieves them according to task intent. ICAL [21] refines suboptimal demonstrations through execution and human feedback, annotating causal relations, state changes, and subgoals; the resulting examples support both retrieval and supervised fine-tuning. For reinforcement learning, DigiRL [1] adapts a pretrained VLM for device control through offline initialization and online interaction, using a VLM evaluator for feedback. VAGEN [27] improves explicit state estimation and transition prediction in multi-turn visual interaction through reinforcement learning. We connect experience-guided source exploration with policy internalization.
Rollout-Based Policy Learning
Rollouts estimate the value of an action from the outcomes that follow it. Monte Carlo policy improvement compares candidate moves by continuing each with a common base policy [26]. In combinatorial optimization and stochastic scheduling, heuristic continuations estimate the future cost of each choice [2, 3]. Sparse sampling [10] estimates action values over sampled lookahead trees, while UCT [11] uses a bandit rule to allocate simulations among branches. Rollout comparisons also provide supervision for policy learning [35]. Classification-based policy iteration [12] uses the preferred action at a sampled state as a training label. AggreVaTe [20] trains on expert cost-to-go estimates, while LOLS [4] evaluates decisions under both reference and learned policies. For language models, LATS [37] guides tree search with model evaluations and environment feedback, and Math-Shepherd [28] uses sampled completions to assess intermediate reasoning steps. We use score gains from a shared game state to supervise action selection.
Method
Experience gathered during interaction can guide subsequent decisions through both contextual guidance and policy learning. Building on this view, LIGHTACT (Figure 3) distills source-game trajectories into textual skills for exploration and compares action outcomes at the collected roots to construct action-level rewards for policy internalization.
Figure 3. Overview of LIGHTACT. Interaction segments are summarized into experience entries, then consolidated into a source-game skill that guides root-state collection. At each root, short rollouts compare candidate actions through their subsequent score gains. These gains are normalized into action-level rewards for reinforcement learning, internalizing the collected experience in a policy evaluated across games.
Decision-Making in Game Environment
In GameWorld [16], an agent pursues a game-specific objective through sequential interaction. At each decision, it receives the game rules, task instruction, role information, and available semantic actions, together with the current screenshot and recent observation–action history. The agent generates a response, parses it into an executable action, observes its effect on the game, and continues toward the objective. This interaction links visual understanding to decisions whose consequences unfold over subsequent steps.
For a game environment with a fixed, finite action space , let be its state, an action, and the visual observation. Interaction follows
where denotes the task specification together with the game instructions, is the current screenshot, and contains the recent observations and actions. The instructions describe the objective, rules, acting role, and semantic action interface. A task-specific evaluator determines whether the current environment state satisfies the task’s completion criteria, with indicating success. Let denote the policy parameterized by . Its pretrained parameters define the frozen initial policy . The policy generates a response from this context. The parser attempts to extract an executable action from : successful parsing yields , while invalid responses yield no executable action. For games with multiple acting roles, the other roles’ behavior is part of the transition process.
Exploration and Root-State Collection
The states reached during source interaction determine the decisions available for policy learning. We collect trajectories with a collection policy , sampling responses and executing successfully parsed actions. The basic collection policy is .
Among the nonterminal states visited during collection, we retain the first occurrence under an environment-specific equivalence criterion. For temporally ordered candidates indexed by , the selected indices are
Each selected state is stored with its decision context . The selected states form the root bank:
Their original contexts and the action rewards constructed below form the training examples for policy internalization.
Skill-Guided Exploration
To guide exploration toward a richer set of decision states, we first collect preliminary trajectories with and partition them into short interaction segments . Each segment contains consecutive visual observations, executed actions, and their resulting changes. Under an entry-summarization prompt, the frozen model distills each segment into an experience entry:
Each entry captures the observed conditions, behavior, and outcome within its segment.
The same model then consolidates the entries across segments into one textual skill for environment :
Through a skill-consolidation prompt, recurring observations are combined into concise guidance on useful behavior and failure patterns, tied to the conditions under which they arise. This guidance helps the agent reuse effective interactions and adjust its behavior when exploration stalls. We include this skill in the collection policy’s decision context:
The policy interprets the guidance using the current observation and recent history. The skill and model parameters remain fixed throughout guided collection. We construct the root bank from the guided trajectories using the root-selection procedure above.
Rollout-Based Action Value Estimation
For each decision root, we restore the same environment state and decision context before evaluating each candidate action. Starting from , we execute action and let select the subsequent actions. The resulting branch contains at most decisions, including the initial action, and ends at the first episode termination if it occurs earlier:
At each subsequent decision, conditions on the task, current observation, and recent interaction history within the branch. Using the same frozen policy across actions and roots provides a shared basis for comparing their outcomes.
Let denote the game score returned by the environment at state and the final state of branch . A continuation from contributes the score gain
For a terminal branch, is the state before reset. The finite-horizon action value and its estimate from continuations are
The estimate measures score gain after choosing and continuing under for the specified horizon.
Reward Shaping
The estimated score gains quantify the outcomes of candidate actions in the game’s score units. To express action preferences on a common reward scale, we normalize these values within each root:
where and . This normalization preserves the estimated ordering of actions and expresses their relative quality within a shared decision context. When all action values coincide, the root supplies no preference signal and all action rewards are zero.
RL-Based Policy Internalization
After collection, we pair each decision context in with its constructed reward function :
This converts the action comparisons in source experience into policy-learning examples: supplies the task and visual history, while provides feedback on the selected action. Starting from , we train to select valuable actions directly from .
The policy generates a response from the decision context , which is parsed into an action when valid. A response is valid when it satisfies the public action syntax and argument constraints and selects an action in . The response reward is
Sampling examples uniformly from , we optimize the expected response reward with reference-policy regularization:
where controls deviation from the initial policy . The current policy samples responses conditioned on and receives ; the collected contexts and reward maps remain fixed during these updates. Established policy-gradient algorithms optimize this objective, with their configurations specified in the experiments.
After learning, the policy generates and executes the parsed action using the task description, visual observations, and action interface of the environment being played.
Experiments
Benchmark and Evaluation Setting
We evaluate LightAct on GameWorld [16], which comprises 34 games and 170 tasks across five categories: Arcade, Platformer, Puzzle, Runner, and Simulation. Each run has a budget of 100 decisions, with the environment paused during inference. At each decision, the policy receives the game instructions, the current screenshot, and two rounds of image–text interaction history, and selects an action from the game’s interface. Action duration, episode termination, and resets follow the protocol. Published GameWorld results use greedy decoding with one run per task. Our Qwen3.5 and LightAct evaluations use temperature 1.0, top- 0.95, a 2,048-token output cap, and three runs per task. We use GameWorld’s state-based evaluator to measure success rate (SR), the fraction of successful runs, and progress (PG), normalized completion toward task-specific targets.
Implementation Details
We instantiate LightAct with Qwen3.5-2B, Qwen3.5-4B, and Qwen3.5-9B [19]. At each scale, experience collection, branch continuation, and skill extraction use frozen copies of the same model that initializes the trainable policy. Acting uses the native non-thinking mode. For each source game–task pair, we collect trajectories within a budget of 10,000 decisions and retain up to 1,024 deduplicated roots, with seed 42 for the initial reset. We equip source games with snapshot support through exportSnapshot() and importSnapshot(), restoring each decision root before evaluating candidate actions. Collection and branch continuation use temperature 1.0, top- 0.95, and a 2,048-token output cap. Branch evaluation uses including the initial action and continuations, processing eight roots concurrently. Rewards use environment-reported score gains with per-root min–max normalization. We train with GRPO [22], REINFORCE++ [7], or DAPO [33] in verl/HybridFlow [23]. Tables S1 and S2 in the appendix give the collection and reward settings and the training hyperparameters, respectively.
Baselines and Main Comparisons
LIGHTACT achieves competitive progress against proprietary models. LIGHTACT brings compact policies close to strong proprietary models in benchmark progress (Table 1). At 4B, it reaches 40.5% overall PG, comparable to GPT-5.2 at 40.6%; at 9B, it reaches 41.8%, approaching Gemini-3-Flash-Preview at 41.9%. On Puzzle tasks, LIGHTACT-9B achieves 59.2% PG, exceeding the reported progress of both models. The remaining gap is clearer in task completion, where overall SR reaches 18.4% for LIGHTACT-9B compared with 21.2% for Gemini. By comparing actions from a shared state, LIGHTACT turns interaction outcomes into supervision for policy learning. These results suggest that internalizing such experience enables compact VLMs to achieve competitive game-playing performance through direct policy inference.
| Model | Arcade SR | Arcade PG | Platformer SR | Platformer PG | Puzzle SR | Puzzle PG | Runner SR | Runner PG | Simulation SR | Simulation PG | Overall SR | Overall PG |
| Human | ||||||||||||
| Novice Player | 45.7 | 55.5 | 60.0 | 65.6 | 51.4 | 63.1 | 60.0 | 72.0 | 60.0 | 62.0 | 55.3 | 64.1 |
| Expert Player | 65.7 | 73.9 | 85.0 | 88.0 | 68.6 | 77.1 | 82.5 | 87.8 | 85.0 | 86.0 | 77.1 | 82.6 |
| Generalist Multimodal Agents | ||||||||||||
| Claude-Sonnet-4.6 | 5.7 | 28.3 | 22.5 | 37.0 | 25.7 | 51.5 | 30.0 | 51.9 | 15.0 | 16.6 | 20.6 | 39.3 |
| Gemini-3-Flash-Preview | 5.7 | 26.3 | 25.0 | 41.2 | 25.7 | 54.8 | 32.5 | 55.4 | 10.0 | 21.1 | 21.2 | 41.9 |
| GLM-4.6V | 8.6 | 22.8 | 20.0 | 33.9 | 5.7 | 29.1 | 27.5 | 49.1 | 0.0 | 5.3 | 14.1 | 30.8 |
| GPT-5.2 | 8.6 | 29.3 | 22.5 | 36.7 | 28.6 | 56.2 | 27.5 | 52.6 | 10.0 | 16.9 | 20.6 | 40.6 |
| Grok-4.1-Fast-Reasoning | 8.6 | 23.7 | 22.5 | 37.3 | 14.3 | 46.6 | 25.0 | 49.0 | 5.0 | 10.4 | 16.5 | 36.0 |
| Kimi-K2.5 | 8.6 | 26.4 | 20.0 | 35.3 | 25.7 | 51.4 | 27.5 | 49.7 | 5.0 | 11.7 | 18.8 | 37.4 |
| Qwen3-VL-Plus | 8.6 | 25.6 | 22.5 | 37.8 | 14.3 | 39.1 | 27.5 | 51.1 | 0.0 | 10.0 | 16.5 | 35.4 |
| Seed-1.8 | 11.4 | 33.5 | 22.5 | 34.6 | 22.9 | 48.7 | 27.5 | 51.2 | 10.0 | 18.8 | 20.0 | 39.0 |
| Qwen3-VL-235B-A22B | 5.7 | 23.2 | 17.5 | 29.5 | 8.6 | 33.3 | 27.5 | 50.4 | 0.0 | 3.6 | 13.5 | 30.8 |
| Qwen3-VL-30B-A3B | 2.9 | 20.3 | 20.0 | 36.5 | 2.9 | 26.1 | 27.5 | 51.1 | 0.0 | 3.5 | 12.4 | 30.6 |
| Qwen3.5-2B | 0.0 | 5.6 | 5.8 | 10.6 | 2.9 | 9.4 | 7.5 | 28.4 | 0.0 | 9.1 | 3.7 | 13.3 |
| Qwen3.5-4B | 4.8 | 9.4 | 2.5 | 6.3 | 4.8 | 21.6 | 1.7 | 8.2 | 5.0 | 12.7 | 3.5 | 11.3 |
| Qwen3.5-9B | 10.5 | 28.8 | 5.0 | 17.6 | 12.4 | 35.3 | 17.5 | 44.1 | 0.0 | 9.5 | 10.0 | 28.8 |
| Ours | ||||||||||||
| LIGHTACT-2B | 7.6 | 28.7 | 17.5 | 30.3 | 18.1 | 49.5 | 15.8 | 46.1 | 6.7 | 24.0 | 13.9 | 36.9 |
| LIGHTACT-4B | 17.1 | 36.4 | 15.0 | 31.0 | 23.8 | 54.5 | 20.8 | 49.7 | 6.7 | 23.7 | 17.6 | 40.5 |
| LIGHTACT-9B | 14.3 | 34.3 | 21.7 | 36.6 | 26.7 | 59.2 | 19.2 | 48.7 | 3.3 | 20.9 | 18.4 | 41.8 |
Table 1. Main results on GameWorld. SR and PG (%) are reported by category and overall. All three LIGHTACT models are trained with GRPO using experience collected in 2048. Qwen3.5 and LIGHTACT results are averaged over three evaluation runs per task. Published baseline results are taken from GameWorld [16].
LIGHTACT consistently improves policy performance across model scales. LIGHTACT increases both SR and PG in all five game categories at 2B, 4B, and 9B, indicating benefits for task completion as well as partial progress. The overall PG gains are 23.6, 29.2, and 13.0 percentage points, respectively. The 4B policy gains most on Runner tasks, where PG rises from 8.2% to 49.7%, even though its training experience comes from a puzzle game. Notably, LIGHTACT-2B surpasses the pretrained 9B baseline in both PG (36.9% versus 28.8%) and SR (13.9% versus 10.0%), showing that adaptation through experience can improve the effectiveness of a smaller policy. The 9B model also gains 8.4 percentage points in SR over its initialization, demonstrating that stronger pretrained performance still leaves room for im-
Ablations and Diagnostics
Policy internalization yields consistent gains across RL algorithms. We compare GRPO, REINFORCE++, and DAPO, using the same reward tables for each model scale (Table 2). All nine trained configurations improve overall PG over their base policies, with gains ranging from 10.8 to 29.2 percentage points. This pattern suggests that the action-level rewards provide useful supervision across different policy-update rules. The optimizer still affects how these gains are distributed across games: at 4B, DAPO achieves higher Puzzle and Simulation PG than GRPO, while GRPO attains higher overall PG. Thus, the common benefit of learning from action outcomes persists alongside algorithm-dependent differences in performance across categories.
| Model | Arc. PG | Plat. PG | Puzz. PG | Run. PG | Sim. PG | Overall PG |
| Qwen3.5-2B | ||||||
| Base | 5.6 | 10.6 | 9.4 | 28.4 | 9.1 | 13.3 |
| +REINFORCE++ | 24.0 | 28.6 | 46.2 | 46.8 | 20.1 | 34.5 |
| +GRPO | 28.7 | 30.3 | 49.5 | 46.1 | 24.0 | 36.9 |
| +DAPO | 26.6 | 29.9 | 50.2 | 45.0 | 20.6 | 35.9 |
| Qwen3.5-4B | ||||||
| Base | 9.4 | 6.3 | 21.6 | 8.2 | 12.7 | 11.3 |
| +REINFORCE++ | 29.7 | 30.9 | 51.9 | 48.9 | 19.5 | 37.9 |
| +GRPO | 36.4 | 31.0 | 54.5 | 49.7 | 23.7 | 40.5 |
| +DAPO | 33.8 | 30.0 | 56.3 | 49.1 | 26.3 | 40.2 |
| Qwen3.5-9B | ||||||
| Base | 28.8 | 17.6 | 35.3 | 44.1 | 9.5 | 28.8 |
| +REINFORCE++ | 36.3 | 35.5 | 53.1 | 49.0 | 11.2 | 39.6 |
| +GRPO | 34.3 | 36.6 | 59.2 | 48.7 | 20.9 | 41.8 |
| +DAPO | 33.1 | 32.4 | 59.7 | 49.7 | 19.4 | 40.7 |
Table 2. Policy internalization. GameWorld PG (%) for base and trained policies across three model scales and RL algorithms.
Skill guidance broadens the score distribution of collected roots. Figure 5 compares 1,024 roots collected with and without skill guidance in 2048; Figure 7 shows the skill used during collection. By drawing on interaction experience, the skill guides the policy away from repeated ineffective moves and helps it explore a broader range of game states. Guided collection extends the score range from 0–552 to 0–944 and raises the mean score from 186.6 to 280.2. The proportion of roots scoring at least 500 increases from 1.3% to 19.4%, while lower-score states remain represented. With GRPO, skill guidance raises PG from 32.98% to 36.90% and SR from 12.35% to 13.92% (lower panels of Figure 5). With REINFORCE++, PG changes little (34.60% to 34.55%), while SR rises from 11.96% to 12.75%. These results suggest that skill guidance broadens the experience available for policy learning, with benefits that depend on the optimizer.
Figure 5. Skill-guided exploration. Top: score distributions of 1,024 roots in 2048 (50-point bins). Bottom: GameWorld PG and SR for 2B policies trained with GRPO or REINFORCE++ (RF++).
Figure 7. 2048 exploration skill. The textual guidance used during root-state collection, with action-selection principles highlighted.
Source-game experience supports generalization across game categories. With Tetris as the source game, both GRPO and REINFORCE++ improve PG over the 2B base policy across all five categories (Figure 4). Arcade PG, for example, rises from 5.6% to 29.8% and 29.6%, respectively. These gains extend beyond the source game’s Puzzle category, suggesting that policy internalization helps LIGHTACT apply experience to different game mechanics. Experience from a single game can provide supervision for broader visual decisions. The varying gains across categories indicate that transfer is uneven, leaving room to improve the breadth of behaviors learned from source experience.
Figure 4. Source-environment ablation. GameWorld SR and PG for 2B policies trained on 2048 or Tetris with GRPO and REINFORCE++.
Rollout outcomes distinguish actions with identical immediate gains. In Figure 6, moving left or right from the same root yields an immediate score gain of 24, but the resulting boards support different continuations. The rightward branch produces a 64 tile, whereas the leftward branch retains a maximum tile of 32. Across four rollouts, moving right achieves a mean gain of 144 versus 67 for moving left, yielding normalized rewards of 1.000 and 0.330. The rollout comparison turns delayed action outcomes into a training preference that immediate score gains alone cannot provide.

Figure 6. From a root state to action rewards in 2048. Left and right gain 24 immediately but yield different outcomes. The table averages four ten-decision continuations per action and normalizes gains into rewards; endpoints are nearest each action’s mean gain.
Conclusion
We presented LIGHTACT, a framework that turns interaction experience into guidance for exploration and supervision for policy learning. Game-specific skills broaden root-state coverage, while short rollouts from shared roots reveal the relative utility of candidate actions. Experiments on Game-World show consistent improvements across model scales and policy optimizers, with compact policies achieving competitive progress against proprietary models. These findings demonstrate the value of organizing experience around states and action outcomes for learning game-playing policies.
Limitations
Generalization across source environments remains limited, and the current framework does not directly support action spaces with continuous parameters. Our study focuses on short-horizon decisions; long-horizon decision-making remains to be validated. Future work will investigate integration with multi-turn reinforcement learning and evaluation on games that require sustained planning. Reward shaping currently relies on scores read from environment states. We will further explore purely visual self-improvement that derives learning signals from observations without access to internal environment states.
LLM Usage Statement
OpenAI Codex was used to assist with code generation and completion, and OpenAI GPT models assisted with related-literature discovery. Codex also assisted with drafting and editing manuscript text. The authors led the research process, problem formulation, computational experiments, and factual verification, and take full responsibility for the scientific claims, results, references, and final manuscript.
Appendix

Figure S2. A Mario decision by LIGHTACT-2B trained with REINFORCE++ on 2048 (task 17_02, repetition 2, step 40). The recorded jump action is followed by mushroom collection and task completion. Shared system instructions appear in Figure S1.

Figure S1. A 2048 decision by LIGHTACT-2B trained with REINFORCE++ (task 01_02, repetition 3, step 46). After a repeated downward move leaves the board unchanged, the policy selects move_right, increasing the score from 204 to 228.

Figure S3. From a root state to action rewards in Tetris. Two initial actions are illustrated with rollout endpoints selected closest to their respective mean gains. Scores and gains beneath the images describe the displayed rollouts; the table aggregates four ten-decision continuations per action under the frozen policy.
One root Different initial actions Rollout outcomes and rewards
Tetris; ,
Root state. Score: 49.
After 10 decisions: move_left. Score: 75. Gain: +26.
After 10 decisions: rotate_ccw. Score: 58. Gain: +9.
| Action | ||
| wait | 18.25 | 0.357 |
| move_left | 24.00 | 0.631 |
| move_right | 19.50 | 0.417 |
| soft_drop | 17.25 | 0.310 |
| rotate_cw | 16.25 | 0.262 |
| rotate_ccw | 10.75 | 0.000 |
| hard_drop | 23.00 | 0.583 |
| hold_piece | 31.75 | 1.000 |
The leftward branch places the O piece on the floor beside the stack. In the rotation branch, it settles above the existing blocks, leaving empty cells beneath it.
Action value: mean score gain across four rollouts. Reward: min–max normalized action value.
| Parameter | Setting |
| Source collection | |
| Runs / decision budget | 1 per game–task pair / 10,000 steps |
| Episode decision limit | 100 |
| Root-state limit | 1,024 per run |
| Deduplication | Environment state |
| Visual context | Current image + 2 image–text history rounds |
| Skill guidance | Fixed game-specific skill |
| Skill extractor | Frozen initial model |
| Temperature / top- | 1.0 / 0.95 |
| Max. response tokens | 2,048 |
| Branch exploration | |
| Candidate actions | All public actions |
| Repetitions per action | 4 |
| Branch horizon | 10 decisions (including initial action) |
| Continuation policy | Frozen initial model |
| Concurrent root states | 8 |
| Reward construction | |
| Score source | Environment state |
| Action value | Mean score gain over continuations |
| Action-value normalization | Per-state min–max |
Table S1. Trajectory collection and reward construction.
Collection details. At each model scale, denotes the corresponding frozen Qwen3.5 initialization. In 2048, deduplication uses the ordered grid of tile values, including empty cells, and retains the first occurrence across episodes. Collection continues after task success and resets at termination or after 100 decisions. During collection and branch continuation, an invalid response consumes one decision step without executing an action or advancing game time.
| Parameter | GRPO | REINFORCE++ | DAPO |
| Training settings | |||
| Base model | Qwen3.5-2B/4B/9B | Qwen3.5-2B/4B/9B | Qwen3.5-2B/4B/9B |
| Freeze vision encoder | Yes | Yes | Yes |
| Batch size (prompts) | 64 | 64 | 64 |
| Minibatch size (prompts) | 64 | 64 | 64 |
| Learning rate | |||
| Training epochs | 2 | 2 | 2 |
| Actor update epochs | 1 | 1 | 1 |
| Max. prompt / response tokens | 8,192 / 1,024 | 8,192 / 1,024 | 8,192 / 1,024 |
| Temperature / top- | 1.0 / 1.0 | 1.0 / 1.0 | 1.0 / 1.0 |
| Strategy / precision | FSDP / BF16 | FSDP / BF16 | FSDP / BF16 |
| Algorithm settings | |||
| Responses per prompt | 5 | 1 | 5 |
| Advantage normalization | Group | Batch | Group |
| KL regularization | Actor loss | Reward penalty | None |
| KL coefficient | 0.01 | 0.001 (fixed) | 0 |
| Loss reduction | Sequence mean of token means | Token mean | Token mean |
| Discount factor | – | 1.0 | – |
| Clip ratio (lower / upper) | 0.2 / 0.2 | 0.2 / 0.2 | 0.2 / 0.28 |
| Entropy coefficient | 0 | 0 | 0 |
| Reward-group filtering | – | – | Enabled |
| Overlong penalty | – | – | Disabled |
Table S2. Table S2. Policy-training configuration. Settings for GRPO, REINFORCE++, and DAPO in verl.
References
- [1]Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning. arXiv preprint arXiv:2406.11896, 2024.arxiv.org/abs/2406.11896
- [2]Dimitri P. Bertsekas and David A. Castañón. Rollout algorithms for stochastic scheduling problems. Journal of Heuristics, 5(1):89–108, 1999.
- [3]Dimitri P. Bertsekas, John N. Tsitsiklis, and Cynara Wu. Rollout algorithms for combinatorial optimization. Journal of Heuristics, 3:245–262, 1997.
- [4]Kai-Wei Chang, Akshay Krishnamurthy, Alekh Agarwal, Hal Daumé, III, and John Langford. Learning to search better than your teacher. In Proceedings of the 32nd International Conference on Machine Learning, pages 2058–2066. PMLR, 2015.arxiv.org/abs/1502.02206
- [5]Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. First return, then explore. Nature, 590:580–586, 2021.
- [6]Yijun Guo, Junli Gong, Weicheng Wang, Hongmin Cai, Yiu-ming Cheung, and Weifeng Su. PEAM: Parametric embodied agent memory through contrastive internalization of experience in Minecraft. arXiv preprint arXiv:2605.27762, 2026.arxiv.org/abs/2605.27762
- [7]Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization. arXiv preprint arXiv:2501.03262, 2025.arxiv.org/abs/2501.03262
- [8]Lanxiang Hu, Mingjia Huo, Yuxuan Zhang, Haoyang Yu, Eric P. Xing, Ion Stoica, Tajana Rosing, Haojian Jin, and Hao Zhang. Image-Bench: How Good are LLMs at Playing Games? arXiv preprint arXiv:2505.15146, 2025.arxiv.org/abs/2505.15146
- [9]Sihao Hu, Tiansheng Huang, Gaowen Liu, Ramana Rao Kompella, Fatih Ilhan, Selim Furkan Tekin, Yichang Xu, Zachary Yahn, and Ling Liu. A survey on large language model-based game agents. ACM Computing Surveys, 2026.
- [10]Michael Kearns, Yishay Mansour, and Andrew Y. Ng. A sparse sampling algorithm for near-optimal planning in large Markov decision processes. In Proceedings of the 16th International Joint Conference on Artificial Intelligence, 1999.
- [11]Levente Kocsis and Csaba Szepesvári. Bandit based Monte-Carlo planning. In Machine Learning: ECML 2006, pages 282–293, 2006.
- [12]Michail G. Lagoudakis and Ronald Parr. Reinforcement learning as classification: Leveraging modern classifiers. In Proceedings of the 20th International Conference on Machine Learning, 2003.
- [13]Chenghao Li, Jun Liu, Songbo Zhang, Huadong Jian, Hao Ni, Lik-Hang Lee, Sung-Ho Bae, Guoqing Wang, Yang Yang, and Chaoning Zhang. Experience transfer for multimodal LLM agents in Minecraft game. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 37143–37153, 2026.arxiv.org/abs/2604.05533
- [14]Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, and Liqiang Nie. Optimus-2: Multimodal Minecraft agent with goal-observation-action conditioned policy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9039–9049, 2025.arxiv.org/abs/2502.19902
- [15]Cong Lu, Shengran Hu, and Jeff Clune. Intelligent Go-Explore: Standing on the shoulders of giant foundation models. In International Conference on Learning Representations, 2025.arxiv.org/abs/2405.15143
- [16]Mingyu Ouyang, Siyuan Hu, Kevin Qinghong Lin, Hwee Tou Ng, and Mike Zheng Shou. GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents. arXiv preprint arXiv:2604.07429, 2026.arxiv.org/abs/2604.07429
- [17]Davide Paglieri, Bartłomiej Cupiał, Samuel Coward, Ulyana Piterbarg, Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuciński, Lerrel Pinto, Rob Fergus, Jakob Nicolaus Foerster, Jack Parker-Holder, and Tim Rocktäschel. BAL-ROG: Benchmarking Agentic LLM and VLM Reasoning On Games. arXiv preprint arXiv:2411.13543, 2024.arxiv.org/abs/2411.13543
- [18]Eduardo Pignatelli, Johan Ferret, Matthieu Geist, Thomas Mesnard, Hado van Hasselt, Olivier Pietquin, and Laura Toni. A survey of temporal credit assignment in deep reinforcement learning. Transactions on Machine Learning Research, 2024.arxiv.org/abs/2312.01072
- [19]Qwen Team. Qwen3.5: Towards native multimodal agents, 2026.
- [20]Stéphane Ross and J. Andrew Bagnell. Reinforcement and imitation learning via interactive no-regret learning. arXiv preprint arXiv:1406.5979, 2014.arxiv.org/abs/1406.5979
- [21]Gabriel Sarch, Lawrence Jang, Michael J. Tarr, William W. Cohen, Kenneth Marino, and Katerina Fragkiadaki. VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought. arXiv preprint arXiv:2406.14596, 2024.arxiv.org/abs/2406.14596
- [22]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300, 2024.arxiv.org/abs/2402.03300
- [23]Guangming Sheng, Chenxiao Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A Flexible and Efficient RLHF Framework. arXiv preprint arXiv:2409.19256, 2024.DOI
- [24]Adith Swaminathan and Thorsten Joachims. Counterfactual risk minimization: Learning from logged bandit feedback. In Proceedings of the 32nd International Conference on Machine Learning, pages 814–823, 2015.arxiv.org/abs/1502.02362
- [25]Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jiechuan Jiang, Yewen Li, Ruiyi An, Molei Qin, Chuqiao Zong, Longtao Zheng, Yujie Wu, Xiaoqiang Chai, Yifei Bi, Tianbao Xie, Pengjie Gu, Xiyun Li, Ceyao Zhang, Long Tian, Chaojie Wang, Xinrun Wang, Börje F. Karlsson, Bo An, Shuicheng Yan, and Zongqing Lu. Cradle: Empowering Foundation Agents Towards General Computer Control. arXiv preprint arXiv:2403.03186, 2024.arxiv.org/abs/2403.03186
- [26]Gerald Tesauro and Gregory R. Galperin. On-line policy improvement using Monte-Carlo search. In Advances in Neural Information Processing Systems, 1996.
- [27]Kangrui Wang, Pingyue Zhang, Zihan Wang, Yaning Gao, Linjie Li, Qineng Wang, Hanyang Chen, Chi Wan, Yiping Lu, Zhengyuan Yang, Lijuan Wang, Ranjay Krishna, Jiajun Wu, Fei-Fei Li, Yejin Choi, and Manling Li. VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents. arXiv preprint arXiv:2510.16907, 2025.arxiv.org/abs/2510.16907
- [28]Peiwi Wang, Lei Li, Zhiyong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-Shepherd: Verify and reinforce LLMs step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9426–9439. Association for Computational Linguistics, 2024.
- [29]Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, Xiaojian Ma, and Yitao Liang. JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models. arXiv preprint arXiv:2311.05997, 2023.arxiv.org/abs/2311.05997
- [30]Bingqing Wei, Zhongyu Xia, Dingai Liu, Xiaoyu Zhou, Zhiwei Lin, and Yongtao Wang. ELITE: Experiential Learning and Intent-Aware Transfer for Self-improving Embodied Agents. arXiv preprint arXiv:2603.24018, 2026.arxiv.org/abs/2603.24018
- [31]Junlin Xie, Zhihong Chen, Ruifei Zhang, Xiang Wan, and Guanbin Li. Large Multimodal Agents: A Survey. arXiv preprint arXiv:2402.15116, 2024.arxiv.org/abs/2402.15116
- [32]Huanjin Yao, Ruifei Zhang, Jiaxing Huang, Jingyi Zhang, Yibo Wang, Bo Fang, Ruolin Zhu, Yongcheng Jing, Shunyu Liu, Guanbin Li, and Dacheng Tao. A Survey on Agentic Multimodal Large Language Models. arXiv preprint arXiv:2510.10991, 2025.arxiv.org/abs/2510.10991
- [33]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv preprint arXiv:2503.14476, 2025.arxiv.org/abs/2503.14476
- [34]Alex L. Zhang, Thomas L. Griffiths, Karthik R. Narasimhan, and Ofir Press. VideoGameBench: Can Vision-Language Models complete popular video games? arXiv preprint arXiv:2505.18134, 2025.arxiv.org/abs/2505.18134
- [35]Mingyu Zhang, Lifeng Zhuo, Tianxi Tan, Guocan Xie, Xian Nie, Yan Li, Renjie Zhao, Zizhu He, Ziyu Wang, Jiting Cai, and Yong-Lu Li. IPR-1: Interactive physical reasoner. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 33415–33425, 2026.arxiv.org/abs/2511.15407
- [36]Xiangxi Zheng, Linjie Li, Zhengyuan Yang, Ping Yu, Alex Jinpeng Wang, Rui Yan, Yuan Yao, and Lijuan Wang. V-MAGE: A game evaluation framework for assessing vision-centric capabilities in multimodal large language models. In Findings of the Association for Computational Linguistics: ACL 2026, pages 17707–17758, 2026.arxiv.org/abs/2504.06148
- [37]Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning, acting, and planning in language models. In Proceedings of the 41st International Conference on Machine Learning, pages 62138–62160. PMLR, 2024.arxiv.org/abs/2310.04406




