betaFeedback

WorksComputer Science

What Makes a Good Move? Learning to Act from Experience in Visual Worlds

Abstract

Learning visual policies requires turning interaction experience into guidance for individual decisions. Trajectory outcomes obscure action quality, while unguided exploration can revisit familiar situations. We present LightAct, an experience-learning framework that connects skill-guided exploration with policy internalization. Interaction segments are summarized into experience entries and consolidated into a game-specific skill that guides root collection toward broader state coverage. At each root, candidate actions are compared through short rollouts under a shared frozen policy. Environment-reported score gains are averaged across continuations and normalized into relative action rewards, producing reusable supervision for reinforcement learning. Policy training uses a compact dataset of just 1,024 decision roots collected in a single source game and paired with their action rewards. Across 170 GameWorld tasks spanning 34 games, LightAct improves overall progress by 23.6, 29.2, and 13.0 percentage points at the 2B, 4B, and 9B scales, respectively, with our strongest model competitive with reported proprietary-model results. Ablations show broader root-state coverage with skill guidance and consistent gains across three policy optimizers. Case studies show how rollouts distinguish actions with identical immediate gains.

Figure 1. Learning from source experience with LIGHTACT.

Introduction

Vision-language models (VLMs) increasingly serve as agents that connect visual perception and language reasoning to action in interactive environments [14, 25, 31, 32]. Video games expose the demands of this process: an agent must recognize the current situation, choose an executable move, and adapt as its actions change the environment [9, 16, 17]. Interaction also provides experience from which to improve. Prior work retains such experience as contextual guidance [21, 29] or uses it to update policy parameters through reinforcement learning [1, 8].

The central challenge is to turn this experience into guidance about individual decisions. A trajectory records the actions taken and their realized outcomes, leaving alternatives at the same decision point unobserved [24]. Its final outcome combines many choices: a useful move can be followed by a mistake, while a poor move can be rescued later. Environment scores summarize these outcomes, but do not directly reveal which action would have been preferable at a particular state [18]. The states encountered during exploration introduce a second limitation. Repeatedly visiting familiar situations concentrates experience on a narrow range of decisions [5, 15].

These observations suggest organizing experience around both the states reached and the choices available there. Summaries of prior interaction can guide exploration toward a broader set of decision states. At a selected state, executing alternative actions under a shared continuation policy makes their subsequent outcomes comparable. Averaging score gains over short rollouts then yields an estimate of each action’s value, which can be converted into supervision for policy learning.

We present LIGHTACT, an experience-learning framework that combines skill-guided exploration with policy internalization. We partition preliminary trajectories collected by a frozen pretrained VLM into short interaction segments. The model summarizes each segment into an experience entry describing the observed conditions, behavior, and outcome, then consolidates these entries into a game-specific textual skill. The skill guides subsequent exploration by providing reusable advice on effective behavior and recurring failures. We retain distinct visited states and their visual decision contexts as a bank of decision roots.

At each root, we compare the available actions through repeated short continuations under the same frozen policy. The resulting environment-reported score gains are averaged and normalized within the root to produce relative action rewards. Pairing these rewards with the collected decision contexts forms a fixed training set. Reinforcement learning then updates the policy by sampling responses to these contexts and assigning rewards to their parsed actions.

We instantiate LIGHTACT at the 2B, 4B, and 9B scales using Qwen3.5 [19] and evaluate on all 170 GameWorld tasks across 34 games and five categories [16]. With experience collected in 2048, overall progress improves by 23.6, 29.2, and 13.0 percentage points at the three scales, respectively. LIGHTACT-2B reaches 36.9% progress and 13.9% success, exceeding the pretrained 9B baseline on both metrics, while LIGHTACT-9B achieves 41.8% progress, competitive with reported proprietary-model results. Further experiments show broader root-state coverage with skill guidance, gains across three RL optimizers, and benefits from source-game experience across game categories. Qualitative examples illustrate how rollout comparisons reveal action preferences that immediate gains alone cannot distinguish.

Our contributions are threefold:

  • We introduce an experience-learning framework that connects skill-guided source exploration with reinforcement learning from action-level feedback.

  • We construct reusable action-reward tables by comparing continuations from shared decision roots under a frozen policy and normalizing their environment-reported score gains.

  • We demonstrate improvements across three model scales on GameWorld and examine how skill guidance, source environments, and policy optimizers contribute to learning from interaction.

A 2048 decision by LIGHTACT-2B

Figure 2. A 2048 decision by LIGHTACT-2B. When repeated downward moves no longer change the board, the policy switches to moving right and gains 24 points.

Related Work

Vision-Language Game Agents

Video games test whether visual understanding supports sequential decisions [36]. Cradle [25] combines perception, planning, and memory to operate through screenshots and keyboard or mouse controls. BALROG [17] evaluates language and vision-language agents across games and reveals weaknesses in visually grounded interaction. VideoGameBench [34] examines play from raw visual input under real-time and paused interaction, separating decision quality from the effects of inference latency. GameWorld [16] provides browser games with standardized interfaces and state-verifiable task evaluation. Complementing these benchmarks, lmgame-Bench [8] studies perception and memory scaffolds and reports that reinforcement learning on individual games can transfer beyond the training environment. Our work examines how source-game interaction supports such policy learning.

Experience Learning of VLMs

VLM agents learn from accumulated interaction through contextual guidance and parameter updates. External memory retains experience for retrieval at decision time [13], while parameter updates encode it in the policy [6]. JARVIS-1 [29] retrieves multimodal Minecraft experience to support planning, while ELITE [30] extracts reusable strategies from trajectories and retrieves them according to task intent. ICAL [21] refines suboptimal demonstrations through execution and human feedback, annotating causal relations, state changes, and subgoals; the resulting examples support both retrieval and supervised fine-tuning. For reinforcement learning, DigiRL [1] adapts a pretrained VLM for device control through offline initialization and online interaction, using a VLM evaluator for feedback. VAGEN [27] improves explicit state estimation and transition prediction in multi-turn visual interaction through reinforcement learning. We connect experience-guided source exploration with policy internalization.

Rollout-Based Policy Learning

Rollouts estimate the value of an action from the outcomes that follow it. Monte Carlo policy improvement compares candidate moves by continuing each with a common base policy [26]. In combinatorial optimization and stochastic scheduling, heuristic continuations estimate the future cost of each choice [2, 3]. Sparse sampling [10] estimates action values over sampled lookahead trees, while UCT [11] uses a bandit rule to allocate simulations among branches. Rollout comparisons also provide supervision for policy learning [35]. Classification-based policy iteration [12] uses the preferred action at a sampled state as a training label. AggreVaTe [20] trains on expert cost-to-go estimates, while LOLS [4] evaluates decisions under both reference and learned policies. For language models, LATS [37] guides tree search with model evaluations and environment feedback, and Math-Shepherd [28] uses sampled completions to assess intermediate reasoning steps. We use score gains from a shared game state to supervise action selection.

Method

Experience gathered during interaction can guide subsequent decisions through both contextual guidance and policy learning. Building on this view, LIGHTACT (Figure 3) distills source-game trajectories into textual skills for exploration and compares action outcomes at the collected roots to construct action-level rewards for policy internalization.

Figure 3. Overview of LIGHTACT. Interaction segments are summarized into experience entries, then consolidated into a source-game skill that guides root-state collection. At each root, short rollouts compare candidate actions through their subsequent score gains. These gains are normalized into action-level rewards for reinforcement learning, internalizing the collected experience in a policy evaluated across games.

Decision-Making in Game Environment

In GameWorld [16], an agent pursues a game-specific objective through sequential interaction. At each decision, it receives the game rules, task instruction, role information, and available semantic actions, together with the current screenshot and recent observation–action history. The agent generates a response, parses it into an executable action, observes its effect on the game, and continues toward the objective. This interaction links visual understanding to decisions whose consequences unfold over subsequent steps.

For a game environment ee with a fixed, finite action space Ae\mathcal{A}_e, let st∈Ses_t \in\mathcal{S}_e be its state, at∈Aea_t \in\mathcal{A}_e an action, and ot=Ωe(st)o_t = \Omega_e(s_t) the visual observation. Interaction follows

st+1∼pe(⋅∣st,at),xt=(g,ot,ht),(1)s_{t+1} \sim p_e(\cdot\mid s_t, a_t), \qquad x_t = (g, o_t, h_t), \tag*{(1)}

where gg denotes the task specification together with the game instructions, oto_t is the current screenshot, and hth_t contains the recent observations and actions. The instructions describe the objective, rules, acting role, and semantic action interface. A task-specific evaluator Eval⁡e,g\operatorname{Eval}_{e,g} determines whether the current environment state satisfies the task’s completion criteria, with Eval⁡e,g(st)=1\operatorname{Eval}_{e,g}(s_t) = 1 indicating success. Let πθ\pi_\theta denote the policy parameterized by θ\theta. Its pretrained parameters θ0\theta_0 define the frozen initial policy π0=πθ0\pi_0 = \pi_{\theta_0}. The policy generates a response yt∼πθ(⋅∣xt)y_t \sim\pi_\theta(\cdot\mid x_t) from this context. The parser P\mathcal{P} attempts to extract an executable action from yty_t: successful parsing yields at=P(yt)a_t = \mathcal{P}(y_t), while invalid responses yield no executable action. For games with multiple acting roles, the other roles’ behavior is part of the transition process.

Exploration and Root-State Collection

The states reached during source interaction determine the decisions available for policy learning. We collect trajectories with a collection policy βe\beta_e, sampling responses yt∼βe(⋅∣xt)y_t \sim\beta_e(\cdot\mid x_t) and executing successfully parsed actions. The basic collection policy is βe(⋅∣xt)=π0(⋅∣xt)\beta_e(\cdot\mid x_t) = \pi_0(\cdot\mid x_t).

Among the nonterminal states visited during collection, we retain the first occurrence under an environment-specific equivalence criterion. For temporally ordered candidates indexed by CC, the selected indices are

I={i∈C:Deduplicate⁡(si)=1}.(2)\mathcal{I} = \{i \in C : \operatorname{Deduplicate}(s_i) = 1\}. \tag*{(2)}

Each selected state is stored with its decision context xix_i. The selected states form the root bank:

Droot={(si,xi):i∈I},N=∣I∣.(3)\mathcal{D}_{\mathrm{root}} = \{(s_i, x_i) : i \in\mathcal{I}\}, \qquad N = |\mathcal{I}|. \tag*{(3)}

Their original contexts xi=(g,oi,hi)x_i = (g, o_i, h_i) and the action rewards constructed below form the training examples for policy internalization.

Skill-Guided Exploration

To guide exploration toward a richer set of decision states, we first collect preliminary trajectories Teinit\mathcal{T}^{\mathrm{init}}_e with π0\pi_0 and partition them into short interaction segments {σj}j=1M\{\sigma_j\}_{j=1}^{M}. Each segment contains consecutive visual observations, executed actions, and their resulting changes. Under an entry-summarization prompt, the frozen model distills each segment into an experience entry:

ηj=Fθ0entry(g,σj).(4)\eta_j = F_{\theta_0}^{\mathrm{entry}}(g, \sigma_j). \tag*{(4)}

Each entry captures the observed conditions, behavior, and outcome within its segment.

The same model then consolidates the entries across segments into one textual skill for environment ee:

κe=Fθ0skill(g,{ηj}j=1M).(5)\kappa_e = F_{\theta_0}^{\mathrm{skill}}\left(g, \{\eta_j\}_{j=1}^{M}\right). \tag*{(5)}

Through a skill-consolidation prompt, recurring observations are combined into concise guidance on useful behavior and failure patterns, tied to the conditions under which they arise. This guidance helps the agent reuse effective interactions and adjust its behavior when exploration stalls. We include this skill in the collection policy’s decision context:

yt∼πθ(⋅∣xt)=π0(⋅∣xt,κe).(6)y_t \sim\pi_\theta(\cdot\mid x_t) = \pi_0(\cdot\mid x_t,\kappa_e). \tag*{(6)}

The policy interprets the guidance using the current observation and recent history. The skill and model parameters remain fixed throughout guided collection. We construct the root bank Droot\mathcal{D}_{\mathrm{root}} from the guided trajectories using the root-selection procedure above.

Rollout-Based Action Value Estimation

For each decision root, we restore the same environment state sis_i and decision context xix_i before evaluating each candidate action. Starting from sis_i, we execute action aa and let π0\pi_0 select the subsequent actions. The resulting branch contains at most HH decisions, including the initial action, and ends at the first episode termination if it occurs earlier:

Bi,a∼BH(⋅∣si,xi,a,π0).(7)B_{i,a} \sim\mathcal{B}_H(\cdot\mid s_i,x_i,a,\pi_0). \tag*{(7)}

At each subsequent decision, π0\pi_0 conditions on the task, current observation, and recent interaction history within the branch. Using the same frozen policy across actions and roots provides a shared basis for comparing their outcomes.

Let Se(s)S_e(s) denote the game score returned by the environment at state ss and sBs_B the final state of branch BB. A continuation from sis_i contributes the score gain

Ge(si,B)=Se(sB)−Se(si).(8)G_e(s_i, B) = S_e(s_B) - S_e(s_i). \tag*{(8)}

For a terminal branch, sBs_B is the state before reset. The finite-horizon action value and its estimate from KK continuations are

QHπ0(i,a)=EBi,a[Ge(si,Bi,a)],(9)Q^{\pi_0}_{H}(i, a) = \mathbb{E}_{B_{i,a}}\left[G_e(s_i, B_{i,a})\right], \tag*{(9)}
Q^i(a)=1K∑k=1KGe(si,Bi,a(k)).(10)\widehat{Q}_i(a) = \frac{1}{K}\sum_{k=1}^{K} G_e\left(s_i, B_{i,a}^{(k)}\right). \tag*{(10)}

The estimate measures score gain after choosing aa and continuing under π0\pi_0 for the specified horizon.

Reward Shaping

The estimated score gains Q^i(a)\widehat{Q}_i(a) quantify the outcomes of candidate actions in the game’s score units. To express action preferences on a common reward scale, we normalize these values within each root:

ri(a)={Q^i(a)−qi−qi+−qi−,qi+>qi−,0,qi+=qi−.(11)r_i(a) = \begin{cases} \frac{\widehat{Q}_i(a)-q_i^-}{q_i^+-q_i^-}, & q_i^+>q_i^-,\\ 0, & q_i^+=q_i^-. \end{cases} \tag*{(11)}

where qi−=min⁡a∈AeQ^i(a)q_i^-=\min_{a\in A_e}\widehat{Q}_i(a) and qi+=max⁡a∈AeQ^i(a)q_i^+=\max_{a\in A_e}\widehat{Q}_i(a). This normalization preserves the estimated ordering of actions and expresses their relative quality within a shared decision context. When all action values coincide, the root supplies no preference signal and all action rewards are zero.

RL-Based Policy Internalization

After collection, we pair each decision context in Droot\mathcal{D}_{\mathrm{root}} with its constructed reward function ri:Ae→[0,1]r_i:A_e\to[0,1]:

Dreward={(xi,ri)∣(si,xi)∈Droot}.(12)\mathcal{D}_{\mathrm{reward}}=\{(x_i,r_i)\mid(s_i,x_i)\in\mathcal{D}_{\mathrm{root}}\}. \tag*{(12)}

This converts the action comparisons in source experience into policy-learning examples: xix_i supplies the task and visual history, while rir_i provides feedback on the selected action. Starting from θ=θ0\theta=\theta_0, we train πθ\pi_\theta to select valuable actions directly from xix_i.

The policy πθ\pi_\theta generates a response yy from the decision context xix_i, which is parsed into an action a=P(y)a=\mathcal{P}(y) when valid. A response is valid when it satisfies the public action syntax and argument constraints and selects an action in AeA_e. The response reward is

Ri(y)={ri(a),if y is valid,0,otherwise.(13)R_i(y)= \begin{cases} r_i(a), & \text{if }y\text{ is valid},\\ 0, & \text{otherwise}. \end{cases} \tag*{(13)}

Sampling examples uniformly from Dreward\mathcal{D}_{\mathrm{reward}}, we optimize the expected response reward with reference-policy regularization:

JRL(θ)=E(xi,ri)∼Dreward[Ey∼πθ(⋅∣xi)[Ri(y)]−λDKL(πθ(⋅∣xi)∥π0(⋅∣xi))].(14)J_{\mathrm{RL}}(\theta)=\mathbb{E}_{(x_i,r_i)\sim\mathcal{D}_{\mathrm{reward}}}\left[ \mathbb{E}_{y\sim\pi_\theta(\cdot\mid x_i)}[R_i(y)] -\lambda D_{\mathrm{KL}}\left(\pi_\theta(\cdot\mid x_i)\mathbin{\|}\pi_0(\cdot\mid x_i)\right) \right]. \tag*{(14)}

where λ≥0\lambda\ge0 controls deviation from the initial policy π0\pi_0. The current policy samples responses conditioned on xix_i and receives Ri(y)R_i(y); the collected contexts and reward maps remain fixed during these updates. Established policy-gradient algorithms optimize this objective, with their configurations specified in the experiments.

After learning, the policy generates yt∼πθ∗(⋅∣xt)y_t\sim\pi_{\theta^*}(\cdot\mid x_t) and executes the parsed action at=P(yt)a_t=\mathcal{P}(y_t) using the task description, visual observations, and action interface of the environment being played.

Experiments

Benchmark and Evaluation Setting

We evaluate LightAct on GameWorld [16], which comprises 34 games and 170 tasks across five categories: Arcade, Platformer, Puzzle, Runner, and Simulation. Each run has a budget of 100 decisions, with the environment paused during inference. At each decision, the policy receives the game instructions, the current screenshot, and two rounds of image–text interaction history, and selects an action from the game’s interface. Action duration, episode termination, and resets follow the protocol. Published GameWorld results use greedy decoding with one run per task. Our Qwen3.5 and LightAct evaluations use temperature 1.0, top-pp 0.95, a 2,048-token output cap, and three runs per task. We use GameWorld’s state-based evaluator to measure success rate (SR), the fraction of successful runs, and progress (PG), normalized completion toward task-specific targets.

Implementation Details

We instantiate LightAct with Qwen3.5-2B, Qwen3.5-4B, and Qwen3.5-9B [19]. At each scale, experience collection, branch continuation, and skill extraction use frozen copies of the same model that initializes the trainable policy. Acting uses the native non-thinking mode. For each source game–task pair, we collect trajectories within a budget of 10,000 decisions and retain up to 1,024 deduplicated roots, with seed 42 for the initial reset. We equip source games with snapshot support through exportSnapshot() and importSnapshot(), restoring each decision root before evaluating candidate actions. Collection and branch continuation use temperature 1.0, top-pp 0.95, and a 2,048-token output cap. Branch evaluation uses H=10H=10 including the initial action and K=4K=4 continuations, processing eight roots concurrently. Rewards use environment-reported score gains with per-root min–max normalization. We train with GRPO [22], REINFORCE++ [7], or DAPO [33] in verl/HybridFlow [23]. Tables S1 and S2 in the appendix give the collection and reward settings and the training hyperparameters, respectively.

Baselines and Main Comparisons

LIGHTACT achieves competitive progress against proprietary models. LIGHTACT brings compact policies close to strong proprietary models in benchmark progress (Table 1). At 4B, it reaches 40.5% overall PG, comparable to GPT-5.2 at 40.6%; at 9B, it reaches 41.8%, approaching Gemini-3-Flash-Preview at 41.9%. On Puzzle tasks, LIGHTACT-9B achieves 59.2% PG, exceeding the reported progress of both models. The remaining gap is clearer in task completion, where overall SR reaches 18.4% for LIGHTACT-9B compared with 21.2% for Gemini. By comparing actions from a shared state, LIGHTACT turns interaction outcomes into supervision for policy learning. These results suggest that internalizing such experience enables compact VLMs to achieve competitive game-playing performance through direct policy inference.

ModelArcade SRArcade PGPlatformer SRPlatformer PGPuzzle SRPuzzle PGRunner SRRunner PGSimulation SRSimulation PGOverall SROverall PG
Human
Novice Player45.755.560.065.651.463.160.072.060.062.055.364.1
Expert Player65.773.985.088.068.677.182.587.885.086.077.182.6
Generalist Multimodal Agents
Claude-Sonnet-4.65.728.322.537.025.751.530.051.915.016.620.639.3
Gemini-3-Flash-Preview5.726.325.041.225.754.832.555.410.021.121.241.9
GLM-4.6V8.622.820.033.95.729.127.549.10.05.314.130.8
GPT-5.28.629.322.536.728.656.227.552.610.016.920.640.6
Grok-4.1-Fast-Reasoning8.623.722.537.314.346.625.049.05.010.416.536.0
Kimi-K2.58.626.420.035.325.751.427.549.75.011.718.837.4
Qwen3-VL-Plus8.625.622.537.814.339.127.551.10.010.016.535.4
Seed-1.811.433.522.534.622.948.727.551.210.018.820.039.0
Qwen3-VL-235B-A22B5.723.217.529.58.633.327.550.40.03.613.530.8
Qwen3-VL-30B-A3B2.920.320.036.52.926.127.551.10.03.512.430.6
Qwen3.5-2B0.05.65.810.62.99.47.528.40.09.13.713.3
Qwen3.5-4B4.89.42.56.34.821.61.78.25.012.73.511.3
Qwen3.5-9B10.528.85.017.612.435.317.544.10.09.510.028.8
Ours
LIGHTACT-2B7.628.717.530.318.149.515.846.16.724.013.936.9
LIGHTACT-4B17.136.415.031.023.854.520.849.76.723.717.640.5
LIGHTACT-9B14.334.321.736.626.759.219.248.73.320.918.441.8

Table 1. Main results on GameWorld. SR and PG (%) are reported by category and overall. All three LIGHTACT models are trained with GRPO using experience collected in 2048. Qwen3.5 and LIGHTACT results are averaged over three evaluation runs per task. Published baseline results are taken from GameWorld [16].

LIGHTACT consistently improves policy performance across model scales. LIGHTACT increases both SR and PG in all five game categories at 2B, 4B, and 9B, indicating benefits for task completion as well as partial progress. The overall PG gains are 23.6, 29.2, and 13.0 percentage points, respectively. The 4B policy gains most on Runner tasks, where PG rises from 8.2% to 49.7%, even though its training experience comes from a puzzle game. Notably, LIGHTACT-2B surpasses the pretrained 9B baseline in both PG (36.9% versus 28.8%) and SR (13.9% versus 10.0%), showing that adaptation through experience can improve the effectiveness of a smaller policy. The 9B model also gains 8.4 percentage points in SR over its initialization, demonstrating that stronger pretrained performance still leaves room for im-

Ablations and Diagnostics

Policy internalization yields consistent gains across RL algorithms. We compare GRPO, REINFORCE++, and DAPO, using the same reward tables for each model scale (Table 2). All nine trained configurations improve overall PG over their base policies, with gains ranging from 10.8 to 29.2 percentage points. This pattern suggests that the action-level rewards provide useful supervision across different policy-update rules. The optimizer still affects how these gains are distributed across games: at 4B, DAPO achieves higher Puzzle and Simulation PG than GRPO, while GRPO attains higher overall PG. Thus, the common benefit of learning from action outcomes persists alongside algorithm-dependent differences in performance across categories.

ModelArc. PGPlat. PGPuzz. PGRun. PGSim. PGOverall PG
Qwen3.5-2B
Base5.610.69.428.49.113.3
+REINFORCE++24.028.646.246.820.134.5
+GRPO28.730.349.546.124.036.9
+DAPO26.629.950.245.020.635.9
Qwen3.5-4B
Base9.46.321.68.212.711.3
+REINFORCE++29.730.951.948.919.537.9
+GRPO36.431.054.549.723.740.5
+DAPO33.830.056.349.126.340.2
Qwen3.5-9B
Base28.817.635.344.19.528.8
+REINFORCE++36.335.553.149.011.239.6
+GRPO34.336.659.248.720.941.8
+DAPO33.132.459.749.719.440.7

Table 2. Policy internalization. GameWorld PG (%) for base and trained policies across three model scales and RL algorithms.

Skill guidance broadens the score distribution of collected roots. Figure 5 compares 1,024 roots collected with and without skill guidance in 2048; Figure 7 shows the skill used during collection. By drawing on interaction experience, the skill guides the policy away from repeated ineffective moves and helps it explore a broader range of game states. Guided collection extends the score range from 0–552 to 0–944 and raises the mean score from 186.6 to 280.2. The proportion of roots scoring at least 500 increases from 1.3% to 19.4%, while lower-score states remain represented. With GRPO, skill guidance raises PG from 32.98% to 36.90% and SR from 12.35% to 13.92% (lower panels of Figure 5). With REINFORCE++, PG changes little (34.60% to 34.55%), while SR rises from 11.96% to 12.75%. These results suggest that skill guidance broadens the experience available for policy learning, with benefits that depend on the optimizer.

Figure 5. Skill-guided exploration. Top: score distributions of 1,024 roots in 2048 (50-point bins). Bottom: GameWorld PG and SR for 2B policies trained with GRPO or REINFORCE++ (RF++).

Figure 7. 2048 exploration skill. The textual guidance used during root-state collection, with action-selection principles highlighted.

Source-game experience supports generalization across game categories. With Tetris as the source game, both GRPO and REINFORCE++ improve PG over the 2B base policy across all five categories (Figure 4). Arcade PG, for example, rises from 5.6% to 29.8% and 29.6%, respectively. These gains extend beyond the source game’s Puzzle category, suggesting that policy internalization helps LIGHTACT apply experience to different game mechanics. Experience from a single game can provide supervision for broader visual decisions. The varying gains across categories indicate that transfer is uneven, leaving room to improve the breadth of behaviors learned from source experience.

Figure 4. Source-environment ablation. GameWorld SR and PG for 2B policies trained on 2048 or Tetris with GRPO and REINFORCE++.

Rollout outcomes distinguish actions with identical immediate gains. In Figure 6, moving left or right from the same root yields an immediate score gain of 24, but the resulting boards support different continuations. The rightward branch produces a 64 tile, whereas the leftward branch retains a maximum tile of 32. Across four rollouts, moving right achieves a mean gain of 144 versus 67 for moving left, yielding normalized rewards of 1.000 and 0.330. The rollout comparison turns delayed action outcomes into a training preference that immediate score gains alone cannot provide.

Root state, after 10 decisions, and four-rollout estimates for actions in 2048

Figure 6. From a root state to action rewards in 2048. Left and right gain 24 immediately but yield different outcomes. The table averages four ten-decision continuations per action and normalizes gains into rewards; endpoints are nearest each action’s mean gain.

Conclusion

We presented LIGHTACT, a framework that turns interaction experience into guidance for exploration and supervision for policy learning. Game-specific skills broaden root-state coverage, while short rollouts from shared roots reveal the relative utility of candidate actions. Experiments on Game-World show consistent improvements across model scales and policy optimizers, with compact policies achieving competitive progress against proprietary models. These findings demonstrate the value of organizing experience around states and action outcomes for learning game-playing policies.

Limitations

Generalization across source environments remains limited, and the current framework does not directly support action spaces with continuous parameters. Our study focuses on short-horizon decisions; long-horizon decision-making remains to be validated. Future work will investigate integration with multi-turn reinforcement learning and evaluation on games that require sustained planning. Reward shaping currently relies on scores read from environment states. We will further explore purely visual self-improvement that derives learning signals from observations without access to internal environment states.

LLM Usage Statement

OpenAI Codex was used to assist with code generation and completion, and OpenAI GPT models assisted with related-literature discovery. Codex also assisted with drafting and editing manuscript text. The authors led the research process, problem formulation, computational experiments, and factual verification, and take full responsibility for the scientific claims, results, references, and final manuscript.

Appendix

A Mario decision by LIGHTACT-2B trained with REINFORCE++ on 2048, showing system instructions, historical messages, the current user message, model output, and observed outcome

Figure S2. A Mario decision by LIGHTACT-2B trained with REINFORCE++ on 2048 (task 17_02, repetition 2, step 40). The recorded jump action is followed by mushroom collection and task completion. Shared system instructions appear in Figure S1.

A 2048 game-agent interaction showing system input, historical messages, current user message, model output, and observed outcome

Figure S1. A 2048 decision by LIGHTACT-2B trained with REINFORCE++ (task 01_02, repetition 3, step 46). After a repeated downward move leaves the board unchanged, the policy selects move_right, increasing the score from 204 to 228.

From a root state to action rewards in Tetris

Figure S3. From a root state to action rewards in Tetris. Two initial actions are illustrated with rollout endpoints selected closest to their respective mean gains. Scores and gains beneath the images describe the displayed rollouts; the table aggregates four ten-decision continuations per action under the frozen policy.

One root →\to Different initial actions →\to Rollout outcomes and rewards

Tetris; K=4K = 4, H=10H = 10

Root state. Score: 49.

After 10 decisions: move_left. Score: 75. Gain: +26.

After 10 decisions: rotate_ccw. Score: 58. Gain: +9.

ActionQ^i(a)\hat{Q}_i(a)ri(a)r_i(a)
wait18.250.357
move_left24.000.631
move_right19.500.417
soft_drop17.250.310
rotate_cw16.250.262
rotate_ccw10.750.000
hard_drop23.000.583
hold_piece31.751.000

The leftward branch places the O piece on the floor beside the stack. In the rotation branch, it settles above the existing blocks, leaving empty cells beneath it.

Action value: mean score gain across four rollouts. Reward: min–max normalized action value.

ParameterSetting
Source collection
Runs / decision budget1 per game–task pair / 10,000 steps
Episode decision limit100
Root-state limit1,024 per run
DeduplicationEnvironment state
Visual contextCurrent image + 2 image–text history rounds
Skill guidanceFixed game-specific skill
Skill extractorFrozen initial model π0\pi_0
Temperature / top-pp1.0 / 0.95
Max. response tokens2,048
Branch exploration
Candidate actionsAll public actions
Repetitions per action KK4
Branch horizon HH10 decisions (including initial action)
Continuation policyFrozen initial model π0\pi_0
Concurrent root states8
Reward construction
Score sourceEnvironment state
Action valueMean score gain over KK continuations
Action-value normalizationPer-state min–max

Table S1. Trajectory collection and reward construction.

Collection details. At each model scale, π0\pi_0 denotes the corresponding frozen Qwen3.5 initialization. In 2048, deduplication uses the ordered grid of tile values, including empty cells, and retains the first occurrence across episodes. Collection continues after task success and resets at termination or after 100 decisions. During collection and branch continuation, an invalid response consumes one decision step without executing an action or advancing game time.

ParameterGRPOREINFORCE++DAPO
Training settings
Base modelQwen3.5-2B/4B/9BQwen3.5-2B/4B/9BQwen3.5-2B/4B/9B
Freeze vision encoderYesYesYes
Batch size (prompts)646464
Minibatch size (prompts)646464
Learning rate5×10−75 \times10^{-7}5×10−75 \times10^{-7}5×10−75 \times10^{-7}
Training epochs222
Actor update epochs111
Max. prompt / response tokens8,192 / 1,0248,192 / 1,0248,192 / 1,024
Temperature / top-pp1.0 / 1.01.0 / 1.01.0 / 1.0
Strategy / precisionFSDP / BF16FSDP / BF16FSDP / BF16
Algorithm settings
Responses per prompt515
Advantage normalizationGroupBatchGroup
KL regularizationActor lossReward penaltyNone
KL coefficient0.010.001 (fixed)0
Loss reductionSequence mean of token meansToken meanToken mean
Discount factor γ\gamma–1.0–
Clip ratio (lower / upper)0.2 / 0.20.2 / 0.20.2 / 0.28
Entropy coefficient000
Reward-group filtering––Enabled
Overlong penalty––Disabled

Table S2. Table S2. Policy-training configuration. Settings for GRPO, REINFORCE++, and DAPO in verl.

References

  1. [1]Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning. arXiv preprint arXiv:2406.11896, 2024.arxiv.org/abs/2406.11896
  2. [2]Dimitri P. Bertsekas and David A. Castañón. Rollout algorithms for stochastic scheduling problems. Journal of Heuristics, 5(1):89–108, 1999.
  3. [3]Dimitri P. Bertsekas, John N. Tsitsiklis, and Cynara Wu. Rollout algorithms for combinatorial optimization. Journal of Heuristics, 3:245–262, 1997.
  4. [4]Kai-Wei Chang, Akshay Krishnamurthy, Alekh Agarwal, Hal Daumé, III, and John Langford. Learning to search better than your teacher. In Proceedings of the 32nd International Conference on Machine Learning, pages 2058–2066. PMLR, 2015.arxiv.org/abs/1502.02206
  5. [5]Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. First return, then explore. Nature, 590:580–586, 2021.
  6. [6]Yijun Guo, Junli Gong, Weicheng Wang, Hongmin Cai, Yiu-ming Cheung, and Weifeng Su. PEAM: Parametric embodied agent memory through contrastive internalization of experience in Minecraft. arXiv preprint arXiv:2605.27762, 2026.arxiv.org/abs/2605.27762
  7. [7]Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization. arXiv preprint arXiv:2501.03262, 2025.arxiv.org/abs/2501.03262
  8. [8]Lanxiang Hu, Mingjia Huo, Yuxuan Zhang, Haoyang Yu, Eric P. Xing, Ion Stoica, Tajana Rosing, Haojian Jin, and Hao Zhang. Image-Bench: How Good are LLMs at Playing Games? arXiv preprint arXiv:2505.15146, 2025.arxiv.org/abs/2505.15146
  9. [9]Sihao Hu, Tiansheng Huang, Gaowen Liu, Ramana Rao Kompella, Fatih Ilhan, Selim Furkan Tekin, Yichang Xu, Zachary Yahn, and Ling Liu. A survey on large language model-based game agents. ACM Computing Surveys, 2026.
  10. [10]Michael Kearns, Yishay Mansour, and Andrew Y. Ng. A sparse sampling algorithm for near-optimal planning in large Markov decision processes. In Proceedings of the 16th International Joint Conference on Artificial Intelligence, 1999.
  11. [11]Levente Kocsis and Csaba Szepesvári. Bandit based Monte-Carlo planning. In Machine Learning: ECML 2006, pages 282–293, 2006.
  12. [12]Michail G. Lagoudakis and Ronald Parr. Reinforcement learning as classification: Leveraging modern classifiers. In Proceedings of the 20th International Conference on Machine Learning, 2003.
  13. [13]Chenghao Li, Jun Liu, Songbo Zhang, Huadong Jian, Hao Ni, Lik-Hang Lee, Sung-Ho Bae, Guoqing Wang, Yang Yang, and Chaoning Zhang. Experience transfer for multimodal LLM agents in Minecraft game. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 37143–37153, 2026.arxiv.org/abs/2604.05533
  14. [14]Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, and Liqiang Nie. Optimus-2: Multimodal Minecraft agent with goal-observation-action conditioned policy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9039–9049, 2025.arxiv.org/abs/2502.19902
  15. [15]Cong Lu, Shengran Hu, and Jeff Clune. Intelligent Go-Explore: Standing on the shoulders of giant foundation models. In International Conference on Learning Representations, 2025.arxiv.org/abs/2405.15143
  16. [16]Mingyu Ouyang, Siyuan Hu, Kevin Qinghong Lin, Hwee Tou Ng, and Mike Zheng Shou. GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents. arXiv preprint arXiv:2604.07429, 2026.arxiv.org/abs/2604.07429
  17. [17]Davide Paglieri, Bartłomiej Cupiał, Samuel Coward, Ulyana Piterbarg, Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuciński, Lerrel Pinto, Rob Fergus, Jakob Nicolaus Foerster, Jack Parker-Holder, and Tim Rocktäschel. BAL-ROG: Benchmarking Agentic LLM and VLM Reasoning On Games. arXiv preprint arXiv:2411.13543, 2024.arxiv.org/abs/2411.13543
  18. [18]Eduardo Pignatelli, Johan Ferret, Matthieu Geist, Thomas Mesnard, Hado van Hasselt, Olivier Pietquin, and Laura Toni. A survey of temporal credit assignment in deep reinforcement learning. Transactions on Machine Learning Research, 2024.arxiv.org/abs/2312.01072
  19. [19]Qwen Team. Qwen3.5: Towards native multimodal agents, 2026.
  20. [20]Stéphane Ross and J. Andrew Bagnell. Reinforcement and imitation learning via interactive no-regret learning. arXiv preprint arXiv:1406.5979, 2014.arxiv.org/abs/1406.5979
  21. [21]Gabriel Sarch, Lawrence Jang, Michael J. Tarr, William W. Cohen, Kenneth Marino, and Katerina Fragkiadaki. VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought. arXiv preprint arXiv:2406.14596, 2024.arxiv.org/abs/2406.14596
  22. [22]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300, 2024.arxiv.org/abs/2402.03300
  23. [23]Guangming Sheng, Chenxiao Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A Flexible and Efficient RLHF Framework. arXiv preprint arXiv:2409.19256, 2024.DOI
  24. [24]Adith Swaminathan and Thorsten Joachims. Counterfactual risk minimization: Learning from logged bandit feedback. In Proceedings of the 32nd International Conference on Machine Learning, pages 814–823, 2015.arxiv.org/abs/1502.02362
  25. [25]Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jiechuan Jiang, Yewen Li, Ruiyi An, Molei Qin, Chuqiao Zong, Longtao Zheng, Yujie Wu, Xiaoqiang Chai, Yifei Bi, Tianbao Xie, Pengjie Gu, Xiyun Li, Ceyao Zhang, Long Tian, Chaojie Wang, Xinrun Wang, Börje F. Karlsson, Bo An, Shuicheng Yan, and Zongqing Lu. Cradle: Empowering Foundation Agents Towards General Computer Control. arXiv preprint arXiv:2403.03186, 2024.arxiv.org/abs/2403.03186
  26. [26]Gerald Tesauro and Gregory R. Galperin. On-line policy improvement using Monte-Carlo search. In Advances in Neural Information Processing Systems, 1996.
  27. [27]Kangrui Wang, Pingyue Zhang, Zihan Wang, Yaning Gao, Linjie Li, Qineng Wang, Hanyang Chen, Chi Wan, Yiping Lu, Zhengyuan Yang, Lijuan Wang, Ranjay Krishna, Jiajun Wu, Fei-Fei Li, Yejin Choi, and Manling Li. VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents. arXiv preprint arXiv:2510.16907, 2025.arxiv.org/abs/2510.16907
  28. [28]Peiwi Wang, Lei Li, Zhiyong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-Shepherd: Verify and reinforce LLMs step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9426–9439. Association for Computational Linguistics, 2024.
  29. [29]Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, Xiaojian Ma, and Yitao Liang. JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models. arXiv preprint arXiv:2311.05997, 2023.arxiv.org/abs/2311.05997
  30. [30]Bingqing Wei, Zhongyu Xia, Dingai Liu, Xiaoyu Zhou, Zhiwei Lin, and Yongtao Wang. ELITE: Experiential Learning and Intent-Aware Transfer for Self-improving Embodied Agents. arXiv preprint arXiv:2603.24018, 2026.arxiv.org/abs/2603.24018
  31. [31]Junlin Xie, Zhihong Chen, Ruifei Zhang, Xiang Wan, and Guanbin Li. Large Multimodal Agents: A Survey. arXiv preprint arXiv:2402.15116, 2024.arxiv.org/abs/2402.15116
  32. [32]Huanjin Yao, Ruifei Zhang, Jiaxing Huang, Jingyi Zhang, Yibo Wang, Bo Fang, Ruolin Zhu, Yongcheng Jing, Shunyu Liu, Guanbin Li, and Dacheng Tao. A Survey on Agentic Multimodal Large Language Models. arXiv preprint arXiv:2510.10991, 2025.arxiv.org/abs/2510.10991
  33. [33]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv preprint arXiv:2503.14476, 2025.arxiv.org/abs/2503.14476
  34. [34]Alex L. Zhang, Thomas L. Griffiths, Karthik R. Narasimhan, and Ofir Press. VideoGameBench: Can Vision-Language Models complete popular video games? arXiv preprint arXiv:2505.18134, 2025.arxiv.org/abs/2505.18134
  35. [35]Mingyu Zhang, Lifeng Zhuo, Tianxi Tan, Guocan Xie, Xian Nie, Yan Li, Renjie Zhao, Zizhu He, Ziyu Wang, Jiting Cai, and Yong-Lu Li. IPR-1: Interactive physical reasoner. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 33415–33425, 2026.arxiv.org/abs/2511.15407
  36. [36]Xiangxi Zheng, Linjie Li, Zhengyuan Yang, Ping Yu, Alex Jinpeng Wang, Rui Yan, Yuan Yao, and Lijuan Wang. V-MAGE: A game evaluation framework for assessing vision-centric capabilities in multimodal large language models. In Findings of the Association for Computational Linguistics: ACL 2026, pages 17707–17758, 2026.arxiv.org/abs/2504.06148
  37. [37]Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning, acting, and planning in language models. In Proceedings of the 41st International Conference on Machine Learning, pages 62138–62160. PMLR, 2024.arxiv.org/abs/2310.04406

Paper details

vlnc:W_nc4b2mpz
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Codex, GPT (multiple versions)

View statement
Looking for related papers…
Contents