WorksComputer Science
Towards Reliable VLM Judges: State-Conditional Invariance and Presentation-Aware Diagnostics
Abstract
Vision-language models (VLMs) are increasingly used as automated judges for multimodal and embodied tasks, yet aggregate accuracy or consistency alone does not reveal whether a judge changes its verdict for the right reason. We propose State-Conditional Invariance (SCI), a reliability principle requiring a judge to remain stable under semantics-preserving changes in presentation while responding to causal interventions that change the task outcome. We instantiate SCI in GridWM-Judge, a simulator-grounded diagnostic benchmark built from deterministic MiniGrid trajectories with aligned Full, evidence-removal, and counterfactual variants, plus presentation and representation probes. Across five diagnostic tasks, GridWM-Judge tests local perception, action-conditioned transitions, presentation stability, representation exchangeability, and Full-CF causal discrimination. Our diagnostic evaluations reveal heterogeneous failure modes, including sensitivity to evidence presentation and encoding, low activation on successful trajectories, and weak paired counterfactual discrimination. Perception-controlled analyses further indicate that visual grounding and objective recovery are substantial, model-dependent bottlenecks. These results motivate jointly reporting correctness, presentation stability, and causal sensitivity instead of relying on a single score.
Introduction
Vision-language models (VLMs) are increasingly used not only as answer generators but also as automated judges. In text-only settings, LLM-based judges have been used to compare conversational responses and evaluate instruction-following systems [1, 2, 3]. In multimodal settings, this paradigm has expanded to rubric-based vision-language evaluation and in-the-wild preference assessment [4, 5, 6], as well as multimodal reward, critic, and agent evaluation [7, 8, 9]. Such judges are attractive because they provide scalable alternatives to rule-based metrics and expensive human annotations. However, once a VLM judge is used to decide whether an embodied trajectory succeeds or fails, reliability becomes more subtle than aggregate accuracy: a judge may be correct for the wrong reason, stable because it defaults to one verdict, or unstable under changes that should not affect the task outcome.
This raises a simple but fundamental question: when should a vision-language judge change its mind? Our answer is that a reliable judge should change its verdict when the underlying world changes, but not when only the presentation of the same evidence changes. For example, a successful trajectory should receive the same verdict under different layouts, explicitly indexed frame orderings, rendering styles, or prompt framing; conversely, a minimal intervention that changes the trajectory outcome from success to failure should change the verdict. We formalize this requirement as *State-Conditional
Invariance (SCI): given the same simulator-grounded trajectory semantics , a judge’s verdict should be conditionally independent of nuisance presentation variables ,
SCI is therefore a selective invariance principle: a reliable judge should be insensitive to how evidence is presented when trajectory semantics are preserved, but should change its verdict when the task outcome changes. We use presentation for temporal display order, visual appearance, and linguistic framing within a fixed input format; representation refers to evidence packaging, such as Composite versus Split images. Both are aspects of the evidence interface: C1 probes presentation invariance, whereas C3 tests representation exchangeability.
Existing multimodal judge benchmarks do not fully isolate this distinction. Many evaluate agreement with human preferences, rubric-following ability, reward-model quality, or arena-style rankings [5, 6, 7, 8]. These evaluations are valuable, but they often conflate two questions: whether a judge is robust to nuisance presentation changes, and whether it remains responsive to causal changes in the underlying task outcome. This distinction is especially important for embodied evaluation, where a counterfactual trajectory may look visually similar to a successful one while differing in a causally decisive event. In other words, existing benchmarks either evaluate judge agreement, preference alignment, or embodied reasoning ability, but rarely provide paired controls that jointly separate presentation invariance from causal sensitivity.
We introduce GridWM-Judge, a grid-world and world-model-inspired diagnostic benchmark for evaluating VLM judges under SCI. Built on deterministic grid-world environments, GridWM-Judge uses the simulator as a Single Source of Truth (SSOT) for states, actions, events, and success predicates. For each trajectory group, it constructs aligned FULL, evidence-replay, and counterfactual variants, and evaluates them through presentation probes and evidence representations. This design separates presentation variants, where the world is fixed but the evidence interface changes, from causal counterfactuals, where an intervention changes the task outcome.
Figure 1 summarizes the overall diagnostic pipeline. Our main contributions can be summarized as:
Figure 1. Overview of GridWM-Judge. The framework integrates simulator-based exam generation, VLM judging, and automatic scoring to assess four SCI diagnostic conditions (C1–C4). Ground-truth metadata are hidden from judges and used for scoring. The example illustrates that presentation consistency alone does not guarantee causal discrimination or reliable judging.
We formulate VLM judge reliability as State-Conditional Invariance, separating nuisance presentation changes from causal outcome changes.
We instantiate SCI in GridWM-Judge, an executable benchmark with aligned trajectory variants, counterfactual interventions, and presentation probes.
We characterize heterogeneous failure patterns, including presentation sensitivity, representation fragility, response bias, and weak causal discrimination, while using perception and action controls to qualify their interpretation.
Related Work
Model-based judges and multimodal evaluation. LLM-as-a-judge is now widely used for scalable evaluation of open-ended model outputs, from comparative or rubric-based text evaluation [1, 2, 3, 10, 11, 12] to multimodal judging, preference assessment, reward modeling, and critic or agent evaluation [4, 5, 6, 7, 8, 13, 14, 9]. These works establish the utility of model-based judges, but primarily study human preference alignment, rubric following, or reward quality. GridWM-Judge instead asks whether a VLM judge changes its verdict for the right reason under simulator-grounded presentation and causal controls.
Judge reliability, bias, and presentation effects. A growing literature studies the reliability of model judges themselves, including fairness concerns, selection bias, verbosity bias, rubric sensitivity, and inconsistency across prompts or candidates [15, 16, 17, 18]. In multimodal judging, VLM evaluators may be unreliable under benign changes in video evidence, judge composition, or input presentation [19, 20]. Separately, VLMs are known to suffer from visual hallucination, image-context reasoning failures, and object hallucination [21, 22, 23]. Long-context and image-sequence studies further show that models can depend on where and how evidence is presented [24, 25]. Recent work further identifies informativeness bias in VLM judges, where detailed responses can be preferred despite conflicting with visual evidence [26]. Relatedly, Li et al. [27] investigate unimodal spurious correlations in multimodal reward models. Together, these findings motivate examining how judges respond to changes in evidence presentation and underlying task semantics. GridWM-Judge instead uses simulator-grounded trajectories and controlled probes to diagnose presentation and representation sensitivity, response bias, and failures of causal discrimination.
Embodied evaluation, world models, and counterfactual diagnostics. Controlled environments such as BabyAI and MiniGrid support auditable grounded language and navigation experiments [28, 29], while video and embodied benchmarks such as CLEVRER and COVER evaluate causal or counterfactual reasoning over dynamic scenes [30, 31]. World-model research studies action-conditioned prediction, planning, and generative environment modeling [32, 33, 34, 35, 36, 37, 38, 39, 40]. Recent grid-world and VLM diagnostics ask whether models solve, plan, act, or form internal world-model-like representations [41, 42, 43, 44]; GridWM-Judge instead evaluates whether a VLM can reliably judge what happened in a trajectory under aligned presentation and causal counterfactuals.
State-Conditional Invariance
We formalize VLM judge reliability as State-Conditional Invariance (SCI). Our goal is not to infer whether a VLM has an internal world model, but to test whether its success/failure verdict behaves correctly under controlled changes to the world and to the presentation of evidence. A reliable judge should remain invariant under semantics-preserving presentation changes, while remaining sensitive to causal interventions that change the task outcome. This selective-invariance view is related to invariant prediction, causal invariance, counterfactual reasoning, and shortcut learning [45, 46, 47, 48], but we apply it to judge verdicts rather than to learned predictors, fairness criteria, or agent policies.
Setup and SCI Principle
Let denote a simulator-based trajectory group. Each group contains an aligned state trajectory, action sequence, rendered evidence, mission text, outcome label, and audit metadata. We write
where denotes simulator-grounded trajectory semantics, is the simulator success predicate, is the resulting success/failure label, is the trajectory variant, and is the presentation variable. In our benchmark, denotes a successful full trajectory, an evidence-removal diagnostic variant, and a counterfactual failure variant, respectively. For variant-specific rollouts, we use and . Full and CF share a group identifier and prescribed action sequence but may follow different state trajectories. The variable describes temporal order, visual appearance, and prompt framing within a fixed evidence representation. We use for the Composite/Split representation when comparing formats, suppressing this index elsewhere for clarity.
| Condition | Question | Instantiation | Metrics |
| C1: Presentation Invariance | Stable under semantics-preserving presentation changes? | Task C: temporal, visual, and framing probes | NS, JCR, LES |
| C2: Action-conditioned Dynamics | Do transition predictions depend on actions? | Task A: next-state prediction; action shuffle/mask | , , |
| C3: Representation Exchangeability | Consistent across Composite/Split interfaces? | Task C: Composite/Split comparison | |
| C4: Causal Sensitivity | Does the judge distinguish matched Full/CF outcomes? | Task C: Full/CF pairs; Task D: reference-aided Full/CF | , , |
Table 1. SCI diagnostic conditions and their benchmark instantiations. Tasks C and D use distinct evaluation protocols. Full metric definitions are provided in Appendix A.
SCI expresses nuisance invariance as conditional independence of the verdict and presentation, given the underlying trajectory semantics:
We assess this requirement by comparing verdicts across semantics-preserving presentations. Reliable judging additionally requires correct decisions on successful Full trajectories and their matched CF failures:
Thus, SCI is not ordinary robustness: it asks whether invariance is appropriate. A reliable judge should ignore nuisance presentation changes, but respond to state changes that alter the task outcome.
Diagnostic Conditions and Metrics
GridWM-Judge instantiates SCI through four diagnostic conditions. Table 1 summarizes the reliability question, benchmark instantiation, and main metrics. The tasks are not independent leaderboards; they diagnose complementary failure sources in the same judge.
We report four classes of metrics. Correctness anchors, such as and , measure whether the judge accepts successful trajectories and rejects counterfactual failures. Presentation and representation metrics, such as NS, JCR, LES, and , measure whether verdicts remain stable under semantics-preserving changes. Causal-sensitivity metrics, especially , , and , measure whether the judge discriminates successful full trajectories from matched counterfactual failures. Task-specific controls, including Task A/B/E metrics, help distinguish perception or action-conditioning failures from higher-level causal judgment failures.
We interpret consistency metrics only together with correctness and causal sensitivity. A model can be stable simply because it always predicts the same label. We therefore summarize failures as presentation leakage, representation fragility, action-ignorant heuristics, causal misgrounding, and stability traps; full definitions and degenerate-judge criteria are provided in Appendix A. Perception and output-format controls help interpret these diagnostic flags.
GridWM-Judge: Benchmark Construction
GridWM-Judge instantiates the SCI framework in executable grid-world tasks. The benchmark is designed around three requirements: (i) the simulator provides an auditable source of truth for states, actions, events, and outcomes; (ii) trajectory-level evaluations support controlled presentation changes and causal interventions; and (iii) the tasks diagnose complementary components of judge reliability rather than forming a single leaderboard. Figure 2 summarizes the six MiniGrid environments, aligned trajectory variants, task-specific dataset composition, and separately reported 3D extensions.
Environment Suite
Our original Tier-0 suite comprises six deterministic MiniGrid environment families [29]. They cover object interaction, ordered door opening, hazard navigation, memory, and multi-room navigation, while remaining compact enough to provide symbolic states, rendered observations, discrete actions, and unambiguous success predicates. MiniGrid is not intended to approximate open-ended embodied benchmarks [49]; instead, it lets us audit whether a VLM judge changes its verdict when the world changes or merely when the same evidence is presented differently. Detailed environment descriptions and specific dataset generation procedures are provided in Appendix § 10.
Figure 2. Benchmark anatomy and dataset composition. GridWM-Judge comprises six MiniGrid environment families and five diagnostic tasks. Simulator-generated Full, NoCue, and CF variants support controlled judging and presentation probes, while task-specific perception and transition tests provide complementary controls. The original benchmark contains 1,202 model-facing task instances; supplementary 3D extensions are evaluated separately.
Trajectory Groups and Variants
For trajectory-level judging tasks, the source unit is a trajectory group , which contains aligned simulator states, actions, rendered frames, mission text, outcome labels, and audit metadata. For each sampled environment seed, we first construct a successful FULL trajectory using a deterministic planner. We then derive matched variants when supported by the task: NoCue, which removes selected early evidence while preserving the success outcome, and CF, a counterfactual failure generated by a localized simulator intervention followed by rerollout under the same action sequence. Although the actions are fixed, simulator states and renderings may diverge after the intervention; Full and CF outcomes are independently verified by the simulator. Thus, NoCue probes evidence dependence without changing the success outcome, whereas CF tests responses to outcome-changing interventions. NoCue is not treated as a semantics-preserving presentation transform.
Local perception and transition-control tasks are generated from task-specific source units, such as environment-level states or local transitions, and are reported as model-facing task instances. This distinction lets GridWM-Judge combine trajectory-level Full/NoCue/CF diagnostics with local task A/B/E controls without treating all items as the same type of group. All generated items pass task-appropriate validation gates before entering the benchmark, including action-identity checks across aligned variants, FULL-success and CF-failure outcome gates, masking-window constraints, interaction preservation checks, physics consistency checks, and symbolic state-encoding checks when applicable. Full generation details and validator definitions are given in Appendix § 10.
Presentations and Diagnostic Tasks
GridWM-Judge distinguishes two aspects of the evidence interface. Temporal order, visual style, and linguistic framing define presentation probes (C1), while Composite and Split formats provide alternative representations for testing exchangeability (C3).
The released benchmark contains five diagnostic tasks. Tasks B and E are perception controls that test whether the model can parse local states, objects, and structured scene information before higher-level judgment failures are interpreted. Task C asks for success/failure judgments under presentation and representation changes. Task A tests action-conditioned transition understanding through next-observation prediction and action ablations. Task D is the primary causal-sensitivity test, comparing successful FULL trajectories with matched CF failures. Full task interfaces, prompt templates, schemas, and representative examples are provided in Appendix § 11.
Dataset Release and 3D Extension
The benchmark is released as machine-readable exam files with image paths, prompts, labels, and group metadata. Model-facing fields include only rendered evidence, mission text, action text, and task instructions; audit-only metadata, such as symbolic states, intervention records, labels, rewards, and validation logs, are reserved for auditing and scoring. Full schemas, data cards, metadata, and reproducibility instructions are provided in Appendix § 14.
Availability. The benchmark, including exam files, prompts, validation records, and scoring resources, is available through the code repository and dataset release. Implementation details, model configurations, data schemas, and reproducibility instructions are documented in Appendix § 14.
Diagnostic Experiments
We evaluate model-based judges against the four SCI diagnostic conditions, focusing on causal discrimination, presentation stability, and representation exchangeability.
Experimental Setup
Models, tasks, and representations. We evaluate a diverse set of proprietary and open VLM judges, including frontier general-purpose models, lightweight flash-style models, and representative open VLM families when complete runs are available [50, 51, 52]. Each model acts as an external judge rather than an agent. We evaluate the five MiniGrid tasks described in § 4, with Tasks B/E providing perception controls, Task A testing action-conditioned transitions, Task C examining presentation and representation effects, and Task D evaluating causal discrimination. The key representation comparison is between composite storyboards and split-image inputs, which encode the same trajectory semantics but expose evidence through different interfaces.
Inference, metrics, and statistics. All models receive task-specific instructions with the same model-facing fields: rendered evidence, mission text when available, action text, and the required answer format, following standard instruction-following and judge-prompting protocols [53, 1, 2]. We use deterministic decoding when supported. Responses are parsed using task-specific scoring rules; invalid outputs and execution failures are tracked separately. The main Task-C summaries use the released eligible-pair protocol, whereas dedicated Task-D and expanded evaluations report conservative all-group estimates. The main Task D metrics are , , , , and ; Task C additionally reports presentation and representation metrics such as NS, JCR, LES, and VCC. We interpret consistency metrics only together with correctness and causal sensitivity, because a constant-verdict judge can be stable without being reliable. Full model identifiers, prompts, parsing rules and other experimental settings are provided in Appendix § 14.
Analysis
Table 2 and Figure 3 summarize Task-C diagnostics of activation, paired discrimination, and representation sensitivity. We use (, IDR) as the primary causal-sensitivity coordinates because they jointly indicate whether a judge accepts genuine successful trajectories and rejects matched counterfactual failures. Dedicated reference-aided Task-D results with clustered confidence intervals and other expanded evaluations are reported in Appendix § 12.
| Model | C/S | C | S | C | S | Deg. (C/S) | |
| Claude Sonnet 4.6† | 0.06/0.11 | 0.06 | 0.12 | 0.06 | 0.12 | 0.924 | AF/— |
| GPT-4o | 0.28/0.78 | 0.28 | 0.78 | 0.22 | 0.72 | 0.706 | —/— |
| GPT-5.4 | 0.06/0.06 | 0.06 | 0.06 | 0.06 | 0.06 | 1.000 | AF/AF |
| Gemini 3 Flash | 0.72/0.67 | 0.72 | 0.67 | 0.67 | 0.67 | 0.707 | —/— |
| Gemini 2.5 Flash Lite | 0.50/0.67 | 0.50 | 0.67 | 0.28 | 0.56 | 0.796 | —/— |
| Gemini 2.0 Flash† | 0.00/0.61 | 0.00 | 0.59 | 0.00 | 0.53 | 0.544 | AF/— |
| Kimi K2.5 | 0.39/0.72 | 0.50 | 0.72 | 0.36 | 0.72 | 0.661 | —/— |
| Qwen3-32B‡ | 1.00/0.67 | 1.00 | 0.67 | 0.00 | 0.00 | 0.667 | SB/— |
| Qwen3-235B | 0.67/0.67 | 0.67 | 0.67 | 0.00 | 0.00 | 0.743 | —/— |
| Qwen3-8B | 0.67/0.00 | 0.67 | 0.00 | 0.00 | 0.00 | 0.407 | —/AF |
Table 2. Task-C diagnostics under Composite (C) and Split (S) evidence interfaces. Deg. denotes the degenerate status under the corresponding representation; AF denotes all-fail degeneracy, SB denotes success bias, and — denotes non-degenerate behavior. † Composite is all-fail degenerate, while Split is non-degenerate. ‡ Success-biased degenerate under Composite.
Figure 3. Task-C diagnostic results. (a) plots Full activation against raw paired Full–CF discrimination under Composite and Split interfaces. (b) shows the corresponding representation shifts in Full-trajectory accuracy and discrimination. Together, the panels illustrate that high acceptance, paired discrimination, and representation consistency need not coincide.
Finding 1: Accuracy does not imply causal sensitivity. Task-C paired diagnostics show that Full acceptance and counterfactual rejection capture different aspects of judge behavior. A judge that predicts failure almost everywhere may achieve high CF rejection but low activation, while a success-biased judge may accept Full trajectories without rejecting their matched CFs. Dedicated reference-aided Task D confirms this distinction: GPT-5.4 achieves Full acceptance of 0.750, CF rejection of 0.500, and raw IDR of 0.333, whereas Claude Sonnet 4.6 obtains 0.250, 0.917, and 0.250, respectively (Table 13 in Appendix § 12). On 432 expanded Task-D groups, their raw IDRs are 0.322 and 0.088, respectively, with group-bootstrap 95% CIs of [0.278, 0.366] and [0.062, 0.116] (Table 22 in Appendix § 12). Thus, we assess causal sensitivity jointly through (, ), with conditional IDR interpreted only when activation is sufficient.
| Model | IDR | Group 95% CI | Env. 95% CI | |
| Claude Sonnet 4.6 | 0.250 | 0.250 | [0.000, 0.500] | [0.000, 0.583] |
| Gemini 2.0 Flash | 0.667 | 0.583 | [0.333, 0.833] | [0.250, 0.917] |
| Gemini 2.5 Flash Lite | 0.750 | 0.167 | [0.000, 0.417] | [0.000, 0.333] |
| Gemini 3 Flash | 0.833 | 0.167 | [0.000, 0.417] | [0.000, 0.500] |
| GPT-4o | 0.750 | 0.500 | [0.250, 0.750] | [0.167, 0.833] |
| GPT-5.4 | 0.750 | 0.333 | [0.083, 0.583] | [0.083, 0.667] |
| Kimi K2.5 | 0.167 | 0.000 | [0.000, 0.000] | [0.000, 0.000] |
| Qwen3-235B | 1.000 | 0.000 | [0.000, 0.000] | [0.000, 0.000] |
| Qwen3-32B | 0.000 | 0.000 | [0.000, 0.000] | [0.000, 0.000] |
| Qwen3-8B | 1.000 | 0.000 | [0.000, 0.000] | [0.000, 0.000] |
Table 13. Reference-aided Task-D discrimination (strict/all-groups). Strict all-group estimates over 12 matched Full/CF Query groups per model. Confidence intervals are reported for raw IDR using group-level and environment-level bootstrap resampling.
| Task | Model | Groups | CF rej. | Group 95% CI | Env. 95% CI | ||
| C | GPT-5.4 | 442 | 0.523 | 0.629 | 0.314 | [0.271,0.357] | [0.249,0.372] |
| C | Claude Sonnet 4.6 | 442 | 0.043 | 0.993 | 0.043 | [0.025,0.063] | [0.009,0.107] |
| D | GPT-5.4 | 432 | 0.720 | 0.530 | 0.322 | [0.278,0.366] | [0.226,0.461] |
| D | Claude Sonnet 4.6 | 432 | 0.113 | 0.933 | 0.088 | [0.062,0.116] | [0.037,0.189] |
| D | Gemini 3 Flash | 432 | 0.972 | 0.637 | 0.613 | [0.567,0.660] | [0.335,0.767] |
| D | Qwen3-32B | 432 | 1.000 | 0.000 | 0.000 | [0.000,0.000] | [0.000,0.000] |
Table 22. Scale-2 paired diagnostics. C and D use different judging contracts. Only evaluated model-task combinations are shown. Confidence intervals apply to raw IDR, with intact trajectory-group and environment-configuration resampling reported separately.
Finding 2: Representation changes judge behavior. Task-C presentation probes reveal varying stability across temporal, visual, and framing changes (Figure 4). Composite and Split interfaces also produce different verdicts for matched trajectories, demonstrating sensitivity to evidence presentation and packaging. Although Split yields a higher mean raw IDR in Table 3, the across-model difference is not statistically significant (). Moreover, the compared interfaces also differed in prompt wording and run conditions, so the effect cannot be attributed solely to image layout. We use to quantify representation consistency and report matched transition analyses in Appendix § 12. Matched temporal, noise, and style probes show both correctness degradation and rescue, although none of the 180 probe-level comparisons survives the specified Holm correction.
Figure 4. Presentation stability across probe families. (a) compares NS and JCR across temporal, visual, and framing probes for each model. (b) summarizes the corresponding JCR distributions.
| Metric | Composite Mean | Split Mean | Mean | Median | W/T/L | -value |
| 0.433 | 0.494 | +0.061 | +0.028 | 5/2/3 | 0.617 | |
| 0.444 | 0.493 | +0.048 | +0.031 | 5/2/3 | 0.641 | |
| 0.163 | 0.337 | +0.173 | +0.031 | 5/5/0 | 0.063 | |
| 0.624 | 0.720 | +0.096 | +0.038 | 4/4/0 | 0.125 |
Table 3. Composite–Split representation ablation. Reported values are model-wise means, with = Split − Composite. W/T/L denotes improvements, ties, and degradations across models; the last column reports Wilcoxon signed-rank -values. All comparisons are not significant at .
*For , models with undefined or low-activation conditional IDR are excluded from W/T/L counting according to the reporting rule in Appendix § 9.
Finding 3: Stable judgments can mask failure; local controls reveal additional bottlenecks. High NS or JCR does not imply reliability: a model can remain stable while failing to discriminate matched Full/CF outcomes because it defaults to a constant verdict or shortcut rule [48]. Figure 5 and Table 4 provide complementary Task-B/E perception and Task-A transition controls. For example, GPT-5.4 achieves Task-B [email protected] of 0.111 despite an all-row mean component score of 0.616, showing that high-threshold accuracy can obscure partial scene recovery. Task-A ablation effects are generally small or mixed, providing limited evidence about action-conditioned reasoning on their own. In a matched 98-group Oracle-State experiment, GPT-5.4’s raw IDR increases from 0.469 for Pixel evidence to 0.765 with State evidence, highlighting visual grounding and objective recovery as potential bottlenecks. Together, these controls help distinguish plausible failure sources without uniquely identifying their causes (Appendix § 12).
Figure 5. Local controls and judgment outcomes. (a) compares Task-B/E perception scores, Task-A transition accuracy and action-ablation effects, and Task-C judging metrics across models. (b) contrasts Task-C Full acceptance and raw intervention discrimination. These complementary diagnostics help interpret, but do not uniquely identify, failure sources.
| Model | B_Acc | E_Acc | Acc_A1 | ||
| Claude Sonnet 4.6 | 0.000 | 0.477 | 0.252 | +0.039 | +0.068 |
| GPT-4o | 0.000 | 0.495 | 0.214 | -0.039 | -0.068 |
| GPT-5.4 | 0.111 | 0.615 | 0.252 | +0.000 | -0.019 |
| Gemini 2.0 Flash | 0.007 | 0.274 | 0.175 | -0.038 | -0.025 |
| Gemini 2.5 Flash Lite | 0.000 | 0.345 | 0.204 | -0.039 | -0.029 |
| Gemini 3 Flash | 0.090 | 0.702 | 0.282 | +0.019 | +0.010 |
| Kimi K2.5 | 0.007 | 0.280 | 0.282 | +0.039 | +0.039 |
| Qwen3-235B | 0.000 | 0.305 | 0.262 | -0.039 | +0.029 |
| Qwen3-32B | 0.000 | 0.357 | 0.252 | +0.000 | +0.000 |
| Qwen3-8B | 0.000 | 0.311 | 0.282 | +0.029 | +0.029 |
Table 4. Local diagnostic controls. Task B reports [email protected], the fraction of items meeting a 0.90 threshold on the normalized weighted scene-component score; Task E measures object/state recognition, and Task A measures next-observation prediction. Action-ablation differences are computed as baseline accuracy minus ablated accuracy.
Summary. Across diagnostic tasks, correctness, stability, and causal discrimination reveal complementary failure modes. Expanded MiniGrid and 3D evaluations further support interpreting judge behavior through these dimensions rather than a single aggregate score. (Appendices § 12 and § 13).
Conclusion
We introduced GridWM-Judge, a simulator-grounded benchmark for auditing model-based judges through State-Conditional Invariance (SCI). By combining matched trajectory variants, controlled evidence-interface probes, and action/perception controls, the benchmark distinguishes appropriate verdict stability from responsiveness to outcome-changing interventions. Across evaluated judges, apparent consistency and successful-trajectory acceptance do not guarantee reliable paired causal discrimination; additional controls reveal model-dependent grounding and response-format bottlenecks. Expanded MiniGrid evaluations and supplementary MiniWorld/Habitat experiments extend these diagnostics to more 2D and 3D settings. Our findings motivate evaluating judge reliability through correctness, presentation stability, and causal sensitivity jointly, rather than relying on a single score.
Acknowledgments and Disclosure of Funding
Funding. This work was supported in part by the Anhui Provincial Innovation Breakthrough Plan (Grant No. 202523j08050018), the Anhui Innovation Platform Construction – High-Level New R&D Institution Project (Grant No. 202524vb12010005), the Centrally Guided Local Science and Technology Development Special Fund of Anhui Province (Grant No. 202407a12020009), and the National Key R&D Program of China (Grant No. 2022YFB3206904).
Competing Interests. The authors declare no competing interests.
[46] Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society Series B: Statistical Methodology, 78(5):947–1012, 2016.
[47] Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. Advances in neural information processing systems, 30, 2017.
[48] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020.
[49] Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. MineDojo: Building open-ended embodied agents with internet-scale knowledge. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 18343–18362, 2022.
[50] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. URL https://arxiv.org/abs/2308.12966.
[51] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 34892–34916, 2023.
[52] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. CogVLM: Visual expert for pretrained language models. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pages 121475–121499, 2024.
[53] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Advances in neural information processing systems (NeurIPS), volume 35, pages 27730–27744, 2022.
Appendix Overview
This appendix provides additional details supporting the main paper. Appendix A defines SCI conditions, metrics, statistical procedures, and failure modes. Appendix B describes MiniGrid environments, dataset construction, validation, and composition. Appendix C presents task interfaces, prompts, and representative examples. Appendix D reports supplementary MiniGrid results, expanded evaluations, ablations, and failure analyses. Appendix E presents MiniWorld and Habitat 3D evaluations. Appendix F documents implementation, data release, and reproducibility protocols. Appendix G discusses limitations, societal impact, and future directions.
Formal Definitions and Metrics
This appendix expands the State-Conditional Invariance (SCI) framework summarized in § 3. We define the simulator-grounded trajectory semantics, presentation variables, trajectory variants, and representation formats used by GridWM-Judge; formalize the four diagnostic conditions C1–C4; and provide metric and statistical definitions for the results reported in the main text. We also define the failure modes and degenerate-judge criteria used throughout the analysis, including presentation leakage, representation fragility, action-ignorant heuristics, causal misgrounding, and stability traps.
State and Presentation Variables
Trajectory groups. Let denote a trajectory group generated from the simulator. Each group contains an aligned action sequence, state trajectory, rendered evidence, mission or task text, outcome labels, and audit metadata. We write
for the simulator-grounded trajectory semantics, where is the symbolic simulator-state sequence, is the fixed action sequence, and is the mission text when provided. The binary outcome label is computed by the simulator success predicate,
By convention, denotes success and denotes failure. The rendered images shown to a model are generated from simulator states but are not treated as privileged state in the model-facing prompt.
Trajectory variants. For variant-specific simulator rollouts, we write and . For each sampled group, GridWM-Judge constructs aligned variants when supported by the corresponding task. The FULL variant is a successful rollout, so . The NOCUE variant removes selected early visual evidence while preserving the success outcome, ; it is an evidence-removal diagnostic, not a semantics-preserving presentation transform. The CF variant is a counterfactual failure generated by a minimal simulator intervention followed by rerollout under the unchanged action sequence, with . Full and NoCue share the successful outcome but not necessarily the same rendered evidence; CF can follow a different state trajectory after the declared intervention while retaining the prescribed action sequence.
Presentations and representations. A VLM judge receives a presentation of trajectory evidence and produces a binary verdict
where denotes the trajectory variant. The presentation variable describes temporal display order, visual appearance, and linguistic framing within a fixed evidence representation. We could denote the canonical presentation by
We use to denote the Composite/Split evidence format when testing representation exchangeability. Oracle-State and Pixel+State inputs are treated separately as evidence conditions that alter the information available to the judge, rather than equivalent representations for C3.
State-Conditional Invariance. The core reliability principle is State-Conditional Invariance (SCI): conditioned on the same underlying trajectory semantics , the judge’s verdict should be independent of nuisance presentation variables ,
As an operational sample-level check, for a semantics-preserving presentation transform , a reliable judge should satisfy
SCI is therefore not ordinary robustness. It does not require the verdict to be unchanged under all input changes. Instead, it asks whether invariance is appropriate: the verdict should remain stable under presentation changes that preserve , but should change under causal interventions that alter the outcome.
Four Diagnostic Conditions
We decompose judge reliability into four diagnostic conditions, each instantiated by one or more tasks in GridWM-Judge.
C1: Presentation Invariance. For the same underlying trajectory semantics , the judge should be invariant to nuisance presentation transforms:
In GridWM-Judge, includes temporal presentation transforms (e.g., original versus reversed display order), visual transforms (e.g., clean, noisy, or style-adjusted renderings), and prompt-framing transforms (e.g., neutral, positive, or negative preambles). These transforms are designed to preserve the trajectory semantics and the success label. C1 is mainly tested by Task C through the NS, JCR, Flip, and LES metrics defined below. In the main analysis, C1 is evaluated within a fixed trajectory condition; Full–CF differences are never counted as presentation invariance because they change the underlying outcome.
C2: Action-Conditioned Dynamics. Judge-relevant reasoning should depend on action-conditioned state transitions, rather than on terminal-frame heuristics or superficial visual matching. This condition is conceptually aligned with action-conditioned world-model learning and model-based control [32, 33, 34, 35]. Let denote the implicit transition model needed to predict future states. A minimal requirement is
Task A tests the one-step case by asking the model to select the correct next observation from four candidates given the current observation and action . Action-shuffling and action-masking ablations provide behavioral evidence of whether predictions depend on the supplied action.
C3: Representation Exchangeability. The same trajectory semantics may be presented through different but equivalent evidence formats, such as a composite storyboard or split-image representation. This concern is related to evidence-ordering and image-sequence reasoning effects observed in long-context and multimodal settings [24, 25]. A reliable judge should not be bound to a single interface:
whenever these representations encode the same trajectory semantics. In the main experiments, C3 is tested primarily by comparing composite and split-image presentations; Structured-state and Pixel+State inputs are instead used as perception-controlled evidence conditions: they alter the information available to the judge and are not treated as purely exchangeable visual layouts.
C4: Causal Sensitivity. A judge should be insensitive to nuisance transforms but sensitive to causal interventions. For each group, the counterfactual variant changes the simulator state or initial condition so that the same action sequence leads to failure:
A reliable judge should therefore satisfy
C4 is mainly tested by Task D, which directly compares full successful trajectories with counterfactual failures. Task C also contains paired Full/CF diagnostic rows; however, its Composite/Split protocol is distinct from the reference-aided Task-D contract.
Task mapping and NoCue. The tasks are not independent leaderboards; they diagnose different components of SCI. Task C tests presentation invariance and representation sensitivity (C1/C3), Task D tests causal sensitivity (C4), Task A tests action-conditioned transition understanding (C2), and Tasks B/E provide perception and state recognition controls. The NoCue variant probes whether the judge depends on early visual cues, but it is not counted as either a semantics-preserving presentation transform for C1 or a causal intervention for C4.
Metrics Formulation
We define all metrics as empirical estimates over their applicable item sets. Let be the indicator function. Unless otherwise stated, denotes the number of trajectory groups in the relevant task, variant set, and representation condition.
Correctness anchors. The main correctness anchor is the neutral full-trajectory judgment accuracy:
where all full trajectories are successful by construction. We also report variant-wise accuracies:
and
Here is diagnostic because the NoCue trajectory preserves the success outcome while removing selected early evidence. We also track the success prediction rate
over the relevant binary judgment set , as a label-bias indicator.
Presentation invariance: NS and JCR. For a probe family , let be the set of non-baseline presentations in that family for group , and let be the family-specific baseline. The Nuisance Stability score is
where . The baseline itself is not counted as a trivial self-comparison.
We also compute the Judge Consistency Rate (JCR). Let contain all probe instances in family , including the family baseline when applicable. Then
The corresponding flip rate is
High NS or JCR is not sufficient for reliability, because a constant-verdict judge can be perfectly stable.
Presentation leakage: LES. To measure whether a presentation family systematically shifts verdicts, we estimate a leakage effect size. For each family , we fit a mixed-effects logistic model over pooled binary predictions:
where is a group-level random intercept and the dummy variables encode the presentation identity within family . We define
We also report the odds ratio and the standardized effect size
A large and statistically significant indicates presentation leakage: the verdict depends systematically on how the same trajectory evidence is presented.
Representation exchangeability. For two equivalent representations and , such as composite and split storyboards, we define the Verdict Consistency Coefficient as
where is the set of groups evaluated under both representations. When all groups are paired, this reduces to an average over matched groups. In the main tables, this is reported as for the composite–split comparison. We can further condition this metric on the baseline representation’s predicted verdict:
and analogously for . When oracle-state evidence is available, we additionally measure verdict agreement between oracle-state and pixel conditions as .
Unlike , this is an evidence-condition diagnostic rather than a pure C3 exchangeability measure, because the available information differs between conditions.
Causal sensitivity: IDR. C4 is measured by whether the judge discriminates successful full trajectories from their counterfactual failures. Define
The raw Intervention Discrimination Rate is
The number of active groups is
Since a model that almost never predicts success on full trajectories cannot meaningfully be tested for full-to-counterfactual discrimination, we also report the conditional IDR:
Equivalently,
We additionally track the reverse error
which should be close to zero for a sensible judge.
Task-specific eligibility. For the Task-C summaries in Table 2, is computed over all 18 Full groups per representation, treating unparseable Full verdicts as incorrect. In contrast, and use the subset of groups with parseable verdicts for both Full and CF. For example, Kimi K2.5 under Composite yields 7/18 for JAcc, 7/14 for activation, and 5/14 for raw IDR. Dedicated Task-D and Scale-2 analyses instead use all complete groups, counting invalid outputs as incorrect.
Task-specific diagnostic metrics. Task A reports one-step transition accuracy:
where is the correct next-observation choice among four candidates. When multi-step variants are available, we analogously report for -step prediction. To test whether the model uses the action condition, we compute action-ablation drops:
If these drops are small and statistically insignificant, it provides evidence that the model is more likely to rely on an action-ignorant heuristic for this diagnostic.
Task B measures structured perception from a single frame. Let be the component-level score for item under the canonical schema. We report
Here is the normalized weighted component score, with invalid or unavailable outputs assigned zero. [email protected] is the high-threshold component accuracy, not JSON exact-match accuracy. Task E analogously reports object/state classification accuracy and mean score:
These perception-oriented diagnostics help separate failures caused by visual parsing or interface mismatch from failures in causal judgment.
Metric summary. Table 5 lists the metrics used in the paper and the primary diagnostic question each addresses.
| Metric | Formula sketch | Task(s) | Interpretation |
| C/D | Neutral full-trajectory correctness anchor | ||
| C/D | Variant-wise full success acceptance | ||
| C/D | NoCue success accuracy; evidence-removal diagnostic | ||
| D; C if CF probes are included | Counterfactual-failure rejection accuracy | ||
| C/D | Overall success-verdict rate; label-bias indicator | ||
| C/D | Rate at which the judge accepts genuine full successes | ||
| C/D | Joint Full–CF discrimination without conditioning away inactivity | ||
| C/D | CF rejection conditional on accepting the matched full trajectory | ||
| D | Reverse Full–CF ordering error | ||
| Pairwise agreement with family baseline | C | Stability under a presentation family | |
| All-presentations-same indicator averaged over groups | C | Per-group consistency across all probes in | |
| C | Presentation-induced verdict flips | ||
| from logistic model | C | Systematic presentation leakage effect size | |
| C | Standardized logit-scale leakage effect size | ||
| C | Composite–split representation exchangeability | ||
| Optional C/D | Agreement between oracle state-table and visual evidence | ||
| A | One-step action-conditioned transition accuracy | ||
| Analogous -step choice accuracy | Optional A | Multi-step transition accuracy when available | |
| Accuracy drops from original | Task A | Whether action text affects next-state prediction | |
| B_MeanScore, B_Acc | Mean weighted component score; accuracy at threshold 0.90 | B | Structured scene-perception control |
| E_Acc, E_MeanScore | Exact-match accuracy; mean component score | E | Object/state tracking control |
Table 5. Summary of SCI metrics. All metrics are empirical estimates over their applicable task-specific item sets.
Failure-Mode Taxonomy and Degenerate-Judge Criteria
The metrics above are intended for diagnosis rather than for constructing a single leaderboard. We therefore interpret each metric through a failure-mode taxonomy. A model may exhibit multiple failure modes at once, and degenerate judges are flagged before drawing conclusions from stability or intervention metrics.
Presentation leakage (C1). The verdict changes under semantics-preserving temporal, visual, or framing transforms. This is reflected by low or , high , or a non-negligible leakage effect size . It indicates that the judge is sensitive to how the same trajectory evidence is presented.
Representation fragility (C3). Equivalent evidence formats, such as composite storyboards and split-image sequences, yield inconsistent verdicts. This is reflected by low or, when available, low . It indicates that the judge is bound to a particular evidence interface rather than to the underlying trajectory semantics.
Action-ignorant heuristic (C2). Task-A performance does not decrease after action shuffling or action masking. This is reflected by small or despite nontrivial . It suggests that the model relies on visual matching or terminal-frame shortcuts rather than action-conditioned transition reasoning.
Causal misgrounding (C4). The model fails to distinguish a successful FULL trajectory from its matched counterfactual failure. This is reflected by low or low/unstable after checking and . It indicates that the judge does not reliably respond to an intervention that changes the simulator outcome.
Stability trap (cross-cutting). The model has high consistency under presentation probes but low correctness or low causal sensitivity. In this regime, high NS or JCR reflects a constant verdict or shortcut decision rule rather than reliable judging.
We additionally flag degenerate judges before interpreting SCI metrics:
All-fail degenerate: . The model rarely accepts genuine successful full trajectories, so high or high stability is not evidence of reliability.
Success-biased degenerate: and (equivalently, ). The model accepts both full and CF trajectories, so high full-trajectory accuracy does not imply causal sensitivity.
Low-activation unstable: . may be reported descriptively, but it should not be treated as strong evidence because the active set is too small.
These labels are interpretive diagnostics, not additional tasks. They are used to connect the four SCI conditions to the qualitative failure cases reported in Appendix D.
Statistical Testing and Uncertainty Estimation
Bootstrap confidence intervals. For reported proportions and paired metrics, including , , , , , NS, JCR, and VCC, we use percentile bootstrap intervals. Because multiple items may share the same underlying trajectory group, the default procedure is a clustered bootstrap: resample trajectory groups with replacement, include all items associated with the resampled groups, recompute the metric, and report the 2.5th and 97.5th percentiles over the bootstrap estimates. This preserves within-group dependence induced by shared states, actions, and variants.
Paired significance tests. For paired binary comparisons, such as composite versus split predictions on the same group or baseline versus ablated Task-A predictions on the same item, we use McNemar’s exact test when the number of discordant pairs is small and the continuity-corrected McNemar test otherwise. If and are the two discordant counts, the large-sample statistic is
Across-model Composite/Split comparisons in Table 3 use the Wilcoxon signed-rank test on paired model-level summary metrics. Item-level paired binary comparisons are analyzed separately using McNemar tests.
Low-activation regimes. can be misleading when the active set is small. We therefore apply the following reporting rule:
If , is undefined.
If , report only as a descriptive value, mark it as unstable, and do not use it as evidence of causal sensitivity.
If , report with a clustered bootstrap confidence interval, but still interpret it together with and .
In all cases, the main C4 diagnostic plot should show both and .
Degenerate-judge criteria. Degenerate-judge criteria are defined in Section A.4. In the statistical workflow below, these criteria are applied before interpreting conditional causal-sensitivity metrics or high stability scores.
Minimal workflow. The statistical workflow used for C4-style diagnostics is:
compute , , , , and ;
bootstrap group-level confidence intervals for the reported rates;
flag low-activation or degenerate regimes before interpreting ;
use paired tests for representation or ablation comparisons;
interpret stability metrics only jointly with correctness and causal sensitivity.
Environment Details and Dataset Generation
This appendix expands § 4 by documenting the MiniGrid Tier-0 environments, the construction of aligned trajectory variants, the validation gates used before exam generation, and the dataset composition. The purpose of this appendix is to make the simulator-grounded source of truth inspectable: for each sampled seed we retain the symbolic state sequence, action sequence, rendered observations, mission text, outcome labels, and audit metadata, while exposing only the model-facing evidence to the VLM judge. Full JSON/JSONL schemas, repository paths, decoding parameters, and response parsing rules are provided separately in Appendix F.
MiniGrid Environment Cards
GridWM-Judge uses six deterministic MiniGrid environments as the 2D Tier-0 suite. They cover object interaction, ordered door opening, hazard navigation, memory-dependent choice, and multi-room traversal. Each environment provides a compact symbolic state, an agent-centric rendered observation, a discrete action space, and an unambiguous success predicate. Table 6 summarizes the environment-level causal structure used by the benchmark.
| Environment | Causal chain and success condition | NoCue evidence removal | CF intervention |
| DoorKey | Pick up key → unlock/open matching door → reach goal. Success requires positive reward and terminal goal entry. | Mask early frames in which the key is visible before pickup. | Relocate or remove the goal at so that the unchanged action sequence no longer reaches the true goal. |
| KeyCorridor | Pick up color-matched key → return to locked door → unlock door → reach the mission target object. | Mask the door-matched key in the pre-pickup window. | Remove the mission target object before rollout, so the same action sequence cannot satisfy the pickup objective. |
| RedBlueDoor | Open red door → open blue door → pass through the ordered-door sequence. Opening blue before red is invalid. | Mask the red door before the first red-door interaction; blue-door evidence is retained. | Replace or block the blue-door passage before rollout, preserving the early red-door subtask but preventing final success. |
| LavaGap | Identify the single safe gap in the lava barrier → cross through the gap → reach goal without stepping on lava. | Mask visible lava evidence in early pre-crossing frames. | Relocate or remove the goal outside the planned path while keeping the safe-gap crossing behavior physically valid. |
| Memory | Observe cue object → traverse corridor → choose the matching terminal object. | Mask the cue object before corridor entry. | Swap the correct and distractor terminal objects before rollout, so the same remembered choice becomes wrong. |
| MultiRoom | Traverse a sequence of rooms, opening intervening doors, then enter the goal room and reach the goal. | Mask early door/goal evidence before entering the goal-containing room. | Remove the final goal object before rollout, preserving room traversal but preventing terminal success. |
Table 6. MiniGrid environment cards. The table lists the causal evidence that the judge should use, the evidence removed in NoCue, and the minimal counterfactual intervention used to create matched failures.
Figure 6 visualizes representative seeds and causal chains for the six MiniGrid environments summarized in Table 6. For each environment, object identities and causal events are inferred from the simulator state rather than from rendered pixels alone. DoorKey records whether the agent is carrying the key and the door state; KeyCorridor resolves the mission target and the color-matched key/door pair; RedBlueDoor checks the red-before-blue ordering; LavaGap identifies the barrier and the unique safe gap; Memory identifies the cue object and the matching terminal object; and MultiRoom tracks door toggles, room transitions, and entry into the goal room. These symbolic event traces are used only for generation and validation, not as model-facing inputs.
Figure 6. Representative MiniGrid environment cards. Each panel shows one canonical seed, the mission text, and a short causal-chain annotation.
Dataset Generation Pipeline
The atomic unit for Tasks C and D is a trajectory group , containing a simulator state sequence, action sequence, mission text, outcome label, and audit metadata. A trajectory group contains a successful Full rollout, an evidence-removal NoCue variant, and a counterfactual CF variant. Tasks A, B, and E are derived from the same validated simulator rollouts but expose more localized transition, perception, or prefix-checkpoint interfaces, as described in Appendix § 11. Figure 7 shows the full generation pipeline from environment seed to validated model-facing exam items.
Figure 7. Dataset generation pipeline from environment seed to validated exam item. Each seed first produces a successful Full rollout; the aligned NoCue and CF variants are then constructed, validated, and converted into task-specific exam items. The model-facing prompts contain only rendered evidence and task instructions, while symbolic states, labels, and intervention metadata remain audit-only.
Stage 1: successful full rollout. For each environment seed, we reset the simulator with a deterministic layout and use an environment-specific shortest-path planner to construct a successful rollout. The planner operates over symbolic states and inserts required interaction actions, such as pickup and toggle. Rollouts with truncation, missing critical events, or ambiguous terminal outcomes are rejected before any exam item is produced.
Stage 2: evidence-removal variant. The NoCue variant keeps the same action sequence and the same simulator outcome as Full, but masks selected early rendered tiles that reveal the causal evidence. The masked frames must occur strictly before the first interaction with the target evidence. Thus NoCue is an outcome-preserving evidence-removal diagnostic: it tests whether the judge relies on early causal evidence, but it is not counted as a semantics-preserving presentation transform for C1 or as a causal intervention for C4. Table 7 lists the environment-specific evidence targets, mask budgets, and alignment thresholds used for NoCue construction. Figure 8 illustrates a representative NoCue audit, showing how the target evidence is masked while preserving the rollout outcome.
| Environment | Masked target evidence | Max masked frames | Alignment threshold |
| DoorKey | Key before pickup | 5 | 0.70 |
| KeyCorridor | Color-matched key before pickup | 5 | 0.70 |
| RedBlueDoor | Red door before first red-door toggle | 5 | 0.70 |
| LavaGap | Lava evidence before safe crossing | 4 | 0.70 |
| Memory | Cue object before corridor entry | 10 | 0.70 |
| MultiRoom | Door/goal evidence before goal-room entry | 12 | 0.70 |
Table 7. Per-environment NoCue targets and masking constraints. The mask budget is intentionally conservative so that masking removes the target evidence without changing the overall trajectory presentation.

Figure 8. Example NoCue audit. The panels show the original frame, the target-tile mask overlay, and the resulting NoCue masked frame used to remove selected early evidence while preserving the rollout outcome.
Stage 3: counterfactual failure variant. The CF variant applies a minimal simulator intervention before action execution and then rerolls the unchanged action sequence. Depending on the environ-
ment, the intervention removes, relocates, swaps, or blocks the causal target. The validation contract requires the counterfactual rollout to be a physically valid execution of the same action sequence and to fail for the intended causal reason. Exact positional identity with Full is required for NoCue; for CF, state divergence is allowed only when it is induced by the declared intervention under the unchanged actions.
Stage 4: rendering and exam assembly. Validated rollouts are rendered into the presentation formats used by the task interfaces. Task C uses event-anchored sparse storyboards and presentation probes such as temporal order and visual style changes. Task D uses matched Full/CF trajectory evidence for causal-sensitivity tests. Task E uses prefix-only checkpoint evidence and deliberately hides terminal frames. The detailed prompt templates, split-image/composite layouts, and example questions are provided in Appendix § 11.
Validator Gates and Quality Control
All generated groups pass validation before they enter the benchmark. The validators prevent data-generation artifacts from being mistaken for VLM-judge failures. Table 8 summarizes the gates used in the release pipeline.
| Gate | Purpose | Rule | Scope |
| D1 | Action identity | The action IDs are identical across matched variants in the same group. | all variants |
| D2 | Length consistency | Frames and states align with the action sequence: . | all variants |
| D3 | Input isolation | Model-facing inputs contain only rendered evidence, mission/action text, and task instructions; labels, rewards, states, and intervention metadata remain audit-only. | all tasks |
| D4 | Outcome contract | Full succeeds, NoCue preserves the Full outcome, and CF fails without truncation artifacts. | C/D |
| D5 | No interaction masking | NoCue never masks the frame in which the agent is facing or interacting with the target evidence. | NoCue |
| D6 | Physics validity | NoCue preserves the exact state trajectory; CF is a valid rerollout under the same action sequence, with divergence allowed only as a consequence of the declared intervention. | all variants |
| E1 | Causal-chain completeness | Environment-specific critical events occur in the required order, e.g., key before door, red before blue, gap crossing without lava contact, cue before terminal choice. | all envs |
| E2 | State encoding validity | Symbolic states use valid MiniGrid object, color, and state indices and remain aligned with rendered frames. | all variants |
| N1 | Mask coverage/alignment | Masked pixels overlap the intended target tile with mean IoU at least 0.70 and satisfy the global mask budget. | NoCue |
| CF1 | Intervention audit | The counterfactual target is removed, relocated, swapped, or blocked according to the environment card, and the failure is attributable to that intervention. | CF |
Table 8. Validator gates. The gates jointly enforce action alignment, outcome correctness, physically valid rerollout, and correct isolation of model-facing evidence from audit-only state.
Environment-specific semantic checks instantiate E1. DoorKey requires key pickup before door opening; KeyCorridor requires the color-matched key before the locked-door interaction; RedBlueDoor requires the red door to be opened before the blue door; LavaGap requires crossing through the safe gap and never stepping on lava; Memory requires cue observation before the terminal choice; and MultiRoom requires the expected sequence of door toggles and goal-room entry.
Dataset Composition
We report the benchmark in terms of exam rows rather than only raw trajectories, because the same validated trajectory group can instantiate multiple variants and presentation conditions. Table 9 summarizes the paper-facing MiniGrid benchmark used in the main experiments. The table describes the diagnostic role of each task, the source environments, the number of model-facing exam rows, and the variant or probe structure used to construct those rows.
| Task | Diagnostic role | Environments | Exam rows | Variant/probe structure |
| A | Action-conditioned next-observation prediction | Six core environments | 103 | Local transitions from 18 source trajectories; 103 baseline questions |
| B | Single-frame perception and interface control | Six core environments | 144 | : 12 source groups per environment, two frames per group; structured scene parsing |
| C | Presentation invariance and representation sensitivity | KeyCorridor, MultiRoom, RedBlueDoor | 486 | 18 groups: temporal/visual items, plus framing items |
| D | Causal sensitivity under matched intervention | Six core environments | 144 | : two query groups per environment, three variants, four presentation conditions; successful Full references |
| E | Prefix/checkpoint perception control | UnlockPickup, BlockedUnlockPickup | 325 | 48 source trajectories (24 per environment); 136 identity and 189 checkpoint-status questions |
Table 9. Dataset composition for the MiniGrid benchmark. Counts are reported at the exam-row level, since a single validated trajectory group may produce multiple model-facing items through variants, presentation probes, or task-specific interfaces.
The five tasks serve complementary diagnostic roles rather than forming a single undifferentiated leaderboard. Task A probes local action-conditioned transition understanding; Task B and Task E provide perception and state recognition controls; Task C instantiates presentation and representation probes; and Task D provides the matched intervention setting for causal-sensitivity analysis. For Task D, the primary C4 comparison is the paired Full/CF judgment, while retained NoCue rows are used only as auxiliary evidence-removal diagnostics. Model-specific run information, aggregate exclusions, and per-model results are reported separately in the experimental appendices.
Data Format and Access Boundary
Each exam item separates the evidence shown to the VLM judge from the simulator metadata used for auditing and scoring. The model-facing payload consists of the rendered image or image sequence, the mission or action text when required by the task, the task instruction, and the requested answer format. Audit-only fields include the ground-truth label, success flag, reward, termination status, symbolic state trace, action IDs, random seed, NoCue metadata, and CF intervention metadata. This separation is a core part of the benchmark design: the VLM must judge from the same visual/textual evidence that appears in the prompt, while the simulator state remains available for reproducible validation.
A simplified schematic of an exam item is shown below:
{
"uid": "C.keycorridor.s000001.full.orig.clean",
"task": "C",
"env_task": "keycorridor",
"group_id": "s000001",
"variant": "full",
"presentation": {
"temporal": "orig",
"visual": "clean",
"rep": "composite"
},
"model_input": {
"image": "...",
"prompt": "..."
},
"audit_only": {
"label": 1,
"success": true,
"state_seq": "...",
"cf_meta": null
}
The exact field names, file locations, dataset card, license, model identifiers, API parameters, and response parsing rules are documented in Appendix § 14. Keeping those implementation details in Appendix § 14 avoids duplicating schema definitions across appendices and keeps Appendix § 10 focused on the data-generation logic that supports the main SCI claims.
Task Interfaces and Prompt Templates
GridWM-Judge contains five diagnostic task interfaces. This appendix specifies what each model-facing item shows to the VLM judge, what answer format is expected, and how the gold answer is obtained from simulator-grounded metadata. Appendix § 9 gives the mathematical definitions of the metrics, Appendix § 10 describes trajectory generation and validation, and Appendix § 14 gives the full machine-readable schemas, decoding settings, and unified output parsing rules. Throughout this appendix, model-facing fields denote information provided to the VLM judge, whereas labels, rewards, symbolic states, intervention records, and audit metadata are used only for scoring and validation. Table 10 summarizes the model-facing evidence, required answer format, and diagnostic role for the five task interfaces. Figure 9 gives representative model-facing layouts for the five tasks, illustrating the difference between local controls, presentation probes, transition prediction, and causal trajectory judgment.
| Task | Model-facing evidence | Required answer | Diagnostic role |
| B | Single MiniGrid frame with a generic scene-description instruction | Structured JSON scene description or component labels | Perception and interface control |
| E | Prefix-only sparse storyboard before the terminal state | JSON label for the first picked object or checkpoint status | Object/state tracking control |
| C | Sparse trajectory storyboard with presentation, framing, and representation probes | Binary success/failure verdict | Presentation invariance; representation exchangeability |
| A | Current observation, action condition, and four candidate next observations | Multiple-choice next-state label | Action-conditioned transition diagnostic |
| D | Full-trajectory evidence for matched Full and CF variants | Binary success/failure verdict | Causal sensitivity under intervention |
Table 10. Summary of the five task interfaces. Appendix § 11 describes the model-facing input/output contract; Appendix § 9 defines the metrics computed from these outputs.
Figure 9. Representative model-facing interfaces for the five GridWM-Judge tasks. Task B uses a single-frame structured perception query; Task E uses a prefix-only checkpoint storyboard; Task C presents the same trajectory through composite or split-image evidence formats; Task A asks for the correct action-conditioned next observation; and Task D compares matched Full/CF trajectory judgments. Audit-only simulator states, labels, rewards, and intervention metadata are not shown to the model.
Task B and E: Perception Controls
Tasks B and E are local controls. They are not intended to establish SCI by themselves; instead, they test whether the model can parse the visual interface before higher-level failures in Tasks C and D are interpreted as presentation or causal-reasoning failures.
Task B: structured scene perception. Task B presents a single egocentric MiniGrid observation and asks the model to produce a structured description of the local state. The item includes a generic
instruction explaining the requested fields, but it does not include the mission, action sequence, reward, terminal flag, or symbolic state.
Prompt template. You are given one MiniGrid observation. Describe the visible local state using JSON only. Include the agent direction, whether the agent is carrying an object, the front cell, and visible objects. Do not infer hidden objects.
The canonical output is a JSON object with fields such as agent, front_cell, and objects. A representative schema-level example is:
{
"agent": {
"dir": "right",
"carrying": null
},
"front_cell": {
"type": "key",
"color": "yellow",
"state": null
},
"objects": [
{"type": "key", "color": "yellow", "relative_pos": "front"},
{"type": "door", "color": "yellow", "state": "locked"}
]
}
Gold answers are derived from simulator-grounded annotations corresponding to the rendered frame. Task B reports a normalized weighted component score and [email protected], the fraction of items whose score reaches 0.90. Invalid or unavailable outputs receive zero in the conservative all-row analysis.
Task E: checkpoint-style long-range state tracking. Task E presents a prefix-only sparse storyboard from frame 0 through a checkpoint , where is strictly before the terminal frame. The model must answer an object-identity or object-status question without using a terminal shortcut. The item does not expose symbolic state, reward, success labels, or future frames after .
Identity-query template. Below is a MiniGrid trajectory storyboard showing the trajectory up to the last shown frame only. Earlier in the shown trajectory, what was the first object the agent picked up? Answer using JSON only: {"answer":"<label>"}.
Status-query template. Below is a MiniGrid trajectory storyboard showing the trajectory up to the last shown frame only. By the last shown frame, what is the status of the first object the agent picked up earlier? Answer using JSON only: {"answer":"<label>"}.
The identity label space contains object–color labels such as key_yellow, key_green, key_purple, ball_yellow, ball_green, and related colors appearing in the canonical artifact. The status label space contains holding, dropped, and reacquired_and_holding. Gold labels are derived from the simulator’s object-tracking events before or at : the first pickup event determines the tracked object, and the object’s checkpoint inventory/grid status determines the status label.
Task C: Presentation Invariance Tests
Task C is the main interface for presentation-aware binary judging. It asks whether a trajectory satisfies the task goal, while varying how the same trajectory evidence is presented. In the SCI terminology, temporal order, visual style, and prompt framing instantiate C1 presentation probes, while composite versus split-image evidence formats instantiate the C3 representation exchangeability probe. These changes should not alter the verdict when the underlying simulator semantics are unchanged.
Input and answer contract. The model sees a sparse storyboard built from event-anchored keyframes and a short success/failure question. The mission text is included when the item uses an environment-level goal. The action sequence, symbolic states, terminal flag, reward, intervention metadata, and gold label are not shown.
Neutral prompt template. Below is a storyboard from one MiniGrid trajectory. Task goal: {mission_string} Did the trajectory satisfy this goal? Answer with ONLY: Yes or No.
The output is mapped to a binary verdict, with Yes/Success corresponding to success and No/Fail corresponding to failure. The exact parser and invalid-response handling are specified in Appendix § 14.
Probe axes. Table 11 summarizes the Task C trajectory, presentation, and representation axes. Task C items are organized by trajectory condition and presentation condition. The trajectory condition
| Axis | Values | Interpretation |
| Trajectory condition | Full, NoCue, CF | Changes the evidence condition or the underlying outcome; not all levels are semantics-preserving. |
| Temporal order | original, reversed | C1 presentation probe over frame ordering. |
| Visual style | clean, noisy, style-transformed | C1 presentation probe over low-level rendering perturbations. |
| Prompt framing | neutral, positive, negative | C1 presentation probe over wording or prior-framing changes, without changing the task goal or evidence. |
| Representation | composite storyboard, split images | C3 equivalent-evidence-format probe for representation exchangeability. |
Table 11. Task C probe axes and their interpretation. Presentation and representation probes are evaluated within a fixed trajectory condition. When CF rows are present, Full–CF differences are treated as causal/outcome changes rather than semantics-preserving presentation changes.
may be Full, NoCue, or CF; only the presentation and representation axes are used as semantics-preserving invariance probes within a fixed trajectory condition. NoCue removes early visual evidence while preserving the successful outcome, and CF changes the underlying outcome through a simulator intervention. Therefore, full–counterfactual changes should be interpreted as causal sensitivity rather than presentation invariance.
Storyboard and representation formats. Figure 10 illustrates the composite storyboard and split-image representations used for the C3 exchangeability probe. For composite inputs, the selected keyframes are arranged into a compact grid and read left-to-right, top-to-bottom unless the temporal probe reverses the order. For split-image inputs, the same selected frames are supplied as separate images slots. Exact prompts and run conditions are documented with the associated evaluation artifacts. The selected frames are event-complete: they prioritize causally relevant events such as first object visibility, pickup, door toggling, hazard crossing, goal visibility, and terminal evidence. Matched conditions for the same trajectory group should use the same frame indices whenever the corresponding frames exist, so that differences in verdicts can be attributed to the intended presentation or causal variable rather than accidental frame selection.
Figure 10. Composite storyboard versus split-image representation for Task C. The left panel arranges selected event-anchored keyframes into a compact storyboard (Composite format). The right panel presents the same keyframes as individually numbered images (Split-image format). Both formats convey identical trajectory semantics evidence.
Visual and framing variants. The clean condition uses the original rendering. The noisy condition applies mild deterministic image noise that should not alter the MiniGrid state. The style condition applies deterministic color, contrast, brightness, or sharpness adjustments. Prompt-framing variants should change only the wording around the same evidence and goal; they must not add hints, labels, rewards, or causal explanations. For example, positive framing can say that the trajectory may have succeeded, whereas negative framing can say that it may have failed, but both must ask the same binary question over the same visual evidence.
Framing prompt templates. All framing variants use the same images, mission text, and answer format; only the preamble changes. The neutral framing is:
Below is a storyboard from one MiniGrid trajectory. Task goal: \{mission_string\} Did the trajectory satisfy this goal? Answer with ONLY: Yes or No.
The positive framing is:
The trajectory may show successful progress toward the task goal. Please judge only from the visual evidence. Task goal: \{mission_string\} Did the trajectory satisfy this goal? Answer with ONLY: Yes or No.
The negative framing is: The trajectory may contain a failure to complete the task goal. Please judge only from the visual evidence. Task goal: {mission_string} Did the trajectory satisfy this goal? Answer with ONLY: Yes or No.
Task A: Action-Conditioned Dynamics
Task A tests whether the VLM judge uses the action condition to identify the correct next observation. Unlike Tasks C and D, Task A is a local transition interface rather than a trajectory-level success/failure judgment.
Input and prompt. The item shows a current observation , an action condition, and four candidate next observations labeled A–D. The candidates are rendered from MiniGrid states and are arranged as a multiple-choice panel. One candidate is the true next observation under the action; the other candidates are visually similar negatives. Figure 11 shows the multiple-choice layout used for Task A; the correct/wrong annotations are included only for illustration and are never shown to the model.
Figure 11. Layout of a Task A multiple-choice item. The current observation and the action text are shown on the left. Four candidate next observations (A–D) are rendered on the right. One candidate is the true next observation ; the others are visually similar hard negatives. Dashed annotations indicate which candidate is the correct transition and which negatives share the same trajectory or similar visual configurations. The gold answer is the letter of the correct candidate. Correct/wrong annotations are shown only for illustration and are not included in model-facing inputs.
Prompt template. Action: {action_string}. On the left is the current state. On the right are four candidate next states (A, B, C, D). Select the correct next state after executing the action. Answer with ONLY one letter: A, B, C, or D.
Candidate construction. Negative candidates are selected to avoid trivial recognition. Same-trajectory negatives are preferred, because they share the same environment, objects, and visual style but correspond to different timesteps. If needed, cross-trajectory negatives with similar local configurations are used as fallback. Candidate order is randomized with the gold candidate uniformly placed among A–D.
Action ablations. Task A supports three action conditions. In the baseline condition, the true action is shown. In the shuffle condition, the action field is replaced by a mismatched action sampled from the action label space. In the mask condition, the action field is replaced with an unknown-action token. The ablated interfaces should not expose an action identifier that trivially recovers the original action. Large accuracy drops indicate sensitivity to the supplied action condition. Small or mixed differences are compatible with action-insensitive behavior but do not establish it alone.
Ablated action templates. The baseline condition displays the true action as Action: {action_string}. In the shuffle condition, the same field is replaced by a different action sampled
from the MiniGrid action set: Action: {shuffled_action_string}. In the mask condition, the action field is replaced by Action: unknown. The image candidates and answer format remain unchanged across these conditions.
Task D: Causal Sensitivity Tests
Task D is the primary interface for causal sensitivity. It compares matched successful Full trajectories against CF trajectories generated by a minimal simulator intervention and rerollout under the unchanged action sequence. The intended question is whether the judge changes its verdict when the underlying task outcome changes.
Input and prompt. The dedicated reference-aided Task-D protocol contains (1) a successful Reference rollout that illustrates the intended task objective and (2) a separate Query rollout whose outcome must be judged. The Query may correspond to a Full, NoCue, or CF variant; Full/CF are the primary paired conditions for discrimination. The judge must infer the specific objective from the Reference and return Success or Fail for the Query; audit-only gold labels, rewards, simulator success flags, and intervention metadata are excluded from the input.
Prompt template. Below are two storyboard trajectories from the same MiniGrid environment. Reference trajectory: This trajectory is a known successful example of completing the task in this environment. Query trajectory: Determine whether this trajectory also completes the same task goal as the reference trajectory. Answer with ONLY: Success or Fail.
Gold answer and scoring. The gold answer is Success for the matched successful Full trajectory and Fail for the corresponding CF trajectory. The main C4 metrics are computed over paired Full–CF groups and are defined in Appendix A. If a NoCue version is included in the Task D artifact, it should be treated as an evidence-removal diagnostic rather than as the causal-intervention pair used for C4.
Representative paired case. Figure 12 illustrates a representative DoorKey Full/CF pair. The pair keeps the same action sequence. In the Full trajectory, the agent picks up the key, unlocks the door, and reaches the goal. In the matched CF trajectory, the initial intervention relocates or removes the goal so that the same action sequence no longer satisfies the task. A reliable judge should accept the Full montage and reject the matched CF montage, while not changing its verdict merely because the same montage is shown in a different but semantically equivalent representation.
Figure 12. Illustration of a matched Full/CF pair for Task D (DoorKey example). Both panels show the same action sequence in the same visual format. In the Full trajectory (left), the goal is reachable and the agent completes the task; the gold verdict is Success. In the matched CF trajectory (right), a minimal intervention relocates the goal before the episode starts; the same action sequence now fails. The gold verdict is Fail. A reliable judge should accept the Full and reject the CF.
Supplementary Results and Analysis
This appendix provides detailed MiniGrid evaluations supporting the SCI diagnostics in the main paper. We report model-level and environment-level results, reference-aided causal discrimination, presentation and representation sensitivity, perception and output-compliance controls, and matched prompt ablations. Expanded evaluations further examine the stability of these findings with larger trajectory sets and clustered uncertainty estimates. Throughout, Task C and Task D retain their distinct judging protocols, and statistical comparisons follow the eligibility and rules defined in Appendix A. Qualitative cases and additional model-family results complement the quantitative analysis.
Main MiniGrid Diagnostics
The main Task-C results are summarized in Table 2. We provide additional diagnostics of representation consistency and reference-aided causal discrimination below, followed by environment-level and controlled experimental analyses.
Representation consistency. The main paper reports that composite and split-image evidence formats can change judge behavior even when the underlying trajectory semantics are fixed. Table 12 provides a compact representation-consistency summary using VCC. Higher values indicate that the judge gives the same verdict across equivalent evidence formats; 1.0 would indicate perfect representation exchangeability.
| Model | |
| Claude Sonnet 4.6 | 0.69 |
| Gemini 2.0 Flash | 0.66 |
| Gemini 2.5 Flash Lite | 0.66 |
| Gemini 3 Flash | 0.73 |
| GPT-4o | 0.71 |
| GPT-5.4 | 0.66 |
| Kimi K2.5 | 0.76 |
| Qwen3-8B | 0.66 |
| Qwen3-32B | 0.66 |
| Qwen3-235B | 0.66 |
| Mean | 0.68 |
Table 12. Task-C representation consistency. measures verdict agreement between matched Composite and Split inputs, pooled across Full and CF trajectory conditions. NoCue is excluded from this pooled comparison.
The mean VCC values are around 0.68–0.70, indicating that representation consistency is imperfect across the model suite. These results support the need to report representation exchangeability separately from both plain accuracy and causal-sensitivity metrics.
Reference-Aided Task-D Results. The dedicated Task-D evaluation provides a successful Reference and a Composite-format Query, which the judge labels Success or Fail. Table 13 reports paired causal discrimination on 12 complete Full/CF Query groups per model, using strict verdict parsing and an all-group denominator. Invalid or truncated outputs are counted as incorrect. The resulting activation and CF-rejection profiles differ substantially: GPT-5.4 obtains and , whereas Claude Sonnet 4.6 obtains 0.250 and 0.250, respectively. These estimates also motivate the larger-scale analyses reported below.
Environment-Level Breakdown
The aggregate results above may hide environment-specific failure patterns. Figure 13 therefore reports an environment-level breakdown for the Task C MiniGrid settings. We restrict this plot to the three environments that share the same Task C presentation and representation probe structure—KeyCorridor, MultiRoom, and RedBlueDoor—so that the environment comparison is not confounded by task-specific coverage differences. Scores are averaged over composite and split representations.
Figure 13. Environment-level MiniGrid breakdown for Task C. The heatmaps report (a) Full-trajectory activation and (b) raw paired Full–CF discrimination . Each environment contains six trajectory groups per representation under neutral presentation. Scores are averaged equally across Composite and Split, using all groups and counting invalid verdicts as incorrect.
Task-B Output Compliance and Failure-Source Accounting
Task B reports [email protected], the all-row fraction whose normalized, weighted scene-component score reaches at least 0.90; it is not JSON exact-match accuracy. The submitted ten-model run contains 1,440 responses: 897 strict JSON outputs, 269 deterministically recoverable outputs, 121 other unparsable outputs, 72 explicit truncations, 81 API errors, and no missing responses. Deterministic recovery is limited to JSON extraction, syntax normalization, and canonical field names. Table 14 reports response validity and all-row performance, while Table 15 breaks down component-level recovery on parseable responses. Together, they distinguish output compliance from partial scene understanding.
| Model | Parsed | API | Trunc. | Other unparse. | [email protected] | Mean-all / parsed |
| Claude Sonnet 4.6 | 141 | 0 | 0 | 3 | 0.000 | 0.186 / 0.190 |
| Gemini 2.0 Flash | 63 | 81 | 0 | 0 | 0.007 | 0.099 / 0.226 |
| Gemini 2.5 Flash Lite | 140 | 0 | 0 | 4 | 0.000 | 0.157 / 0.161 |
| Gemini 3 Flash | 30 | 0 | 0 | 114 | 0.090 | 0.168 / 0.807 |
| GPT-4o | 144 | 0 | 0 | 0 | 0.000 | 0.223 / 0.223 |
| GPT-5.4 | 144 | 0 | 0 | 0 | 0.111 | 0.616 / 0.616 |
| Kimi K2.5 | 72 | 0 | 72 | 0 | 0.007 | 0.210 / 0.420 |
| Qwen3-235B | 144 | 0 | 0 | 0 | 0.000 | 0.400 / 0.400 |
| Qwen3-32B | 144 | 0 | 0 | 0 | 0.000 | 0.119 / 0.119 |
| Qwen3-8B | 144 | 0 | 0 | 0 | 0.000 | 0.119 / 0.119 |
Table 14. Task-B output compliance and partial semantic recovery. Every model has 144 submitted rows. Mean-all includes infrastructure and format failures as zero; mean-parsed conditions on deterministically parsed responses. The all-row [email protected] values correspond to the Task-B results reported in Table 4.
| Model | Position | Carrying | Front cell | Object Jaccard | |
| Claude Sonnet 4.6 | 141 | 0.064 | 0.709 | 0.035 | 0.182 |
| Gemini 2.0 Flash | 63 | 0.206 | 0.714 | 0.095 | 0.052 |
| Gemini 2.5 Flash Lite | 140 | 0.043 | 0.721 | 0.007 | 0.111 |
| Gemini 3 Flash | 30 | 1.000 | 0.867 | 0.533 | 0.817 |
| GPT-4o | 144 | 0.201 | 0.708 | 0.090 | 0.058 |
| GPT-5.4 | 144 | 0.903 | 0.847 | 0.444 | 0.228 |
| Kimi K2.5 | 72 | 0.569 | 0.694 | 0.208 | 0.254 |
| Qwen3-235B | 144 | 0.618 | 0.715 | 0.257 | 0.014 |
| Qwen3-32B | 144 | 0.000 | 0.715 | 0.000 | 0.000 |
| Qwen3-8B | 144 | 0.000 | 0.715 | 0.000 | 0.000 |
Table 15. Task-B component scores on parsed rows only. The parsed support varies considerably across models and must not be interpreted as a uniform full-sample estimate.
Separate generation-fix check. On the same 114 Gemini 3 Flash Task-B UIDs, a separately labeled output-generation/routing fix improved parsing from 0.208 to 0.993, the all-row mean component score from 0.168 to 0.788, and [email protected] from 0.090 to 0.451. This additional run diagnoses an output pipeline limitation and does not silently replace the originally submitted scores. High-threshold failures therefore cannot be equated with total absence of partial spatial or object understanding.
Perception-Controlled Oracle-State Hierarchy
The Oracle-State control evaluates GPT-5.4 and Gemini 3.1 Pro on 98 shared groups in nine environments. Each group includes a successful Reference from which the judge must recover the task type and objective before classifying a separate Query. The Pixel condition supplies rendered frames; State adds selected observable trajectory facts; and Pixel+State provides both. The state allowlist excludes the exact objective, gold verdict, reward, success, and termination fields. These are differences in available evidence, not merely alternative layouts of identical pixels.
For GPT-5.4, State increases joint task/objective/verdict accuracy from 0.320 to 0.820 and raw IDR from 0.469 to 0.765; for Gemini, those values increase from 0.347 to 0.895 and from 0.602 to 0.929. The group interval for GPT’s gain excludes zero, but the more conservative environment-level interval includes zero. These contrasts implicate visual/interface grounding and objective recovery as major, model-dependent bottlenecks without establishing that every Pixel error originates in causal reasoning. Table 16 and Table 17 report the corresponding stage-wise and paired metrics.
| Model | Evidence | Task | Objective | Verdict | Joint |
| GPT-5.4 | Pixel | 0.442 | 0.405 | 0.731 | 0.320 |
| GPT-5.4 | State | 0.905 | 0.905 | 0.895 | 0.820 |
| GPT-5.4 | Pixel+State | 0.854 | 0.874 | 0.881 | 0.765 |
| Gemini 3.1 Pro | Pixel | 0.704 | 0.395 | 0.755 | 0.347 |
| Gemini 3.1 Pro | State | 0.949 | 0.912 | 0.973 | 0.895 |
| Gemini 3.1 Pro | Pixel+State | 0.983 | 0.929 | 0.997 | 0.929 |
Table 16. Oracle-State diagnostic hierarchy (98 shared groups). Task, objective and verdict are the individually scored stages; joint requires their simultaneous success. These values correspond to the same matched three-arm experiment.
| Model | Evidence | Group 95% CI | |||
| GPT-5.4 | Pixel | 0.745 | 0.469 | [0.367,0.571] | 0.630 |
| GPT-5.4 | State | 0.929 | 0.765 | [0.684,0.847] | 0.824 |
| GPT-5.4 | Pixel+State | 0.918 | 0.765 | [0.684,0.847] | 0.833 |
| Gemini 3.1 Pro | Pixel | 0.612 | 0.602 | [0.500,0.694] | 0.983 |
| Gemini 3.1 Pro | State | 1.000 | 0.929 | [0.878,0.980] | 0.929 |
| Gemini 3.1 Pro | Pixel+State | 1.000 | 0.990 | [0.969,1.000] | 0.990 |
Table 17. Oracle-State paired Full/CF diagnostics. 95% intervals apply to raw IDR and are based on complete matched groups.
Matched Presentation and Representation Probes
Presentation changes preserve simulator state, actions, and outcome labels but may change perceptual accessibility. The submitted ten-endpoint tests therefore report both correctness degradation and correctness rescue on exact matched items. Holm correction is applied within the 180-cell family (60 per probe type).
| Probe | Exact pairs | Degradation | Rescue | Acc. | Correctness flip |
| Noise | 1080 | 59 | 75 | +0.015 | 0.124 |
| Style | 1080 | 46 | 55 | +0.008 | 0.094 |
| Temporal reversal | 1080 | 136 | 92 | -0.041 | 0.211 |
Table 18. Historical matched presentation effects. Counts aggregate exactly matched comparisons across the original model suite. None of the 180 cells survives the stated global Holm correction.
| Variant | Pairs | Verdict flip | Gain / Loss | Accuracy | 95% CI |
| Full | 1620 | 35.7% | 277 / 274 | +0.002 | [-0.173,0.167] |
| NoCue | 1620 | 33.8% | 235 / 293 | -0.036 | [-0.198,0.122] |
| CF | 1620 | 25.6% | 328 / 84 | +0.151 | [0.028,0.272] |
Table 19. Outcome-stratified historical Composite/Split transitions. Flips are changes in predicted verdict, whereas gain/loss counts refer to correctness and may cancel in the net mean. Historical interfaces also varied in prompt wording, packaging, and run date; these contrasts do not isolate image resolution or tiling. The CF mean effect is exploratory and unadjusted.
In the original 270 finer Composite/Split comparison cells, only 10 survived the stated global Holm correction. In a separate Scale-2 probe check comprising 101 groups, ten environment configurations and 12 Full/CF cells, no comparison survived that experiment’s own global Holm correction. These multiple-testing families are distinct and should not be merged post hoc. Tables 18 and 19 summarize the matched presentation and representation effects.
Matched Prompt-Structure Ablation
A further Task-D study evaluates identical groups, images, and ordering under three prompting conditions: Direct, Structured Verdict, and Structured Rationale. The results demonstrate changes in response bias, not a monotonic benefit from requesting more reasoning (Table 20).
For GPT-5.4, the Rationale–Direct raw IDR difference is +0.031, with a paired 95% CI from −0.102 to +0.163. For Claude, the Rationale–Verdict difference is −0.173, with a CI from −0.286 to −0.071.
Expanded Scale-2 Evaluation and Clustered Uncertainty
The expanded scale-2 evaluations provide substantially larger task-specific samples without pooling them with the base MiniGrid results. Table 21 summarizes the evaluation scale and environment coverage. Task C and Task D use distinct judging contracts, and model coverage varies by task. Table 22 reports the corresponding paired causal diagnostics and uncertainty estimates.
| Task | Base rows | Expanded rows | Ratio | Source groups | Env. configs |
| A | 103 | 1,030 | 10.0× | 76 | 9 |
| C | 486 | 1,326 | 2.7× | 442 | 10 |
| D | 144 | 1,296 | 9.0× | 432 | 9 |
Table 21. Expanded Evaluation scale and coverage. The expanded evaluation increases task-specific sample coverage while preserving the distinct Task-C and Task-D protocols. Counts refer to exam rows and complete source groups, not independent model evaluations.
| Model | Prompt | Full accept | CF reject | [95% CI] | |
| GPT-5.4 | Direct | 0.827 | 0.541 | 0.367 [0.276,0.459] | 0.444 |
| GPT-5.4 | Structured verdict | 0.704 | 0.520 | 0.265 [0.184,0.357] | 0.377 |
| GPT-5.4 | Structured rationale | 0.449 | 0.878 | 0.398 [0.306,0.490] | 0.886 |
| Claude Sonnet 4.6 | Direct | 0.255 | 0.939 | 0.224 [0.143,0.306] | 0.880 |
| Claude Sonnet 4.6 | Structured verdict | 0.694 | 0.816 | 0.541 [0.439,0.643] | 0.779 |
| Claude Sonnet 4.6 | Structured rationale | 0.520 | 0.796 | 0.367 [0.276,0.459] | 0.706 |
Table 20. Paired Task-D prompt ablation. The three arms use matched Task-D groups; the reported intervals apply to raw IDR.
All reported confidence intervals use 20,000 percentile-bootstrap resamples. The primary intervals resample intact trajectory groups, preserving their Full/NoCue/CF dependence and matched models; the sensitivity analysis resamples complete environment configurations. Invalid and unparsable outputs count as incorrect in these all-group summaries.
For Task C, the conditional IDR estimates were 0.602 for GPT-5.4 and 1.000 for Claude Sonnet 4.6; for Task D, they were 0.447 and 0.776 for the same two models, 0.631 for Gemini 3 Flash, and 0.000 for the Qwen3-32B. On Task-D groups, the paired GPT-5.4 minus Claude Sonnet 4.6 difference in raw IDR was +0.234, with group-bootstrap 95% CI [+0.185, +0.282] and environment-bootstrap 95% CI [+0.127, +0.358]. The larger groups sharpen the uncertainty analysis, but the point estimates do not all remain numerically constant: Claude’s original Task-D IDR is 0.250, versus 0.088 on Scale-2. The supported conclusion is persistence of heterogeneous activation, CF rejection, and paired discrimination rather than invariant model rankings.
Qualitative and Exploratory Results
To complement the aggregate metrics, Figure 14 shows representative qualitative failures selected from the final frozen MiniGrid manifest. The purpose is not to introduce new metrics, but to make the failure-mode taxonomy in Appendix A concrete.
Exploratory LLaVA-Family Results. For completeness, we also evaluated LLaVA-family multimodal models under the same MiniGrid diagnostic protocol. These runs are exploratory and are not used for the main claims, because they extend the model suite beyond the primary judge set discussed in the main text. Table 23 summarizes the compact results.
| Model | Task C Acc. | Task A Acc. | Task E Acc. | Strict Rate |
| LLaVA-Critic-R1-7B | 0.33 | 0.28 | 0.11 | 0.27 |
| LLaVA-Critic-7B | 0.65 | 0.25 | 0.00 | 0.00 |
| LLaVA-OneVision-7B | 0.63 | 0.27 | 0.00 | 0.00 |
| LLaVA-OneVision-Chat | 0.63 | 0.24 | 0.00 | 0.00 |
| Mean | 0.56 | 0.26 | 0.03 | – |
Table 23. Exploratory LLaVA-family model performance on MiniGrid Tier-0. The metrics are averaged over composite and split representations. These results are reported as auxiliary evidence rather than as part of the main model suite.
The exploratory LLaVA results are substantially below the main judge suite on both correctness and intervention-discrimination metrics. The main failure surface appears to combine visual-interface parsing difficulty with weak causal-discrimination ability. We therefore report these results only as a reference point for open VLM-family models, not as evidence for the primary SCI claims.
Figure 14. Representative MiniGrid failure cases. Each row shows the model-facing evidence, simulator-derived gold label, and model verdict before and after the diagnostic change. Row 1 illustrates representation fragility, where the same Full trajectory is judged differently under composite and split representations. Row 2 illustrates causal misgrounding, where a matched Full/CF pair receives an incorrect or unchanged verdict. Row 3 illustrates a stability trap, where the model remains stable across evidence conditions but is wrong or degenerate. Gold labels are derived from simulator state and are not shown to the model.
Summary
The supplementary MiniGrid analyses reinforce three complementary observations. First, judge accuracy, verdict stability, and paired causal discrimination capture distinct aspects of reliability, as shown by the reference-aided and expanded evaluations. Second, presentation and representation probes reveal model-dependent interface sensitivity, while prompt ablations expose substantial changes in activation and rejection behavior. Third, perception controls, output-compliance analyses, and Oracle-State comparisons identify additional bottlenecks that cannot be attributed uniquely to causal reasoning. Together, these results support the joint use of correctness, invariance, and causal-sensitivity diagnostics.
3D Extensions: MiniWorld and Habitat
We extend GridWM-Judge to two simulated 3D environments, MiniWorld and Habitat-Sim/HSSD, to examine SCI diagnostics under richer visual evidence. These evaluations cover action-conditioned transitions, trajectory judgments, presentation effects, and paired causal discrimination, using simulator-grounded outcomes for evaluation.
MiniWorld Environments and Diagnostic Tasks
We include a small MiniWorld extension as a 3D external-validity probe for GridWM-Judge. Unlike the Tier-0 MiniGrid suite, which is designed for tightly audited symbolic-state control, MiniWorld renders trajectories from a first-person 3D viewpoint. This introduces viewpoint variation, depth cues, partial occlusion, object-scale variation, and navigation ambiguity while preserving the same simulator-backed single-source-of-truth principle used throughout the benchmark. Figure 15 illustrates the MiniWorld evidence interface, simulator metadata, and matched success/failure structure.

Figure 15. MiniWorld evidence interface and simulator-grounded source of truth. The figure shows representative first-person 3D frames, aligned simulator state/action traces, and a matched success/failure pair. Rendered frames are model-facing, while states, interventions, and labels remain audit-only metadata.
The MiniWorld extension examines whether SCI diagnostics remain informative under first-person 3D visual evidence. We retain simulator-grounded state traces, actions, observations, and outcome labels, while exposing only task-relevant evidence to the judge. For each MiniWorld rollout, we retain a simulator state trace, an action sequence, rendered visual evidence, a mission or task description, and an outcome label. The model-facing item contains only rendered evidence and task instructions; states, intervention records, labels, and validation metadata remain audit-only. Table 24 summarizes how the MiniWorld extension mirrors the SCI diagnostic logic while remaining separate from the Tier-0 MiniGrid results.
| Component | Diagnostic role | Model-facing evidence | Reported output |
| MiniWorld suite | 3D external-validity probe | First-person RGB storyboards; simulator state traces retained as audit metadata | Env./group/item counts |
| Task A-style | C2: action-conditioned evidence use | Current 3D observation, action text, and candidate next observations or trajectory slices | |
| Task C-style | C1: presentation-sensitive trajectory judgment | Composite 3D storyboard with mission and action context | |
| Framing probe | C1: prompt-framing sensitivity | Alternative framing prompt applied to the same 3D trajectory evidence | |
| Task D-style | C4: causal sensitivity | Successful trajectory evidence versus matched failure or counterfactual evidence | , |
Table 24. MiniWorld extension design. The 3D extension mirrors the SCI diagnostic logic of MiniGrid but is reported separately because the visual interface and trajectory distribution differ from the Tier-0 2D suite.
The MiniWorld extension focuses on three diagnostic questions: (i) whether the judge can use action-conditioned 3D evidence, (ii) whether success/failure judgments change under prompt framing, and (iii) whether the judge distinguishes successful trajectories from matched failure cases. We report the resulting accuracies separately from the MiniGrid Tier-0 results.
MiniWorld Results and Analysis
Table 25 reports MiniWorld results as a 3D external-validity probe rather than as a second leaderboard. The table should be read along three axes: action-conditioned evidence use (), trajectory-level success/failure judgment (), and Task-D-style failure discrimination (). The framing-delta columns report how much the same evaluation changes under the finalized alternative framing prompt.
Table 25: MiniWorld diagnostic results. Values are point estimates from the MiniWorld composite-SSOT result file. denotes the change from the neutral prompt to the alternative framing probe; negative values indicate lower accuracy under framing. Because the current MiniWorld result file reports aggregate accuracies rather than paired Full–CF discrimination rates, should be interpreted as a coarse Task-D-style failure-discrimination diagnostic rather than the full C4 IDR analysis used in MiniGrid. Support counts are provided in the released MiniWorld manifest.
| Model | |||||
| Claude Sonnet 4.6 | 0.50 | 0.38 | +0.12 | 0.68 | −0.13 |
| Gemini 2.0 Flash | 0.40 | 0.43 | −0.15 | 0.20 | +0.17 |
| Gemini 2.5 Flash Lite | 0.48 | 0.35 | +0.02 | 0.50 | −0.01 |
| Gemini 3 Flash | 0.55 | 0.81 | −0.07 | 0.78 | +0.02 |
| GPT-4o | 0.55 | 0.75 | −0.16 | 0.50 | +0.05 |
| GPT-5.4 | 0.50 | 0.51 | +0.03 | 0.66 | −0.02 |
| Kimi K2.5 | 0.50 | 0.82 | −0.14 | 0.54 | +0.01 |
| Qwen3-8B | 0.49 | 0.67 | −0.11 | 0.50 | +0.00 |
| Qwen3-32B | 0.49 | 0.67 | −0.11 | 0.50 | +0.00 |
| Qwen3-235B | 0.51 | 0.55 | −0.03 | 0.50 | +0.00 |
| Mean | 0.50 | 0.59 | −0.06 | 0.54 | +0.01 |
Table 25.
Figure 16 visualizes the same MiniWorld results as a diagnostic profile, separating accuracy-style axes from framing effects across models. The main pattern is that no single MiniWorld accuracy number explains judge reliability. For example, GEMINI 3 FLASH is among the strongest models on the trajectory-level and Task-D-style columns, with and . However, its Task-C score changes under framing (), showing that strong trajectory-level accuracy does not by itself imply presentation invariance. Similarly, GPT-4o obtains a high but a lower , indicating that success/failure judgment and matched failure discrimination should be interpreted separately.
Figure 16. MiniWorld diagnostic profile across models. (a) Accuracy Profile: point-estimate accuracies for three MiniWorld diagnostic axes: action-conditioned evidence use (), trajectory-level success/failure judgment (), and Task-D-style failure discrimination (). Darker cells indicate higher accuracy, and the overlaid numbers give the corresponding point estimates. (b) Framing Effect: changes in accuracy under the finalized alternative framing prompt, computed relative to the corresponding neutral prompt as and . Negative values indicate lower accuracy under framing, while positive values indicate higher accuracy. The summary row reports the mean value of each metric across models. The figure highlights that MiniWorld model rankings differ across diagnostic axes and that framing sensitivity is not reducible to overall accuracy.
Model ranking also changes across diagnostic axes. KIMI K2.5 obtains the highest in this MiniWorld evaluation (0.82), but its Task-D-style score is lower than that of GEMINI 3 FLASH (0.54 vs. 0.78). Conversely, CLAUDE-SONNET-4-6 performs better on (0.68) than on (0.38). This separation is consistent with the SCI framing: presentation robustness, trajectory-level correctness, and Task-D-style failure discrimination are distinct properties and should not be collapsed into a single aggregate score.
The mean MiniWorld scores remain close to the mid-range: , , and . This suggests that the 3D extension is not saturated by current judges. In particular, several models remain near chance on Task-D-style metrics, and GEMINI-2.0-FLASH obtains a low . Because the current MiniWorld result file reports aggregate accuracies rather than paired IDR-style discrimination rates, these numbers should be interpreted as coarse diagnostic evidence rather than as the full C4 analysis used in MiniGrid.
Compared with the 2D MiniGrid benchmark, the MiniWorld extension preserves the same methodological logic but exposes a different failure surface. Both settings use simulator-backed labels, aligned visual evidence, and controlled presentation or causal variants. MiniWorld additionally introduces first-person 3D perception burdens, including camera pose, partial observability, occlusion, depth cues, and object-scale variation. It is therefore best viewed as a stress test of whether SCI-style diagnostics remain informative under a richer evidence interface, not as a replacement for the fully audited Tier-0 MiniGrid suite.
Habitat-Sim/HSSD Experimental Setup
We evaluate GridWM-Judge in Habitat-Sim using HSSD synthetic articulated indoor scenes, with GPT-5.4 and Gemini 3 Flash as the evaluated VLM judges. Task A comprises 150 action-conditioned transition items from 10 scenes per model. Tasks C and D evaluate the same 100 underlying Query groups across 18 scenes, with matched Full/CF conditions. Task C uses trajectory-level judgments, whereas Task D additionally provides a successful Reference to establish the task objective. Counterfactual failures are generated through articulated action-effect interventions while preserving the Query action sequence.
| Model | Task | Scenes | Acc. | CF rej. | Group 95% CI | |||
| GPT-5.4 | A | 150 | 10 | 0.447 | – | – | – | – |
| Gemini 3 Flash | A | 150 | 10 | 0.553 | – | – | – | – |
| GPT-5.4 | C | 100 | 18 | 0.590 | 0.280 | 0.900 | 0.230 | [0.150,0.320] |
| Gemini 3 Flash | C | 100 | 18 | 0.720 | 0.870 | 0.570 | 0.470 | [0.370,0.570] |
| GPT-5.4 | D | 100 | 18 | 0.550 | 0.580 | 0.520 | 0.160 | [0.090,0.240] |
| Gemini 3 Flash | D | 100 | 18 | 0.590 | 0.970 | 0.210 | 0.200 | [0.120,0.280] |
Table 26. Habitat synthetic-3D evaluation. denotes transition items for Task A and matched Query groups for Tasks C/D. For Tasks C/D, accuracy is the mean of Full acceptance and CF rejection. Raw IDR is reported with group-bootstrap 95% confidence intervals.
Habitat Results
Table 26 reports the HSSD evaluation results. The findings reveal distinct patterns of Full activation, CF rejection, and paired discrimination under synthetic 3D visual evidence. For GPT-5.4 on Task C, Full activation is 0.280 and CF rejection reaches 0.900, but raw IDR is only 0.230. Conversely, Gemini 3 Flash achieves 0.970 Full activation on reference-aided Task D, yet its CF rejection is 0.210 and raw IDR is 0.200. These contrasting profiles show that aggregate accuracy and successful-trajectory acceptance do not fully characterize paired causal discrimination. The results extend the central SCI diagnostic distinction to a richer synthetic 3D setting.
Cross-simulator interpretation. MiniWorld and Habitat-Sim/HSSD use different visual interfaces, intervention protocols, and model subsets. Their results are therefore analyzed separately rather than pooled into a single ranking. Both evaluations demonstrate the diagnostic value of distinguishing judgment correctness, presentation sensitivity, and causal discrimination under simulated 3D evidence.
Implementation Details and Reproducibility
This appendix documents the implementation, data release, schema conventions, prompting and parsing protocol, and reproducibility information for GridWM-Judge. The experiments comprise the base MiniGrid Task A–E diagnostics, the Scale-2 Task A/C/D scale extension, the MiniWorld and Habitat Task A/C/D 3D environment extensions, and controlled analyses. Scale-2 expands the trajectory bank and environment coverage. MiniWorld and Habitat evaluate judging in 3D environments. The controlled analyses use Oracle-State evidence to control for visual perception, structured-CoT conditions to examine response format and reasoning, and matched presentation probes to compare evidence representations and presentation changes. The following sections describe the code, data, configurations, and commands used to generate, run, and score these experiments.
Code and Data Availability
The public code and dataset are available at the following links:
Code: GitHub repository.
Data: Hugging Face dataset.
The released artifacts include the generated exam files, rendered images, ground-truth annotations, prompt templates, validation metadata, scoring scripts, and result summaries used in the paper. The code is released under the MIT License. Project-authored dataset annotations and metadata use Apache-2.0. HSSD-derived rendered images retain CC BY-NC 4.0 and the associated attribution requirements; third-party assets are not relicensed by this release. Component-specific terms are documented in the dataset card and the Habitat LICENSES.md file.
Repository Structure and Compute
The repository is organized around generation, inference, scoring, and export scripts. The top-level structure is:
Towards-Reliable-VLM-Judge/
scripts/ # shell launchers
tools/
minigrid/ # trajectory and exam generation
miniworld/ # MiniWorld workflows
habitat/ # Habitat generation and validation
exams/ # exam and request assembly
inference/ # shared inference gateway
scoring/ # parsers, scores and aggregation
experiments/ # controlled diagnostic analyses
utils/ # shared helpers
environments/ # Conda environment specifications
requirements/ # pinned Python dependencies
README.md
LICENSE
Generated trajectories, exams, responses, and scores are written to an external run directory selected by the user, rather than bundled with the source repository.
This work involves no model training or fine-tuning. Dataset generation and validation are CPU-bound and use fixed seeds. VLM inference is performed either through external APIs or local inference for open-weight models. Local model runs use a single NVIDIA RTX 4060 Ti 16GB GPU. On our reference generation setup, generating the 1,202 base MiniGrid Task A–E exam items takes approximately 3.6 minutes. The 3,652 Scale-2 Task A/C/D items take 25.8 minutes, the 214 MiniWorld Task A/C/D items take 7.9 minutes, and the 550 Habitat Task A/C/D items take 82.1 minutes. Generation time does not scale directly with the number of exam items: multiple presentation variants can share the same underlying trajectory.
For example, the 486 base Task C items are derived from 18 trajectory groups with temporal, visual, and framing variations, whereas the 1,326 Scale-2 Task C items use 442 trajectory groups with three variants per group (Full, NoCue, and CF). Scale-2 therefore requires substantially more trajectory simulation, planning, validation, and frame rendering, rather than merely more presentations of existing trajectories. Its construction also includes reference matching and candidate-quality checks. The base and Scale-2 totals cover different task sets (A–E versus A/C/D), so their item counts alone are not comparable measures of generation workload.
The dominant runtime and monetary cost comes from VLM inference. Table 27 lists the main software dependencies used for generation, inference, rendering, and scoring in the core reproduction environment (Python 3.12.12). Dependency versions are pinned in the released environment specifications and lock files. Habitat uses a separate Python 3.9.23 environment with Habitat-Sim 0.3.3, NumPy 1.26.4, and Pillow 10.4.0; its environment specification and lock file are released separately.
| Dependency | Version | License | Purpose |
| MiniGrid | 3.0.0 | Apache 2.0 | Grid-world environments |
| Gymnasium | 1.2.3 | MIT | Environment interface |
| OpenAI SDK | 2.29.0 | Apache 2.0 | API inference client |
| MiniWorld | 2.1.0 | Apache 2.0 | 3D environments |
| Pygame | 2.6.1 | LGPL | Rendering support |
| Pillow | 12.1.1 | HPND | Image processing |
Table 27. Main software dependencies in the core reproduction environment; Habitat uses a separate environment.
Data Format and Metadata
The released benchmark uses JSONL files, with one exam item per line. The distinction between model-facing and audit-only fields is semantic: native exam records generally use task-specific, flat fields rather than universal model_input and audit_only containers. Only the designated prompt and ordered image inputs are sent to a VLM judge in the pixel-based experiments. Audit-only fields contain simulator states, labels, intervention metadata, validation logs, and other information used only for scoring and reproducibility.
The following is a conceptual illustration of that separation, not the literal on-disk schema:
{
"uid": "C.keycorridor.g000001.full.orig.clean.comp",
"task": "C",
"schema_version": "gridwm.exam.v1",
"env_task": "keycorridor",
"group_id": "g000001",
"variant": "full",
"presentation": {
"temporal": "orig",
"visual": "clean",
"framing": "neutral",
"rep": "composite"
},
"model_input": {
"images": ["relative/path/to/image_or_frame.png"],
"prompt": "model-facing task instruction"
},
"audit_only": {
"label": 1,
"success": true,
"state_seq": "relative/path/to/state_trace.json",
"actions": ["forward", "toggle", "..."],
"reward": 1.0,
"nocue_meta": null,
"cf_meta": null,
"validator_status": "passed"
}
}
Task-specific records implement this logical separation. Task A items store the action condition and multiple-choice candidate order. Task B items store the structured perception target, including agent direction, carried object, front cell, and visible objects. Task C items store trajectory variant and presentation-probe metadata. Task D items store matched Full/CF group identifiers for causal-sensitivity scoring. Task E items store the checkpoint index and object/status label. Actual field names and example JSONL records are provided by the released task files. For example, a native Task C record uses top-level fields such as uid, task, schema_version, env_task, group_id, variant, temporal, visual, image, prompt, answer, and label. Here image and prompt specify model-facing evidence; answer and label are retained for scoring and are not appended to the prompt. Other tasks may assemble their prompt or image list when exporting inference requests. The Croissant reading tables are derived views in which nested arrays and objects are encoded as reversible JSON strings; the original exam JSONLs retain their native structure.
For Task B, directions follow the MiniGrid convention:
| Value | Direction |
| 0 | right / east |
| 1 | down / south |
| 2 | left / west |
| 3 | up / north |
Table 32.
Door states follow the MiniGrid object-state convention: state 0 denotes open, state 1 denotes closed, and state 2 denotes locked. The front_cell occupancy type may be empty, wall, door, goal, ball, key, lava, or box. The objects list contains only visible non-background objects (keys, balls, doors, goals, lava, and boxes), excluding the front cell itself; walls, floors, and empty cells are not listed as objects. Agent information is reported separately under agent.
Prompting, Decoding, and Parsing
All models are evaluated as external judges of model-facing evidence. In the pixel-based experiments, the prompts contain only rendered images, mission or action text when required by the task, task instructions, and the expected answer format. They do not contain ground-truth labels, rewards, simulator states, validation flags, or intervention metadata. The Oracle-State control analysis adds simulator-derived state evidence in its State and Pixel+State conditions to control for visual perception; Pixel remains the image-only condition. These analysis conditions do not expose the gold verdict, reward, exact simulator mission, or audit-only validation and intervention annotations.
The evaluated model aliases in the paper are summarized in Table 28. Exact provider endpoint strings, run timestamps, and API parameters are recorded in the released run metadata files.
| Alias in paper | Model family / provider | Inference route |
| Claude Sonnet 4.6 | Claude Sonnet family | API |
| GPT-4o | GPT-4o family | API |
| GPT-5.4 | GPT-5-family judge run | API |
| Gemini 3 Flash | Gemini Flash family | API |
| Gemini 3.1 Pro | Gemini Pro family (Oracle analysis) | API |
| Gemini 2.5 Flash Lite | Gemini Flash Lite family | API |
| Gemini 2.0 Flash | Gemini Flash family | API |
| Kimi K2.5 | Kimi family | API |
| Qwen3-8B | Qwen open-weight family | local/API run |
| Qwen3-32B | Qwen open-weight family | local/API run |
| Qwen3-235B | Qwen large open-weight family | local/API run |
Table 28. Model aliases used in the paper and corresponding inference route. Exact endpoint strings and run metadata are included in the released machine-readable logs.
The following table reports the decoding settings for the base MiniGrid direct-answer experiments; temperature is set to zero when supported by the provider:
| Parameter | Value |
| temperature | 0.0 |
| top_p | not set unless required by provider |
| top_k | not set unless required by provider |
| max tokens, Task A | 512 |
| max tokens, Task B | 2048 |
| max tokens, Task C/D/E | 256 |
Table 33.
The output-token limits above are not shared by all experiments or reruns. Controlled analyses, including the four-field Oracle-State responses, and truncation-recovery reruns may use larger output budgets. For example, the Gemini 3.1 Pro Oracle run uses an 8192-token limit. Experiment-specific run metadata governs these exceptions; a response that reaches its output limit may be truncated and is scored according to the actual recoverability of its answer.
Invalid, empty, refused, or unparsable responses are logged separately. In conservative all-group analyses, such outputs are counted as incorrect. The main Task-C paired metrics retain the released valid-pair eligibility rules, whereas dedicated Task-D and Scale-2 analyses use their specified all-group protocols (Appendix A). The parser is task-specific:
Task A. The parser extracts the first isolated multiple-choice label in . If no valid label is found, the response is marked invalid.
Tasks C and D. Binary success/failure judgments are parsed with case-insensitive keyword matching. Success words include success, succeed, succeeded, successful, and yes. Failure words include fail, failed, failure, and no. If both classes appear, the parser applies the deterministic conflict rule implemented in the released scoring script; otherwise invalid or ambiguous outputs are marked incorrect.
Tasks B and E. The scorers distinguish strict, recoverable, and failed parses rather than rejecting all formatting deviations. Task B first extracts a JSON object and attempts standard JSON parsing, with limited recovery for surrounding text, code fences, and trailing commas. Task B requires the expected structured fields for the agent, front cell, and visible objects. Task E expects a JSON object with an answer field and also attempts recovery from code fences or surrounding text. Unambiguously recoverable answers contribute to the scores, with parse modes logged separately. Exact-match and component-level scores are both logged when applicable.
Reproduction Commands and Reporting
| Item | Documentation / release location |
| Code | public GitHub repository |
| Dataset | public Hugging Face dataset |
| Exam files | JSONL files in dataset release |
| Rendered images | image folders referenced by JSONL records |
| Prompt templates | repository and dataset metadata |
| Ground-truth labels | audit-only fields in JSONL records |
| Model outputs | prediction logs and score files |
| Scoring scripts | tools/scoring/, via scripts/score.sh |
| Random seeds | workflow-specific; see commands and run metadata |
| Training or fine-tuning | none |
| Dataset license | Apache-2.0; HSSD images: CC BY-NC 4.0 |
| Code license | MIT License |
| Privacy | synthetic simulator data only; no human subjects or scraped images |
Table 29. Reproducibility summary.
Table 29 summarizes the released artifacts and documentation needed to reproduce the benchmark results. A typical reproduction workflow is:
# 1. Install dependencies
conda env create -f environments/core.yml
conda activate gridwm-repro-core
python -m pip install -r requirements/core.lock.txt
export RUN_ROOT="$HOME/gridwm-runs"
# 2. Generate validated trajectories and exam items
bash scripts/generate_minigrid.sh --profile formal \
--output "$RUN_ROOT/base"
# 3. Run model inference
export BUNDLE="$RUN_ROOT/base/minigrid/bundle"
export REQUESTS="$BUNDLE/_requests_from_manifest"
bash scripts/infer.sh \
--requests "$REQUESTS/requests_ABCDE_full.jsonl" \
--protocol chat --base-url "$VLM_BASE_URL" --model "$MODEL" \
--max-tokens 8192 --temperature 0 --workers 2 --rps 1 \
--output "$RUN_ROOT/base-responses"
# 4. Score predictions and export metrics
bash scripts/score.sh --protocol minigrid \
--exam-dir "$BUNDLE/exam" \
--responses "$RUN_ROOT/base-responses/responses.jsonl" \
--ci --output "$RUN_ROOT/base-scores"
Set VLM_BASE_URL and MODEL, and configure the provider credentials as documented in the repository README, before inference. The inference command illustrates the shared launcher with an 8192-token budget. For the base MiniGrid direct-answer settings, run task-specific request files with the corresponding limits listed above. The repository README provides wrappers for the other environments and analysis protocols.
Seeds are workflow-specific. The generation launcher defaults to 830000; base Task A exam sampling uses 42. Habitat navigation generation uses base seed 20260725, while its action-candidate construction and exam sampling use 20260726. The structured-CoT and Oracle-State scoring protocols use bootstrap seeds 20260724 and 20260725, respectively; the general confidence-interval workflow defaults to 20261103. Seeds and overrides should be reported with the corresponding generation or scoring command.
The released scoring scripts include the deterministic metric aggregation and uncertainty-estimation procedures used to produce the tables and figures in the paper.
Scope and Impact
This appendix discusses the scope, limitations, and broader impacts of GridWM-Judge. We summarize the validity boundaries of the controlled diagnostics, consider responsible benchmark use and release, and outline directions for extending SCI evaluation to more diverse models and environments.
Scope and Limitations
Table 30 summarizes the scope of claims supported by our work and the corresponding limitations.
| Axis | What the paper supports | What remains outside scope |
| Domain | Controlled MiniGrid diagnostics with auditable states, actions, and success predicates. | Direct deployment claims for real robots, photorealistic video, or open-ended embodied tasks. |
| Agent behavior | Deterministic rollouts that isolate judge-side reliability failures. | Noisy control, stochastic policies, suboptimal exploration, and long-horizon recovery behavior. |
| Presentation | Temporal, visual, prompt-framing, and representation-format probes over matched trajectory semantics. | All possible UI formats, natural-language task framings, or video encodings. |
| Models | A finite set of model-based judges under documented task-specific evaluation protocols. | Universal claims about all VLMs, future models, or heavily optimized prompting/training regimes. |
| Metrics | Diagnostic evidence for correctness, presentation stability, representation exchangeability, and causal sensitivity. | A single deployment-readiness score or safety certificate. |
Table 30. Scope of claims and corresponding limitations.
Controlled diagnostic scope. GridWM-Judge is designed as a controlled diagnostic benchmark rather than a realistic robotics benchmark. The Tier-0 evaluation suite uses MiniGrid-style symbolic environments because they provide auditable states, actions, object identities, and success predicates. This control is essential for constructing aligned Full, NoCue, and CF variants, and for separating presentation changes from causal interventions. The corresponding limitation is that MiniGrid abstracts away continuous perception, 3D geometry, photorealistic appearance, multi-agent interaction, and real-world sensor noise. Our results therefore diagnose reliability failures under controlled embodied trajectories; they should not be interpreted as direct estimates of deployment performance in real robotic systems.
Environment and agent-behavior scope. The current benchmark emphasizes short-horizon, discrete-action navigation and interaction tasks with deterministic simulator dynamics. The trajectories are produced by deterministic planners so that the underlying success/failure labels and causal events can be audited exactly. This design does not cover noisy control, partial actuation failure, stochastic policies, exploration behavior, or suboptimal agent trajectories. Future versions should include more diverse behavior policies and longer-horizon tasks, but the present paper intentionally uses deterministic rollouts to isolate judge-side failures rather than agent-side failures.
Model and prompting scope. The evaluated models constitute a finite set of model-based judges under the task-specific prompting and decoding protocols described in Appendix § 14. The benchmark is not a comprehensive leaderboard over all proprietary and open VLMs. The reported failures should be read as evidence that current representative judges can violate state-conditional invariance, not as a universal impossibility result. Likewise, we use standard judge prompts rather than systematically optimizing chain-of-thought prompting, few-shot demonstrations, tool use, self-consistency voting, or specialized fine-tuning. Although we include controlled prompt-structure ablations, we do not exhaustively optimize few-shot demonstrations, tool use, self-consistency voting, or specialized fine-tuning. Better prompting or training may improve some scores, but such improvements would need to be tested under the same causal and presentation controls.
Task and SCI-condition scope. GridWM-Judge operationalizes four diagnostic conditions: presentation invariance, action-conditioned dynamics, representation exchangeability, and causal sensitivity. Task C primarily probes presentation invariance through temporal, visual, and prompt-framing perturbations; Task A probes action-conditioned transition use; the composite–split comparison probes representation exchangeability; and Task D probes causal sensitivity through matched Full/CF comparisons. Tasks B and E are local perception and state-tracking controls that help interpret higher-level failures. This coverage is diagnostic rather than exhaustive: a judge that performs well on these tasks may still fail under different environments, longer contexts, ambiguous goals, or naturalistic video evidence.
NoCue interpretation. NoCue removes selected early visual evidence while preserving the underlying successful trajectory. It is therefore an evidence-removal diagnostic, not a semantics-preserving presentation transformation in the same sense as temporal order, visual style, prompt framing, or representation format. We use NoCue to ask whether a judge relies on early causal cues, but we do not treat NoCue stability alone as proof of presentation invariance.
3D extensions. The MiniWorld and Habitat experiments in Appendix § 13 provide supplementary 3D diagnostic evaluations. They should be interpreted separately from the Tier-0 MiniGrid results because the 3D environments introduce different perceptual and rendering factors. We do not aggregate MiniWorld, Habitat and MiniGrid results into a single headline score.
Threats to validity. We summarize three main validity threats. First, internal validity may be affected by parser ambiguity, API nondeterminism, or generator artifacts. We mitigate these risks by using constrained answer formats, deterministic parsing rules, fixed seeds where applicable, and validation gates over generated trajectories. Second, external validity is limited by the synthetic grid-world domain and by the finite set of evaluated models. The MiniWorld and Habitat extensions partially probe this issue but do not eliminate it. Third, construct validity depends on whether SCI captures the aspects of judge reliability that matter in downstream use. We view SCI as a necessary diagnostic principle: a reliable judge should not change its verdict merely because evidence is reformatted, and should change when a causal intervention changes the outcome. However, SCI is not sufficient for deployment safety, because real settings may involve ambiguous goals, incomplete observations, distribution shift, and human factors not represented here.
Societal Impact and Outlook
Table 3.1 summarizes the main impact considerations and recommended mitigations for responsible release and use.
Positive impacts. VLM judges are increasingly used to evaluate multimodal and embodied systems. A diagnostic benchmark such as GridWM-Judge can help researchers identify when a judge is correct for the wrong reason, unstable under benign presentation changes, or insensitive to causal outcome changes. This may reduce overreliance on opaque aggregate accuracy and encourage more careful reporting of judge reliability, especially in evaluation pipelines that influence model selection, dataset curation, or simulated safety testing.
Risks of overclaiming and misuse. The main risk is not that the benchmark directly enables harmful physical actions, since it uses synthetic grid-world data and releases no personal, private, or safety-critical operational data. The more realistic risk is overclaiming: users may treat high GridWM-Judge scores as a guarantee that a VLM judge is safe for real-world robotics or high-stakes decision-making. This would be inappropriate. The benchmark should be used as one diagnostic layer, not as a deployment certificate. Another risk is benchmark gaming: once examples and prompts are public, models or systems may be tuned to exploit the benchmark without genuinely improving causal grounding. Future releases should therefore include held-out diagnostic slices and versioned evaluation protocols.
| Consideration | Potential concern | Mitigation in this paper / release |
| Overclaiming | Treating SCI scores as deployment-readiness guarantees. | Explicitly state the controlled diagnostic scope and report limitations alongside results. |
| Benchmark gaming | Tuning models to public prompts or examples without improving real reliability. | Document prompts and probes; plan versioned splits and held-out diagnostic slices for future releases. |
| Misuse of judge diagnostics | Crafting inputs that exploit known judge weaknesses. | Frame results as reliability audits and avoid presenting the benchmark as an adversarial attack toolkit. |
| Privacy and data rights | Releasing sensitive images or trajectories. | Use simulator-generated synthetic data with no human subjects or private data. |
| Compute cost | Repeated VLM inference across many variants. | Provide modular subsets, deterministic scripts, and transparent compute reporting. |
Table 31. Impact considerations and recommended mitigations.
Responsible release. The dataset is synthetic and simulator-generated. It does not contain human-subject data, personal information, scraped images, or private trajectories. The release should nevertheless include documentation of generation scripts, prompt templates, schemas, validation logs, licenses for dependencies, and known limitations. We recommend that any public release state that the benchmark evaluates VLM judges under controlled MiniGrid, MiniWorld, and Habitat environments, and that it is not intended as a standalone safety evaluation for deployed embodied systems.
Environmental considerations. MiniGrid trajectory generation and validation are lightweight relative to large-scale VLM inference. The main compute footprint comes from repeated model calls over multiple tasks, variants, and presentation probes. The benchmark is intentionally modular so that researchers can run smaller diagnostic subsets when compute is limited. Reporting compute resources and inference settings in Appendix F is important for reproducibility and for understanding the environmental cost of evaluation.
Future directions. The most important next step is to test whether the same SCI failures persist in richer domains, such as longer MiniWorld tasks, photorealistic simulators, real robot videos, and embodied tasks with ambiguous or compositional goals. A second direction is to expand the model suite, including open-weight VLMs, video-language models, and specialized judge models. A third direction is to evaluate mitigation strategies such as structured output constraints, calibrated abstention, evidence localization, multi-view inputs, self-consistency, or training on counterfactual diagnostic data. Future benchmark versions should also include adversarially selected but simulator-valid trajectories to test whether apparent judge reliability survives more challenging causal and presentation shifts.
Conclusion on scope. GridWM-Judge provides reproducible diagnostics for examining whether model-based judges respond appropriately to presentation changes and causal interventions. Its findings are bounded by the evaluated domains, models, and protocols, and should not be interpreted as deployment or safety guarantees.
References
- [1]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 46595–46623, 2023.arxiv.org/abs/2306.05685
- [2]Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023). G-Eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 2511–2522).
- [3]Zhu, L., Wang, X., & Wang, X. (2025). JudgeLM: Fine-tuned large language models are scalable judges. International Conference on Learning Representations (ICLR).
- [4]Seongyun Lee, Seungone Kim, Sue Park, Geewook Kim, and Minjoon Seo. Prometheus-Vision: Vision-language model as a judge for fine-grained evaluation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 11286–11315, 2024.arxiv.org/abs/2401.06591
- [5]Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. MLLM-as-a-judge: Assessing multimodal LLM-as-a-judge with vision language benchmark. In Proceedings of the 41st International Conference on Machine Learning (ICML), volume 235, pages 6562–6595, 2024.arxiv.org/abs/2402.04788
- [6]Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, and Bill Yuchen Lin. WildVision: Evaluating vision-language models in the wild with human preferences. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pages 48224–48255, 2024.arxiv.org/abs/2406.11069
- [7]Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li. LLaVA-Critic: Learning to evaluate multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13618–13628, 2025.arxiv.org/abs/2410.02712
- [8]Lei Li, Yuancheng Wei, Zhihui Xie, Xuqing Yang, Yifan Song, Peiyi Wang, Chenxin An, Tianyu Liu, Sujian Li, Bill Yuchen Lin, Linpeng Kong, and Qi Liu. VL-RewardBench: A challenging benchmark for vision-language generative reward models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24657–24668, 2025.arxiv.org/abs/2411.17451
- [9]Zhuge, M., Zhao, C., Ashley, D., Wang, W., Khizbullin, D., Xiong, Y., Liu, Z., Chang, E., Krishnamoorthi, R., Tian, Y., & al., . et . (2025). Agent-as-a-judge: Evaluate agents with agents. In Proceedings of the 42nd International Conference on Machine Learning (ICML).
- [10]Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative judge for evaluating alignment. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2310.05470
- [11]Kim, S., Shin, J., Cho, Y., Jang, J., Longpre, S., Lee, H., Yun, S., Shin, S., Kim, S., Cho, J. T., Park, J., & Moon, S.-J. (2024). Prometheus: Inducing fine-grained evaluation capabilities in language models. International Conference on Learning Representations (ICLR).
- [12]Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., & Hashimoto, T. B. (2023). Alpacaeval: An automatic evaluator of instruction-following models. <a href="https://doi.org/https://github.com/tatsu-lab/alpaca_eval" rel="noopener noreferrer" target="_blank">https://doi.org/https://github.com/tatsu-lab/alpaca_eval</a>
- [13]Michihiro Yasunaga, Luke Zettlemoyer, and Marjan Ghazvininejad. Multimodal RewardBench: Holistic evaluation of reward models for vision language models. arXiv preprint arXiv:2502.14191, 2025.arXiv
- [14]Di Zhang, Jingdi Lei, Junxian Li, Xunzhi Wang, Yujie Liu, Zonglin Yang, Jiatong Li, Weida Wang, Suorong Yang, Jianbo Wu, Peng Ye, Wanli Ouyang, and Dongzhan Zhou. Critic-V: VLM critics help catch VLM errors in multimodal reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9050–9061, 2025.arxiv.org/abs/2411.18203
- [15]Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Kong, L., Liu, Q., Liu, T., & al., . et . (2024). Large language models are not fair evaluators. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), Volume 1: Long Papers, 9440–9450.
- [16]Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2309.03882
- [17]Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. Verbosity bias in preference labeling by large language models. arXiv preprint arXiv:2310.10076, 2023.arxiv.org/abs/2310.10076
- [18]Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, and Ion Stoica. JudgeBench: A benchmark for evaluating LLM-based judges. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2410.12784
- [19]Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Ridwan Mahbub, Ahmed Masry, Mizanur Rahman, Amran Bhuiyan, Mir Tafseer Nayeem, Shafiq Joty, Enamul Hoque, and Jimmy Xiangji Huang. Judging the Judges: Can large vision-language models fairly evaluate chart comprehension and reasoning? In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), Volume 6: Industry Track, pages 1203–1216, 2025.arxiv.org/abs/2505.08468
- [20]Ming Liu and Wensheng Zhang. Is your video language model a reliable judge? In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2503.05977
- [21]Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. HallusionBench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14375–14385, 2024.arxiv.org/abs/2310.14566
- [22]Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 292–305, 2023.arxiv.org/abs/2305.10355
- [23]Wang, J., Zhou, Y., Xu, G., Shi, P., Zhao, C., Xu, H., Ye, Q., Yan, M., Zhang, Y., Zhu, J., & Zhang, J. (2023). Evaluation and analysis of hallucination in large vision-language models. arXiv Preprint.arXiv
- [24]Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024.arxiv.org/abs/2307.03172
- [25]Wang, X., Zhou, Y., Liu, X., Lu, H., Xu, Y., He, F., Yoon, J., Lu, T., Liu, F., Bertasius, G., & al., . et . (2024). Mementos: A comprehensive benchmark for multimodal large language model reasoning over image sequences. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), Volume 1: Long Papers, 416–442.
- [26]Zou, X., Sridhar, R., Safarzadeh, M., & Roth, D. (2026). When vision-language models judge without seeing: Exposing informativeness bias. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL), Volume 1: Long Papers, 15417–15448.
- [27]Zichao Li, Xueru Wen, Jie Lou, Yuqi Ji, Yaojie Lu, Xianpei Han, Debing Zhang, and Le Sun. The devil is in the details: Tackling unimodal spurious correlations for generalizable multimodal reward models. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267, pages 36281–36297, 2025.arxiv.org/abs/2503.03122
- [28]Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. BabyAI: First steps towards grounded language learning with a human in the loop. In International Conference on Learning Representations (ICLR), 2019.
- [29]Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo Perez-Vicente, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & Miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 73383–73394, 2023.arxiv.org/abs/2306.13831
- [30]Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. CLEVRER: collision events for video representation and reasoning. In International Conference on Learning Representations (ICLR), 2020.arxiv.org/abs/1910.01442
- [31]Qiji Zhou, Yifan Gong, Guangsheng Bao, Hongjie Qiu, Jinqiang Li, Xiangrong Zhu, Huajian Zhang, and Yue Zhang. Reasoning is all you need for video generalization: A counterfactual benchmark with sub-question evaluation, 2025. Findings of ACL 2025.arxiv.org/abs/2503.10691
- [32]Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In Proceedings of the 36th International Conference on Machine Learning (ICML), 2019.arxiv.org/abs/1811.04551
- [33]Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations (ICLR), 2020.arxiv.org/abs/1912.01603
- [34]Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for continuous control. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2310.16828
- [35]Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., Lillicrap, T., & Silver, D. (2020). Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839), 604–609.
- [36]Bruce, J., Dennis, M. D., Edwards, A., Parker-Holder, J., Shi, Y., Hughes, E., Lai, M., Mavalankar, A., Steigerwald, R., Apps, C., & al., . et . (2024). Genie: Generative interactive environments. Proceedings of the 41st International Conference on Machine Learning (ICML), 235, 4603–4623.
- [37]Parker-Holder, J., Ball, P., Bruce, J., Dasagi, V., Holsheimer, K., Kaplanis, C., Moufarek, A., Scully, G., Shar, J., Shi, J., Spencer, S., Yung, J., Dennis, M., Kenjeyev, S., Long, S., Mnih, V., Chan, H., Gazeau, M., Li, B., … Rocktäschel, T. (2024). Genie 2: A large-scale foundation world model. Google DeepMind blog. https://deepmind.google/blog/genie-2-a-large-scale-foundation-world-model/deepmind.google/blog/genie-2-a-large-scale-foundation-world-model
- [38]Shaofeng Yin, Jialong Wu, Siqiao Huang, Xingjian Su, Xu He, Jianye Hao, and Mingsheng Long. Trajectory world models for heterogeneous environments. In Proceedings of the 42nd International Conference on Machine Learning (ICML), pages 72462–72484, 2025.arxiv.org/abs/2502.01366
- [39]Duan, H., Yu, H.-X., Chen, S., Fei-Fei, L., & Wu, J. (2025). WorldScore: A unified evaluation benchmark for world generation. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 27713–27724.
- [40]Archana Warrier, Dat Nguyen, Michelangelo Naim, Moksh Jain, Yichao Liang, Karen Schroeder, Cambridge Yang, Joshua B. Tenenbaum, Sebastian Josef Vollmer, Kevin Ellis, and Zenna Tavares. Benchmarking world-model learning with environment-level queries. In Proceedings of the 43rd International Conference on Machine Learning (ICML), volume 306, pages 133215–133246, 2026.arxiv.org/abs/2510.19788
- [41]Weijiang Li, Yilin Zhu, Rajarshi Das, and Parijat Dube. Do LLMs build spatial world models? evidence from grid-world maze tasks. In ICLR 2026 the 2nd Workshop on World Models: Understanding, Modelling and Scaling, 2026.arxiv.org/abs/2604.10690
- [42]Alex F Spies, William Edwards, Michael I Ivanitskiy, Adrians Skapars, Tilman Räuker, Katsumi Inoue, Alessandra Russo, and Murray Shanahan. Transformers use causal world models in maze-solving tasks. In ICLR 2025 Workshop on World Models: Understanding, Modelling and Scaling, 2025.arxiv.org/abs/2412.11867
- [43]Raghu Arghal, Phoebe Chen, Niall Dalton, Evgenii Kortukov, Calum Mcnamara, Angelos Nalmpantis, Moksh Nirvaan, Gabriele Sarti, and Mario Giulianelli. A behavioural and representational evaluation of goal-directedness in language model agents. In Proceedings of the 43rd International Conference on Machine Learning (ICML), volume 306, pages 3434–3451, 2026.arxiv.org/abs/2602.08964
- [44]Qiyue Gao, Xinyu Pi, Kevin Liu, Junrong Chen, Ruolan Yang, Xinqi Huang, Xinyu Fang, Lu Sun, Gautham Kishore, Bo Ai, Stone Tao, Mengyang Liu, Jiaxi Yang, Chao-Jung Lai, Chuanyang Jin, Jiannan Xiang, Benhao Huang, Zeming Chen, David Banks, Hao Su, Tianmin Shu, Ziqiao Ma, Lianhui Qin, and Zhiying Hu. Do vision-language models have internal world models? Towards an atomic evaluation. In Findings of the Association for Computational Linguistics: ACL 2025, pages 26170–26195, 2025.arxiv.org/abs/2506.21876
- [45]Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019.arxiv.org/abs/1907.02893
- [46]Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society Series B: Statistical Methodology, 78(5):947–1012, 2016.arxiv.org/abs/1501.01332
- [47]Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1703.06856
- [48]Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020.DOI
- [49]Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. MineDojo: Building open-ended embodied agents with internet-scale knowledge. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 18343–18362, 2022.arxiv.org/abs/2206.08853
- [50]Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. URL https://arxiv.org/abs/2308.12966.
- [51]Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023). Visual instruction tuning. Advances in Neural Information Processing Systems (NeurIPS), 36, 34892–34916.
- [52]Wang, W., Lv, Q., Yu, W., Hong, W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L., Song, X., & al., . et . (2024). CogVLM: Visual expert for pretrained language models. Advances in Neural Information Processing Systems (NeurIPS), 37, 121475–121499.
- [53]Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., & al., . et . (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems (NeurIPS), 35, 27730–27744.













