WorksComputer Science
Phantom Success, Silent Failure: Auditing the Success Checkers of Simulated Manipulation Benchmarks
Abstract
Simulated manipulation benchmarks measure policy performance using hand-written success checkers that label each episode as a success or failure. Passing an expert demonstration shows that a checker accepts one way of completing the task. Learned policies, however, may stop after partial progress or complete the task in a different way. A checker may then accept an incomplete execution (phantom success) or reject an execution that satisfies the instruction (silent failure). In this paper, we introduce SimAudit, an LLM-based pipeline that audits success checkers against task instructions. SimAudit analyzes the instruction and checker code to identify scenarios where the two may disagree, recreates those scenarios in simulation, and runs the original checker. Multi-view renderings are then used to verify whether the observed behavior satisfies the instruction. Across RoboTwin 2.0, RoboCasa365, and RoboDojo, 183 simulated test cases produced the predicted checker verdict. Visual verification and deduplication yielded 61 distinct defects across 49 tasks, including 40 phantom successes and 21 silent failures. Most defects in RoboCasa365 accepted incomplete executions, most in RoboDojo rejected valid executions, and both error types were common in RoboTwin 2.0. These findings suggest that validating success checkers requires testing both incomplete executions and valid alternatives to expert demonstrations, and motivate more rigorous evaluation protocols for future benchmarks.
Introduction
Task success rate is the primary metric for comparing robot manipulation policies in simulation. Each episode is scored by a success checker, a hand-written rule that uses simulator state to determine whether the robot completed the task specified by the instruction. Evaluating generalist manipulation models, including vision-language-action models ([4, 3] (Kim et al. 2024; Song et al. 2026b; Lu et al. 2026b)) and world action models ([2] (Li et al. 2025; Lin et al. 2026)), on physical robots is costly and difficult to reproduce across laboratories ((Zhou et al., 2025; Song et al., 2026a)). Recent simulated benchmarks such as RoboTwin 2.0 ([6]), RoboCasa365 ((Nasiriany et al., 2026)), and RoboDojo ([7]) therefore play a central role in model evaluation. They aggregate checker verdicts into the success rates used to rank policies. Prior work has shown that these rates can be affected by memorization of training configurations ([11] (Zhou et al. 2026b)), task shortcuts and overfitting to fixed test sets ([16]), and artifacts in the physics engine ((Zhou et al., 2026a)). The reliability of the success checkers that produce these rates has received less attention. This raises a basic question: do checker verdicts accurately reflect whether a robot’s execution satisfies the task instruction?
Expert demonstrations can establish that a success checker accepts a reference execution. RoboTwin 2.0 ([6]), for example, retains an evaluation seed only if the scripted expert passes the checker on that seed. A learned policy can reach states outside those covered by the demonstrations by stopping before the task is complete or by completing it in a different way. Figure 1 illustrates two errors that can arise in these states. The first task illustrates a phantom success. The checker reports success when the robot lifts the bottle 3 cm off the table without shaking it. A phantom success occurs when the checker marks an execution as successful even though it does not satisfy the instruction. The second task illustrates a silent failure. The robot picks up one bottle with each arm as instructed, but the checker reports failure because it additionally requires both bottles to be at the centre of the table. A silent failure occurs when the checker marks an execution as failed even though it satisfies the instruction.

Figure 1. A checker validated on the expert can be wrong elsewhere. On the expert demonstration (left), the checker agrees with the instruction. On other executions (right), the same checker accepts a bottle that is lifted but not shaken, and rejects two bottles that are picked up away from the centre.
Validation with scripted experts can miss checker errors because expert trajectories may never expose discrepancies between the instruction and the conditions the checker tests. To uncover these errors, we introduce SimAudit, an LLM-based pipeline that audits success checkers on states that scripted experts never reach. Given a task instruction and its checker code, an LLM agent compares the requirements in the instruction with the conditions tested by the checker. It uses discrepancies between the two to propose states that the checker may misclassify. For each proposed state, the agent predicts from the code whether the checker will return success or failure. The agent then constructs the state in simulation, using robot motion governed by simulated physics wherever possible. The pipeline runs the unmodified checker on the resulting state and renders the scene from multiple camera views. A candidate is retained only if the rendered views establish that the execution’s outcome under the instruction differs from the checker verdict. Across 142 tasks in the three benchmarks, the agent proposed 568 candidate states. The actual checker verdict matched the agent’s prediction in 183 cases. Visual verification and deduplication yielded 61 distinct checker defects.
We identify 61 defects across 49 tasks in RoboTwin 2.0, RoboCasa365, and RoboDojo, including 40 phantom successes and 21 silent failures. RoboCasa365 has 17 phantom successes and one silent failure, while RoboDojo has three phantom successes and nine silent failures. Both types are common in RoboTwin 2.0, with 20 phantom successes and 11 silent failures. These patterns reflect the conditions each benchmark’s checkers enforce. RoboCasa365 checkers often omit conditions required by the instruction, allowing incomplete executions to pass. RoboDojo checkers often impose conditions beyond those in the instruction, causing executions that satisfy the instruction to fail. These errors can inflate or depress reported success rates. To measure their effect on released policies, we repair the defects identified by SimAudit and score the same policy rollouts with the original and repaired checkers. We count a verdict change whenever the two checkers disagree on an episode.
Our contributions are threefold. First, we define two types of success checker errors, phantom success and silent failure, and illustrate both with rendered examples from states outside expert trajectories. Second, we introduce SimAudit, an agentic pipeline that proposes candidate error cases from task instructions and checker code, constructs them in simulation, and verifies errors by comparing the unmodified checker’s verdict with renders from multiple camera views. Third, we report 61 visually verified defects across three benchmarks, with different proportions of the two error types. After repairing the defects, we score the same rollouts from released policies with both checkers. The original and repaired checkers disagree on up to 9.1% of a policy’s rollouts.
Related Work
Auditing manipulation benchmarks. Recent audits of simulated manipulation benchmarks [15] (Yu et al. 2021; Mees et al. 2022; Tao et al. 2025; Pumacay et al. 2024; Zhang et al. 2024; Nasiriany et al. 2024; Li et al. 2024a; Lu et al. 2026a) examine two things. The first is the task. LIBERO-PRO (Zhou et al., 2026b) and LIBERO-Plus [11] perturb objects, layouts and instructions, and Jiang et al. [16] test five suites for shortcut solvability, statistical significance, overfitting and data-source dependence. The second is the simulator (Li et al., 2024b). SimuGuard (Zhou et al., 2026a) traces contact artifacts in RoboTwin to one constant in the physics engine and shows that success on container tasks rises in a repaired engine. SimuGuard also notes in passing that RoboTwin’s checks read the commanded gripper value, and we measure what that choice does to verdicts. These audits use the benchmarks’ own success checkers to measure how changes to tasks or simulator physics affect reported success rates. This approach assumes that the checker verdicts accurately reflect whether the task was completed. We test that assumption directly by holding the task and simulator fixed and checking whether the verdicts agree with the task instructions.
Auditing benchmarks for software agents. Auditing how benchmarks (Zhou et al., 2024; Xie et al., 2024; Yao et al., 2024) score agent outputs is an established practice in software-agent evaluation [1] (Liu et al. 2023a). The Agentic Benchmark Checklist (Zhu et al., 2025) identifies outcome validity, the extent to which passing an evaluation check indicates that the task was completed, as a core requirement. UTBoost (Yu et al., 2025) augments the test suites in SWE-bench [17] and changes the scores of 24–41% of leaderboard entries. SWE-bench Verified (OpenAI, 2024) improves evaluation reliability through task curation, and recent work extends automated auditing to a broader range of benchmarks (Wang et al., 2026b,a). Software-agent audits can use artifacts such as execution logs and test results to assess disputed outcomes. In simulated manipulation, testing a suspected checker defect may first require constructing an execution that exposes it. For example, determining whether a robot shook a bottle requires evidence of its motion over time. SimAudit constructs targeted cases in simulation, uses physics-based robot motion where possible, and compares the original checker’s verdict with evidence from multi-view renderings.
Success checkers in manipulation. Some work replaces the hand-written binary checker ([9, 10] (Rocamonde et al. 2024; Liu et al. 2023b)). RoboEval (Wang et al., 2026c) adds process metrics, and AutoEval (Zhou et al., 2025) uses learned judges, which have errors and biases of their own ([13] (Xu et al. 2026)): the best VLM-based detector in FailBench reaches 0.77 balanced accuracy (Navasardyan et al., 2026). Leaderboards still count the verdicts of hand-written or LLM-written checkers ((Wang et al., 2024b,a)), and a survey of verifiers for robot policies (Wan et al., 2026) observes that their error rate is reported far less often than the success rate they produce. The study most closely related to ours is a manual audit of LIBERO and its variants, as well as RoboTwin, RoboCasa, and VLABench ([5]). The audit identifies 22 bugs and four design limitations. Examples include unrealistic object masses, tasks that can pass without a grasp, and success checkers that impose requirements beyond the task instructions, such as a specific bottle orientation in the “adjust bottle” task. We differ from it in three ways. First, our automated search covers every task across three benchmarks, including RoboDojo, which was outside the scope of the prior audit. Second, each defect is documented with rendered evidence and can be reproduced by running the unmodified checker on the same simulated state. Third, we quantify the direction of checker errors in each benchmark and measure how many verdicts for released policies change after the checkers are repaired.
Auditing Success Checkers
Phantom success and silent failure
A task comes with an instruction and a checker, a function of the simulator state that returns success or failure. The benchmark ends the episode at the first state on which the checker returns success and records a success; if none occurs before the time limit, it records a failure. A phantom success is credit for an execution that does not fulfil the instruction: the robot does part of the task, does something else, or only passes through the accepted state. A silent failure is credit withheld from an execution that fulfils it: the robot completes the task by a route, or to an end state, that the instruction allows and the checker does not. A person who reads the instruction and looks at the scene decides whether it is fulfilled. Both are judged on the state at which the benchmark gives its verdict; what the robot would have done after the episode ends does not count.
A checker defect is a flaw in a checker together with a state that shows it. We count a defect when three conditions hold. (i) The state is rendered, and the disagreement between the checker and the person is clear from the pictures. (ii) The checker value is read from the unmodified benchmark code at that state. (iii) The state is reachable: it is produced by motions of the robot or of the scripted expert under physics, or it is a pose in which an object rests after it is released and left to settle. A state that exists only while an object is teleported into place, such as an object floating without support or two objects interpenetrating, shows what the code computes but is not counted. Each defect is counted once: several states that show the same flaw in the same task are one defect.
SimAudit: read, guess, stage, review
SimAudit is an automated pipeline that audits a benchmark’s success checkers against its task instructions. Given a task, it constructs targeted test executions, records the original checker’s verdicts, and uses rendered evidence to verify potential mismatches. We illustrate the pipeline in Figure 2. The framework could be divided into four stage as following.

Figure 2. SimAudit workflow. (A) An LLM agent reads the task instruction and checker code. (B) It proposes cases likely to expose checker errors. (C) A probe in the unmodified simulator triggers a success verdict while the mouse is still held above the stand. (D) A separate VLM reviewer uses multi-view renderings to confirm that the task is incomplete, identifying a phantom success.
Read. SimAudit starts by establishing what the task asks for and what the checker tests. An LLM agent receives the instruction templates given to the policy, the source of the task with its success checker and every helper it calls, the benchmark’s evaluation loop, and the scripted expert when the benchmark provides one. The agent first writes, from the instruction alone, what has to be true of the scene for the task to count as done. It then writes, from the code alone, what the checker accepts. We ask for the two descriptions separately so that the agent’s reading of the task is not taken from the checker’s conditions. As shown in Figure 2A, the instruction of “place object stand” asks for the object to be placed on the stand, and the checker accepts any state in which the object is within 3 cm of the stand horizontally and both gripper commands are open; it tests neither the height of the object nor its contact with the stand.
Guess. Given the two descriptions, the agent proposes up to four hypotheses for the task. Each hypothesis names a state, the verdict the checker will give on it, and the opposite judgement of a person who reads the instruction, together with the line of code responsible. The agent also states how the state can be built and which measurement would confirm the hypothesis, so that the next stage has a prediction to test. As shown in Figure 2B, the agent writes four hypotheses for “place object stand”. The object is held above the stand and never set down, the open command is sent while the fingers are still closed, the object is set down off-centre but within the tolerance, or the stand is pushed along with the object.
Stage. A hypothesis is a claim about code until the state exists. For each hypothesis the agent writes a probe, a short program against a small API that can run the scripted expert or individual robot motions under physics, set a gripper command, move an object and let it settle, step the simulator, and read the checker. The API exposes the benchmark’s own motions so that, wherever possible, a state is reached the way a policy would reach it. The probe runs in the unmodified simulator. The harness records the checker value and the first step at which it is true. The agent then reads the measurements; if the probe failed or did not produce the intended state, the agent revises it. A hypothesis whose probe returns the predicted checker value is agent-confirmed. As shown in Figure 2C, the probe for the first of these hypotheses grasps a mouse with the benchmark’s own motions, carries it over the stand, lifts it and commands the gripper open. The checker is false before the command and true on the next physics step, and the episode ends with the mouse still in the gripper, 21 cm above the stand.
Review. An agent-confirmed hypothesis rests on a value returned by a probe, and the probe was written by the agent whose hypothesis it tests. The value does not show what the scene looks like: a probe can place an object below the table top, or describe a bottle as lying down when the asset’s axis makes it stand. SimAudit therefore re-runs every agent-confirmed probe with rendering. Cameras aimed at the manipulated objects capture the scene after every teleport, at the first success step, again after a short wait to make sure the accepted state persists, whenever the checker value flips, and at the end. A reviewer that did not write the probe sees the instruction, the frames and the checker value at each frame, and never the code, so that its judgement of the task cannot follow the checker’s logic. It assigns one of four labels. valid: the frames show the disagreement that the hypothesis claims, on a reachable state. unrealistic: the checker behaves as claimed, but the state exists only by teleportation. wrong: the frames contradict the claim, or the state the checker accepts is a reasonable success. not shown: the frames do not show the claimed state. Only valid items are counted. As shown in Figure 2D, the reviewer VLM sees a mouse held in the air and a checker value of success, answers that the task is not done, and labels the item valid.
From defects to Verified checkers
A defect confirmed by SimAudit comes with the state that shows it and the line of code responsible. This is what a repair needs. We repair each checker so that its verdict on that state agrees with the instruction. The tasks, assets and episode protocol stay as released. Thresholds stay as released unless a defect concerns the threshold itself. We call the result a Verified checker.
A checker reaches its verdict in three steps. It measures quantities of the scene, it tests conditions on these quantities, and it decides at which moment the verdict is final for the episode. A defect enters at one of these steps. The checker measures the wrong quantity, it tests too few or too many conditions, or it gives its verdict too early. We therefore repair a checker step by step, in the order in which the steps depend on each other.
Measure the physical state. A condition is only as reliable as the quantity it is tested on, so the repair starts with the measurement. Some simulation benchmarks decide success from a quantity that stands in for the state of the scene. For example, a checker may read the commanded gripper value to decide whether an object has been released. The Verified checker measures the physical state directly. A gripper holds an object when its links exert force on it, and has released it when they do not. For example, the original checker accepts the mouse held above the stand as soon as the open command is sent. The Verified checker rejects this state, because the fingers still press on the mouse.
Match the conditions to the instruction. Correct measurements are not enough when the conditions tested on them differ from the instruction. The difference has two directions, and each produces one of the two errors. A checker that tests less than the instruction states gives phantom successes, so the Verified checker adds the missing condition. For example, a bottle must be shaken to count as shaken. A checker that tests more than the instruction states gives silent failures, so the Verified checker removes every condition that the instruction does not mention. For example, an instruction that only asks the robot to pick up an object does not require the object to end at a fixed target position.
Require success to persist. A checker with correct measurements and correct conditions can still give its verdict too early. The conditions can hold for a few steps while an object falls or a stack topples, and the episode ends at the first of these steps. The Verified checker therefore counts a success only after the repaired conditions have held for one second. For example, three blocks are stacked with an overhang, and the original checker accepts the stack while it is already toppling. The Verified checker rejects this state, because the stack does not stay up for one second.
Validation. A repair can go wrong in two ways, and we test for both. First, it can make a task harder than the instruction requires. The scripted expert completes every task as the benchmark intends, so it must still pass after repair. The Verified checker accepts 1,233 expert episodes. Second, a repair can leave the defect open. The rendered states that revealed the 61 defects therefore serve as a regression suite. In all of these tests the Verified checker runs as an observer next to the original. It reads the simulator state at every step at which the original checker is evaluated and never changes it.
Benchmark Audit Results
Audit Setup
We audit three benchmarks: RoboTwin 2.0 [6], RoboCasa365 (Nasiriany et al., 2026) and RoboDojo [7]. All checker values come from unmodified benchmark code. We assign each defect one primary cause. A checker can omit a condition that the instruction requires, or add a condition that the instruction does not state. It can read an indirect signal instead of the physical state, or accept a region that is too large. It can also give its verdict before the scene settles. We call these five causes missing condition, unstated requirement, proxy signal, loose geometry and transient success, as detailed in Appendix § 8.
Audit Findings Across Benchmarks
Table 1 gives the main result, and Figure 3 shows the defects at the state where the original checker gives its verdict. The three benchmarks do not share a failure profile, as detailed in Appendix § 9.

Figure 3. Verified checker defects across three benchmarks. Each tile pairs the task instruction with the observed state at the checker’s verdict. Phantom successes appear above and silent failures below. cmd denotes the commanded gripper value. The tiles show 51 defects, and the inset counts all 61, including ten additional RoboTwin 2.0 successes reported while the object was still held.
| RoboTwin 2.0 | RoboCasa365 | RoboDojo | Total | |
| Defects | 31 | 18 | 12 | 61 |
| Tasks with a defect | 25 | 13 | 11 | 49 |
| By direction of the error | ||||
| Phantom success | 20 | 17 | 3 | 40 |
| Silent failure | 11 | 1 | 9 | 21 |
| By primary cause | ||||
| Missing condition | 10 | 7 | 0 | 17 |
| Unstated requirement | 5 | 1 | 9 | 15 |
| Proxy signal | 15 | 0 | 1 | 16 |
| Loose geometry | 0 | 8 | 2 | 10 |
| Transient success | 1 | 2 | 0 | 3 |
caption: Verified checker defects in the three benchmarks. The defects are split first by the direction of the error and then by their primary cause.
:::
RoboTwin 2.0 errs in both directions. Its main cause is the proxy signal, as detailed in Appendix § 10. The checkers read the commanded gripper value instead of the state of the fingers. A placement is credited when the open command is issued with the object still in the gripper, and a finished grasp is rejected when the command stops short of a threshold. RoboCasa365 over-credits. Most of its defects are loose geometry or missing conditions. For example, a straw lying across the rim of a cup counts as inside the cup. RoboDojo under-credits. Most of its defects are silent failures, and all of these are unstated requirements. For example, the digit 8 is rejected after a half turn that the instruction does not mention and the picture does not show. The statement that simulated success rates are inflated therefore holds for RoboCasa365, holds in part for RoboTwin 2.0, and is reversed for RoboDojo. The cause of a defect fixes its direction. Missing conditions, loose geometry and transient successes are all phantom successes, and all unstated requirements are silent failures. The proxy signal is the one cause that errs in both directions, with 10 phantom successes and 6 silent failures.
Evaluation of SimAudit
We evaluate SimAudit by asking independent annotators to verify the reported defects and examining which candidates are filtered out during visual review.
Precision. Two annotators independently examine each reported defect. They see the instruction, the rendered frames and the checker’s verdict, and they decide whether the verdict is wrong. Both annotators confirm every defect.
Effect of the review stage. Of the 183 candidates for which the checker returned the predicted verdict, approximately two-thirds were rejected during visual review. Appendix § 11 breaks down the reasons for rejection. The annotators then examined a random sample of 50 rejected candidates and found no valid defects among them.
Policy Re-evaluation with Verified Checkers
The defects above are states we constructed. We now ask how often released policies reach them, and whether the repair changes how the policies rank. On RoboTwin 2.0 we evaluate five policies with public weights: LingBot-VA (Li et al., 2026), LingBot-VLA 2.0 (Wu et al., 2026), Evo-1 (Lin et al., 2025), X-VLA (Zheng et al., 2025) and [14]. On RoboCasa365 we evaluate GR00T N1.5 (NVIDIA et al., 2025) fine-tuned on 100% and on 30% of the training data, Xiaomi-Robotics-1 (Team et al., 2026) and REKI-2B (Marrie et al., 2026) fine-tuned on RoboCasa365, and the and [3] checkpoints released with the benchmark. On RoboDojo we evaluate Rex-M1, VPP [12], Awomo-0.5 (Team, 2026), Xiaomi-Robotics-1 and two StarVLA (Community, 2026) models with different action heads QwenPI and QwenOFT. Each policy runs with its released inference code under the benchmark’s official protocol, as detailed in Appendix § 12.
We score each rollout with both the original and Verified checkers. The original checker records an episode as successful at its first success verdict. During re-evaluation, policy execution continues beyond this point, and the Verified checker can reach success on subsequent states of the same rollout. An episode accepted by the original checker but rejected by the Verified checker is counted as a phantom success; the reverse is counted as a silent failure.
Before and after the repair. Figure 4 compares the two checkers on four RoboTwin 2.0 rollouts. Each row shows the frames of one episode, and the bars under the frames show the value of each checker over the time axis. In the first row, X-VLA lowers the gripper towards the bell and stops above it. The original checker turns true, so the benchmark records a success and ends the episode. The

Figure 4. Checker verdicts on four RoboTwin 2.0 rollouts. Each row shows a rollout scored by the original and Verified checkers; green denotes a true verdict. The top three rows show phantom successes: the gripper stops above the bell or beside the stapler, or holds the roller with one gripper. The bottom row shows a silent failure: the stapler is placed on the mat, yet the original checker rejects it for a yaw requirement absent from the instruction.
Verified checker is never true, because the gripper never touches the bell. The second and third rows follow the same pattern. Evo-1 stops beside the stapler without touching it, and holds the roller with one gripper although the instruction asks for both. In the fourth row the comparison is reversed. places the stapler on the mat and releases it. The original checker stays false until the limit because it is turned by , so the benchmark records a failure. The Verified checker turns true as soon as the stapler rests on the mat.
Verdicts change on every benchmark. Rescoring the same rollouts with the Verified checker changed episode verdicts across all three benchmarks and for nearly every policy. The share of episodes affected varied by policy, with the largest change observed for VPP on RoboDojo. Verdicts remained unchanged only for and on RoboCasa365. As Table 2 shows, the share of episodes with changed verdicts can exceed the net change in success rate because some episodes gain credit while others lose it.
| Policy | Orig. | Verif. | Phan. succ. | Silent fail. | Flip | |
| RoboTwin 2.0 | ||||||
| LingBot-VA | 93.1 | 90.0 | -3.1 | 3.3 | 0.0 | 3.1 |
| LingBot-VLA 2.0 | 93.9 | 90.8 | -3.1 | 3.4 | 1.5 | 3.3 |
| Evo-1 | 68.3 | 65.5 | -2.7 | 5.8 | 3.9 | 5.2 |
| X-VLA | 64.9 | 61.6 | -3.4 | 6.2 | 2.0 | 4.7 |
| 46.2 | 42.3 | -3.9 | 10.2 | 1.6 | 5.6 | |
| RoboCasa365 | ||||||
| GR00T N1.5 (100%) | 41.2 | 38.8 | -2.4 | 5.8 | 0.0 | 2.4 |
| GR00T N1.5 (30%) | 37.7 | 33.8 | -3.8 | 12.2 | 1.2 | 5.4 |
| Xiaomi-Robotics-1 | 38.2 | 34.8 | -3.4 | 12.5 | 2.2 | 6.1 |
| REKI-2B | 29.8 | 28.2 | -1.5 | 5.1 | 0.0 | 1.5 |
| RoboDojo | ||||||
| Rex-M1 | 35.5 | 40.0 | +4.5 | 0.0 | 7.0 | 4.5 |
| VPP | 30.0 | 39.1 | +9.1 | 0.0 | 13.0 | 9.1 |
| Awomo-0.5 | 20.0 | 21.0 | +1.0 | 0.0 | 1.2 | 1.0 |
| Xiaomi-Robotics-1 | 10.9 | 11.8 | +0.9 | 0.0 | 1.0 | 0.9 |
| StarVLA (QwenPI) | 5.5 | 9.1 | +3.6 | 0.0 | 3.8 | 3.6 |
| StarVLA (QwenOFT) | 0.9 | 5.5 | +4.5 | 0.0 | 4.6 | 4.5 |
Table 2. Table 2: Re-scoring released policies on the same rollouts. is the Verified minus original success rate in percentage points. Phan. succ. and Silent fail. are the percentages of original successes and failures reversed, respectively. Flip is the percentage of all episodes whose verdict changes.
The direction follows the audit. The sign of the change on each benchmark matches the audit: success rates fall on RoboCasa365, rise on RoboDojo, and drop on RoboTwin 2.0 by less than the share of flipped verdicts, because the two errors cancel in part. As shown in Figure 5, the two errors occur in different tasks. Seven of the eight tasks lose successes under the Verified checker, and only “move stapler pad” gains them. On “click bell”, keeps none of the successes it is credited with, because the gripper stops above the bell and never touches it. The same defect does not affect all policies equally. On this task, LingBot-VA keeps all of its successes, and X-VLA falls from 95% to 20%. On “move stapler pad”, Evo-1 is credited with no success and succeeds in most episodes under the Verified checker, because the stapler is on the pad a few degrees off the yaw that the checker demands. A leaderboard that averages over tasks reports neither.
Figure 5. Per-task success rates. Bars show the original checker and lines the Verified checker on the same rollouts.
Checker repairs change the gaps between policies. Changes in episode verdicts were concentrated in a few tasks, so the effect of rescoring varied with how each policy performed on those tasks. On RoboTwin 2.0, the gaps between policies shifted only slightly. On RoboCasa365, both GR00T N1.5 variants scored lower under the Verified checker, with a larger drop for the variant trained on less data. The gap between the two variants therefore widened. On RoboDojo, VPP gained more from rescoring than either Rex-M1 or StarVLA (QwenPI), leaving it close to the leading policy and further ahead of StarVLA (QwenPI). As shown in Table 2, the rankings stayed the same on all three benchmarks, even as rescoring changed how closely some policies matched their nearest competitors.
Conclusion
Conclusion
Prior audits of simulated manipulation benchmarks have focused on task design and simulator fidelity, leaving the reliability of success checkers less explored. We introduced SimAudit to test these checkers beyond expert demonstrations using targeted simulations and multi-view renderings. Our audit uncovered both phantom successes and silent failures, and re-scoring released policy rollouts with repaired checkers showed that these defects affect reported success rates. We release SimAudit, the repaired checkers, and the rendered counterexamples to support future validations.
LLM Usage Statement
We used Claude Code for language editing, grammar correction, and LaTeX formatting of the manuscript, as well as for drafting code snippets and debugging scripts.
Defect taxonomy
Table 3 lists the five causes, grouped by the step of the verdict at which the checker diverges from the task instruction.
| Cause | The checker … |
| Specification: tests conditions that differ from the instruction | |
| Missing condition | omits a required condition, such as bottle motion in “shake bottle”. |
| Unstated requirement | adds a condition, such as a centre target in “pick dual bottles”. |
| Measurement: tests a condition on an unsuitable quantity | |
| Proxy signal | reads an indirect signal, such as the commanded gripper value for a grasp. |
| Loose geometry | uses a region or threshold that is too permissive, such as a straw across the rim of a cup counted as inside it. |
| Timing: gives its verdict too early | |
| Transient success | accepts a condition that holds briefly while an object falls, slides or topples. |
Table 3. Checker defect taxonomy. Five causes in three categories. Each category corresponds to one step of a checker’s verdict.
Defects of each benchmark
RoboTwin 2.0. The phantom successes come from missing conditions (10), the gripper proxy (9) and a transient success (1). The missing conditions are a shake that “shake bottle” never tests, an orientation that “adjust bottle” never tests, and a location for a stack that the instruction places at the centre. The silent failures come from the same proxy (6) and from unstated requirements (5).
RoboCasa365. Many defects trace to shared placement helpers that test contact and horizontal distance only, and every task that calls a helper inherits its tolerance. Fruit beside an overturned colander is in the colander; a mug upside down under the spout is ready for coffee.
RoboDojo. The most frequent unstated requirement is the orientation of an object that looks the same after a half turn: the digit 8, an operator tile, a tape measure.
Gripper commands and early termination
RoboTwin’s checkers decide whether a gripper is open by reading the last commanded value, and require it to be above 0.8 for a release or below 0.2 for a grasp (Zhou et al., 2026a). The command is not the state of the fingers, and the proxy fails in both directions: 9 phantom successes and 6 silent failures.
The silent failures arise because the thresholds are strict. In “place empty cup” the cup stands on its coaster and the arm has left, and the checker returns failure for as long as the command is 0.7. A policy that outputs continuous gripper actions and stops short of the extremes completes the task and is scored as failing.
The phantom successes arise because a command is issued before the fingers move, and because the episode ends too early to see them move. The checker turns true on the first physics step after the open command, with the fingers still closed on the object. The episode ends at that step, so this momentary agreement becomes the final verdict. In “place mouse pad” the mouse is recorded as placed while it is held above the mat, and in “stamp seal” the seal has not touched the mat.
Ten RoboTwin defects are successes recorded while the object is still in the gripper. Eight come from the gripper proxy. The other two are in “place bread skillet” and “put bottles dustbin”, where the checker has no release condition at all. To see what the recorded episode leaves out, we continued these ten probes past the first success step and kept the open command. In all ten the object ended in the correct place: it dropped a few centimetres into its slot, onto its plate or onto its mat, or stayed in the receiving hand. A policy that goes on to release is therefore credited early, and a policy that stops there is credited for a task it did not finish. The recorded episode is the same in both cases.
Early termination also hides the opposite case, in which the state is right at the first success step and wrong afterwards. In “stack blocks three”, three blocks are placed with an overhang that is within tolerance for each pair and accumulates across the stack. They pass the checker at the first step. Within 60 ms, the stack is falling and the checker value is false again, but the episode has already been recorded as a success.
Evaluation of SimAudit: details
Rejected states by label. On RoboTwin 2.0, the 91 agent-confirmed hypotheses split into 31 distinct defects, 1 repeat of one of them, 37 unrealistic, 19 wrong and 3 not shown.
Cost. Auditing one task costs $0.78 in LLM calls and takes 18 minutes on average over the 142 tasks we audited, including the time to run the probes in the simulator.
Policy re-evaluation details
Protocol. On RoboTwin 2.0, each policy runs with its released inference code on all 50 tasks of the clean setting, with the official seeds, expert filter and unseen instructions. For we use the checkpoint released with Motus [2].
Protocol on RoboCasa365 and RoboDojo. On RoboCasa365 we evaluate 13 tasks in which the audit found a defect. Every episode runs to the task horizon. On RoboDojo we use the released policy servers and evaluation client, and evaluate 11 tasks in which the audit found a defect, on layouts 0 to 9 of each task. On both benchmarks the two checkers score the same rollout, as on RoboTwin 2.0.
A few repairs account for most flips. Table 4 attributes each flipped verdict to the repair that caused it. The largest group is pressing without touching: in “click bell”, “click alarmclock” and “press stapler” the original checker accepts a contact point that the simulator reports across an air gap of up to 2 cm. The second group is success that does not last: a phone that tips over within a second of being placed on its stand. The flips concentrate in 4 to 19 of the 50 tasks, and they are largely the same tasks for every policy.
| Verdict change | VA | VLA | 2 | Evo-1 | X-VLA | |
| Press without contact | 6 | 23 | 20 | 24 | 19 | |
| Missing condition | 0 | 2 | 4 | 3 | 9 | |
| Command-based grasp | 0 | 0 | 5 | 3 | 7 | |
| Transient success | 2 | 9 | 13 | 11 | 15 | |
| Silent failures | 0 | 1 | 13 | 7 | 9 |
Table 4. Table 4: Verdict changes by repair type and policy. The first four rows count corrected phantom successes, and the fifth counts corrected silent failures.
References
- [1]Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. Swe-bench+: Enhanced coding benchmark for llms, 2024. URL https://arxiv.org/abs/2410.06992.
- [2]Hongzhe Bi, Hengkai Tan, Shenghao Xie, Zeyuan Wang, Shuhe Huang, Haitian Liu, Ruowen Zhao, Yao Feng, Chendong Xiang, Yinze Rong, Hongyan Zhao, Hanyu Liu, Zhizhong Su, Lei Ma, Hang Su, and Jun Zhu. Motus: A unified latent action world model, 2025. URL https://arxiv.org/abs/2512.13030.
- [3]Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. π0: A vision-language-action flow model for general robot control, 2026. URL https://arxiv.org/abs/2410.24164.
- [4]Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspair Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-2: Vision-language-action models transfer web knowledge to robotic control, 2023. URL https://arxiv.org/abs/2307.15818.
- [5]Qiwei Chen, Kaijun Zhou, Nuohui Shi, Zhiyang Li, Yuxuan Feng, and Jinyu Gu. Faster and better? benchmark bugs and design limitations distort the evaluation of vision-language-action acceleration, 2026a. URL https://arxiv.org/abs/2609.37771.
- [6]Tianxing Chen, Zanxin Chen, Baijun Chen, Zijian Cai, Yibin Liu, Qiwei Liang, Zixuan Li, Xianliang Lin, Yiheng Ge, Zhenyu Gu, Weiliang Deng, Yubin Guo, Tian Nian, Xuanbing Xie, Qiangyu Chen, Kailun Su, Tianling Xu, Guodong Liu, Mengkang Hu, Huan-ang Gao, Kaixuan Wang, Zhixuan Liang, Yusen Qin, Xiaokang Yang, Ping Luo, and Yao Mu. Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. CoRR, abs/2506.18088, 2025. doi: 10.48550/ARXIV.2506.18088. URL https://doi.org/10.48550/arXiv.2506.18088.
- [7]Tianxing Chen, Yue Chen, Zixuan Li, Junyuan Tang, Kailun Su, Weijie Wan, Baijun Chen, Haoran Lu, Haowen Yan, Honghao Su, Zhiyang Dou, Kaixuan Wang, Dandan Zhang, Yunze Liu, Yan Qin, Qiwei Liang, Qiwei Wu, Zijian Lin, Wenwei Lin, Yuran Wang, Minghua He, Tianshu Wu, Ruihai Wu, Jingquan Zhou, Kai-Chong Lei, Haibao Yu, Yuanfeng Ji, Weiyang Jin, Guanyu Lin, Xiaofan Li, Qi Xiong, Renjing Xu, Zhongyu Li, Wenhao Chai, Enze Xie, Ziwei Wang, Yao Mu, Hao Dong, Wojciech Matusik, Mingyu Ding, Wenbo Ding, Ping Luo, and Masayoshi Tomizuka. Robodojo: A unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies, 2026b. URL https://arxiv.org/abs/2607.04434.
- [9]Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, and Serkan Cabi. Vision-language models as success detectors, 2023. URL https://arxiv.org/abs/2303.07280.
- [10]Jiafei Duan, Wilbert Pumacay, Nishanth Kumar, Yi Ru Wang, Shulin Tian, Wentao Yuan, Ranjay Krishna, Dieter Fox, Ajay Mandlekar, and Yijie Guo. Aha: A vision-language-model for detecting and reasoning over failures in robotic manipulation, 2024. URL https://arxiv.org/abs/2410.00371.
- [11]Senyu Fei, Siyin Wang, Junhao Shi, Zihao Dai, Jikun Cai, Pengfang Qian, Li Ji, Xinzhe He, Shiduo Zhang, Zhaoye Fei, Jinlan Fu, Jingjing Gong, and Xipeng Qiu. Libero-plus: In-depth robustness analysis of vision-language-action models, 2025. URL https://arxiv.org/abs/2510.13626.
- [12]Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, and Jianyu Chen. Video prediction policy: A generalist robot policy with predictive visual representations, 2025. URL https://arxiv.org/abs/2412.14803.
- [13]Yuan Huang, Zirui Song, and Xiuying Chen. Do MLLM judges judge the edit? auditing bias in image editing evaluation with verified quality preservation, 2026.arxiv.org/abs/2610.01670
- [14]Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky. π0.5: a vision-language-action model with open-world generalization. CoRR, abs/2504.16054, 2025. doi: 10.48550/ARXIV.2504.16054. URL https://doi.org/10.48550/arXiv.2504.16054.
- [15]Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. Rlbench: The robot learning benchmark & learning environment, 2019. URL https://arxiv.org/abs/1909.12271.
- [16]Tianchong Jiang, Xiangshan Tan, Samuel Wheeler, Luzhe Sun, Teodwos W. Ayalew, and Matthew Walter. What are we actually benchmarking in robot manipulation?, 2026. URL https://arxiv.org/abs/2606.04233.
- [17]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024. URL https://arxiv.org/abs/2310.06770.
