WorksComputer Science
RoboDx: Repair and Diagnosis of Faults in Robot Software
Abstract
A patch to a robot's navigation software can pass every unit test and still drive the robot off its path. Checking how the repaired robot drives catches such patches. It does not show that the agent found everything that was wrong. We introduce RoboDx, a generator of repair tasks for simulated robot navigation software with known injected faults. A hidden verifier marks a repair as verified only if the robot then drives correctly in hidden scenarios, which the agent never sees. A second track hides the code and asks the agent to pick the cause from a list of hypotheses. A task with two faults can contain a quiet fault, a fault that makes no hidden scenario fail unless the other fault is also present. On our set of 100 tasks, RoboDx-Hard-100, almost every repair by Claude Opus 5.5 with standard coding tools is verified. We analyse 94 of the tasks, 44 with one fault and 50 with two. On the 50 two-fault tasks, only 17 of Opus's 49 verified repairs remove both injected edits. Most faults left behind are configuration constants whose correct value the task's documentation does not state. Opus names both faults on 10/50 of these tasks when it cannot see the code. We then give four current models the same measurements of each faulty robot, picked so that, in the simulator, only the true cause produces all of them. The models name the cause on at most 38 of 94 tasks. They are right on almost all when they also get the simulator's output for every hypothesis. Their own predictions of that output give no detectable gain. On 97 tasks from real bug-fix pull requests, Claude Haiku 5.5, the weakest model on the generated tasks, scores lowest, and the other three models are not ordered.
Introduction
A coding agent is asked to repair a robot that drives off its route. The robot’s navigation software has two faults. One line sets the heading gain ten times too low. The heading gain controls how hard the robot steers back toward its route, so the robot now turns sluggishly and misses waypoints, the points along its route. The other fault is a deleted speed cap. Without the cap the robot no longer slows down while it turns. With a correct heading gain the robot turns quickly, so alone the missing cap makes no test mission fail. The agent restores the gain. The robot then completes every test mission, and the repair counts as solved. The speed cap is still missing from the code (Figure 1).

Figure 1. The running example in four steps, with Claude Opus 5.5 (Opus) as the agent. Each step after the first is tagged with the research question it illustrates. Section 2 tells the example in full.
Robot software holds much code like the speed cap. Speed caps, timeouts and safety stops do nothing until something else fails, so a check of the robot’s behaviour can miss them. Coding agents are now applied to robot software [[34, 37, 72]], and finding the cause of a robot’s misbehaviour is a research problem of its own [[46, 44, 58]]. The closest benchmark for robot repair holds 33 real defects [[104]]. With real defects only the reported fault is known, so a fault that a repair leaves behind cannot be counted.
Benchmarks built from repository issues, such as SWE-bench, grade a fix with the repository’s unit tests [[52, 114]]. Unit tests accept wrong patches [[92, 82, 110, 5]]. Table 1 sets a ROBODX task beside such a task. (i) The fix is graded by how the robot drives, because a wrong line can pass every unit test and still send the robot off its route. Faulty patches for robotic vehicles have been found by their effect on physical behaviour [63]. (ii) Every fault is known, because we inject it, so we can count the faults a repair leaves behind. That count matters because a fault can cause no failure unless a second fault is present, as with the missing speed cap. (iii) The cause is asked for directly. A second track hides the code and asks the agent to name the cause from measurements of the robot.
| SWE-bench-style task | RoboDx task | |
| How a fix is graded | the repository’s unit tests | hidden simulated missions that the robot must drive correctly |
| What is known about the faults | one reported bug, with its reference patch | every injected fault, so a verified repair can be checked for a fault left behind |
| Second task | none | name the cause from measurements of the robot, without the code |
Table 1. The three ways in which a RoboDx task differs from a task in a benchmark built from repository issues, such as SWE-bench [52].
We introduce RoboDx, a generator of such tasks and the hidden verifier that grades them (Figure 2). The generator injects faults into correct navigation software that runs in a robot simulator. Each fault is a small edit of one of 143 kinds, such as a wrong change of coordinates or a wrong configuration value. A repair is verified if the robot passes every hidden check, among them 21 hidden scenarios, simulated missions that the agent never sees. A quiet fault is an injected fault that fails no hidden scenario unless the task’s other fault is also present. RoboDx-HARD-100 is our set of 100 generated tasks. We analyse 94 of them, 44 with one fault and 50 with two. A companion real set has 97 tasks from merged bug-fix pull requests of three open-source robot projects.

Figure 2. How a RoboDX task is built (top row) and what happens in a task (bottom row). The generator injects one to three faults, each a small code edit, and keeps a task only if the faulty robot fails a hidden check. The agent never sees the generator or the verifier. The verifier judges a repair from the simulator’s record of how the robot moved. In diagnosis the agent gets a list of hypotheses and a probe budget, and the code is withheld.
RQ1: Does a verified repair remove every injected fault? A verified repair often leaves a fault in place. With standard tools, Opus has a verified repair on 98 of the 100 tasks. On the 50 two-fault tasks, only 17 of its 49 verified repairs remove both faults (Section 4.1).
RQ2: Can agents name the cause without the code? Opus often does so on single-fault tasks and rarely on two-fault tasks. It names both faults on 10/50 of the two-fault tasks. A reference procedure that picks its measurements at random, and is given the simulator’s output for each hypothesis, is right on 47/50 (Section 4.2).
RQ3: What do agents lack when they miss the cause? Our evidence suggests that they lack the simulator’s output for each hypothesis. Shown the same measurements of the faulty robot, and choosing none themselves, four current models are right on 27 to 38 of 94 tasks. Adding that output raises them to 89 to 94, and their own predictions of that output give no detectable gain (Section 4.3).
RQ4: Do real bug-fix tasks separate the same agents? The real tasks separate the agents only in part. On the 97 tasks of the real set, Claude Haiku 5.5 scores lowest and the other three models are not ordered (Section 4.4).
A verified repair rate near 1.00 can make repair look solved. Of Opus’s 49 verified two-fault repairs, 32 still hold a fault. A benchmark that reports only the verdict does not count these faults. Opus, when it may ask a simulator what a hypothesis would show, names the cause on every task we analyse. This suggests that diagnosis agents need a way to simulate their hypotheses.
Contributions.
A benchmark generator. ROBODX builds robot-software repair tasks with known injected faults, and its verifier grades a repair by how the simulated robot drives.
Two additions to the verified repair rate. The full-repair rate is the share of verified two-fault repairs that remove both faults. The diagnosis track asks for the cause without the code and is scored against reference procedures that are given the simulator’s output.
Two task sets. ROBODX-HARD-100 has 100 generated tasks and the real set has 97 tasks from real bug-fix pull requests.
An evaluation of current coding agents on both sets, which answers RQ1 to RQ4 above.
The running example in full
This section walks through one task with two injected faults, from what the agent gets to both verdicts. The task is a pair from an earlier set built by the same generator, and ROBODX-HARD-100 contains pairs with the same two faults (Appendix § 14.§ 1).
What the agent gets. A simulated robot with two driven wheels must drive a mission, a route of waypoints. The agent gets the faulty code that steers the robot, a simulator it can run, and a README. The README states the specification and holds an issue report that describes the symptoms and names no injected fault (Appendix § 8).
How a repair is checked. A hidden verifier checks every repair in three ways. It runs unit tests, some of which the agent has seen. It runs 21 hidden scenarios, simulated missions that the agent never sees. It checks that the robot’s path is unchanged when the whole world is shifted and rotated. We call these the hidden checks. A repair is verified only if it passes every one of them (Appendix § 9.§ 3).
A loud fault and a quiet fault. The first fault lowers the heading gain, how hard the controller steers the robot back toward the route, from 2.0 to 0.2. With the low gain the robot turns sluggishly and misses waypoints. The low gain is the loud fault, a fault that fails hidden scenarios on its own. Alone it fails six of the 21 hidden scenarios.
The second fault deletes a cap on the speed command. The controller caps that command at the configured maximum and then lowers it while the robot points away from its target. With the cap deleted, the robot no longer slows down while it turns toward a distant waypoint. A later line still limits the final command, so the robot never exceeds its rated speed. With the correct gain the robot turns quickly, and the missing cap makes no hidden scenario fail. The missing cap is the quiet fault, which fails no hidden scenario unless the low gain is also present. The low gain makes the robot turn slowly. Without the cap it also keeps its full forward speed while it turns, so it swings wider off its route. The two faults together fail 16 scenarios.
What Opus did in repair (RQ1). Claude Opus 5.5 (Opus) repaired the task with standard tools, which read, search and edit files, run tests and run the simulator. It restored the gain and left the cap out. The repair was verified, because no hidden scenario fails when the cap is missing and the gain is correct. The quiet fault stayed in the code.
What the diagnosis agent gets. In the diagnosis track the agent gets the same task without the code. It receives the README and the initial observation, the outcome of one mission with the two faults. It also receives a catalogue of twelve candidate causes. Each candidate cause is a hypothesis. Besides the true pair, the catalogue holds each of its two faults alone, other single software faults, wrong causes made of two faults, faults of the simulated robot itself such as wheels 10% smaller than configured, and no fault (Appendix § 8).
The agent learns about the faulty robot by buying probes, measurements such as a mission on a long straight. Each probe has a price, and the agent may spend at most a budget of 12. Each probe returns what the true system shows.
For any hypothesis, the simulator can compute what a probe would return if that hypothesis were the cause. We call this result the hypothesis’s signature under that probe. The signature of the low gain alone under the long-straight probe is a peak speed of 0.6 m/s. The true pair has the same signature. A reference procedure, the EIG oracle, is given every hypothesis’s signature, which is hidden from the agent. It picks each probe by expected information gain (EIG) per unit cost, where EIG is how much the probe’s outcome is expected to narrow down the hypotheses. On this task only the true pair’s signature matches the initial observation, so the EIG oracle answers at cost 0 without buying a probe.
What Opus did in diagnosis (RQ2). Table 2 lists Opus’s diagnosis run on this task, probe by probe. At the fourth probe, Opus ruled out the missing speed cap because the peak speed on a long straight was “0.6 m/s, which is the rated limit and not above it”. A later line of the code still limits the final speed command, so this probe gives an observation that is the same with and without the cap. The fifth probe, the mission given in rotated coordinates, pointed Opus to the low heading gain. Opus answered that the low gain was the only fault, with confidence 0.5.
| probe | cost | what it returned | what Opus concluded | |
| 1 | 2 | mission in which the network link to the robot is cut | the robot stopped after the cut; a fault in the safety stop is unlikely | |
| 2 | 3 | test-bench drive along an arc | the software’s motion estimate matches the measured motion; a fault in the motion estimate is unlikely | |
| 3 | 1 | stream of velocity commands | sent 30 times a second, none lost; cited, no hypothesis ruled out | |
| 4 | 2 | mission on a long straight | peak speed 0.6 m/s, the rated limit | missing speed cap unlikely |
| 5 | 2 | the mission given in rotated coordinates | the robot circles nearly four full turns (24 rad) and does not finish | points to the low heading gain |
| 6 | 1 | controller status | the controller’s internal state | not cited in the answer |
| 7 | 1 | estimator status | the position estimator’s internal state | not cited in the answer |
| answer | 12 of 12 | the low heading gain alone, confidence 0.5, which is wrong |
Table 2. Opus’s diagnosis run on the running example, one run. Probes are listed in the order recorded. Cost is each probe’s price, out of a budget of 12. The true cause is the low heading gain together with the missing speed cap. At probe 4 the peak speed is the same with and without the cap, and Opus uses it to rule the missing cap out.
Two controls (RQ3). We then gave Opus what the true system shows under every probe, with no budget. We call this control the full observation table. Opus gave the same answer. In a second control, under the same budget, Opus could ask the simulator for the signature of any hypothesis under any probe. With this tool, which returns any hypothesis’s signature, Opus named both faults (Appendix § 12.§ 2).
The benchmark
This section says how tasks like the one in § 2 are built and scored, and it defines only what the research questions use.
How a task is built. ROBODx starts from correct navigation software for a simulated robot (Appendix § 9.§ 1). The generator applies one to three operators to it (Figure 2), and ROBODx-HARD-100 uses one or two. An operator is a small edit to one module that still compiles and injects one fault. Lowering the heading gain is one operator, and deleting the speed cap is another. The 143 operators break four kinds of mechanism, which are coordinate transforms, timestamps, message routes, and control laws or configuration values (Appendix § 9.§ 2). The generator keeps a candidate as a task only if the correct software passes every hidden scenario and the faulty software fails a hidden check.
A task with one injected fault is a single. A task with two is a pair, like the running example. In a pair, a quiet fault is an injected fault that fails no hidden scenario alone. The missing speed cap is one. A pair can hold one quiet fault, as in the running example, or two.
How a repair is scored (RQ1). The agent gets the faulty code and an issue report, edits the code and submits. A repair is verified if it passes every hidden check (§ 2). The verified repair rate is the share of tasks whose repair is verified. The full-repair rate is the share of verified pair repairs that remove every injected edit. An edit counts as removed when its injected text no longer appears in the repaired code. This includes a constant that the repair sets to a value other than the reference. Opus’s repair of the running example is verified. It is not a full repair, because the speed-cap edit remains (Appendix § 9.§ 4).
How a diagnosis is scored (RQ2). In diagnosis the agent does not see the code, because an agent that reads the code can find a one-line edit by inspection. It gets the catalogue and the initial observation, buys probes within the budget and names the cause, as in § 2. An answer counts as correct if it names exactly the true hypothesis or, where no probe can tell the true hypothesis from others, exactly that whole set. On a pair the true hypothesis is a composite, a hypothesis made of two faults. A correct answer must name both faults, including one that is harmless alone, so an answer that names only the fault needed to explain the robot’s misbehaviour is scored wrong.
Two reference procedures show what the probes allow. Each is given every hypothesis’s signature, which is hidden from the agent. The EIG oracle picks each probe by expected information gain per unit cost, a greedy rule that goes back to model-based diagnosis [27, 41]. It keeps the hypotheses whose signatures equal the observations. Random probing with exact inference picks its probes at random and keeps hypotheses in the same way.
How we find what agents lack (RQ3). A second experiment removes probing (Appendix § 9.§ 5). No model chooses probes. Every model is shown the same observations of the faulty robot and must name the cause. The observations are those of the initial mission and of the probes that the EIG oracle chose for the task. Only the true hypothesis has signatures equal to all of them, apart from hypotheses that no probe can tell apart from it. The experiment has three versions. In one version the model gets these observations and nothing else. In a second version it also gets the signature table, which lists every hypothesis’s signature on the same probes. In a third version the table is replaced by the model’s own signature predictions. A signature prediction is the model’s guess at a signature, written before it sees any observation (Appendix § 9.§ 5). The first two versions are the last two blocks of Table 4.
| singles (44) | pairs (50) | all (94) | |
| Reference procedures, given every hypothesis’s signature | |||
| EIG oracle | 44/44 | 50/50 | 94/94 |
| random probing with exact inference | 22/44 | 47/50 | 69/94 |
| Agents that choose their own probes within the budget (RQ2) | |||
| Opus 5.5 | 32/44 | 10/50 | 42/94 |
| GPT-5.5 | 34/44 | 2/50 | 36/94 |
| Haiku 5.5 | 10/44 | 0/50 | 10/94 |
| No probing, the same observations for every model (RQ3) | |||
| Opus 5.5 | – | 3/50 | 38/94 |
| GPT 6.1 | – | 5/50 | 36/94 |
| Sonnet 5.5 | – | 2/50 | 30/94 |
| Haiku 5.5 | – | 0/50 | 27/94 |
| No probing, the same observations plus every hypothesis’s signature (RQ3) | |||
| Opus 5.5 | – | 50/50 | 94/94 |
| GPT 6.1 | – | 50/50 | 93/94 |
| Sonnet 5.5 | – | 50/50 | 93/94 |
| Haiku 5.5 | – | 48/50 | 89/94 |
Table 4. Diagnosis on the 94 tasks of the main run of ROBODX-HARD-100, one attempt per task. Entries count correct answers as defined in § 3. The first two blocks answer RQ2. The EIG oracle picks each probe by expected information gain, and random probing picks probes at random. The last two blocks answer RQ3. In them no model chooses probes and every model gets the same observations. They report only pairs and all tasks. The pairs column carries both comparisons.
The task sets. RoboDX-HARD-100 is our set of 100 generated tasks, 44 singles and 56 pairs. The name refers to how the tasks were selected. Its pairs fail mostly in scenarios that need both faults. Of its singles, 30 fail no public scenario and 6 need a value stated in the README (Appendix § 9.§ 2). We analyse 94 of the 100 tasks, called the main run, with 44 singles and 50 pairs. The other six pairs come from a separate generator run and are reported in Appendix § 11.§ 2. For RQ4, the test set has 97 tasks built from merged bug-fix pull requests of three open-source robot projects, in the same task format (Appendix § 13.§ 1).
Checking the verifier. We tested the verifier on repairs whose verdict we know. Each task has a reference repair, the edit that undoes its injected faults. Over the 296 generated tasks of ROBODX-HARD-100 and four earlier sets, the verifier verified the reference repair in 592 of 592 checks, two per task. We also applied five kinds of wrong repair to the faulty code of each task, 1480 in all, and the verifier rejected all of them. A harder test on the main run applied wrong edits on top of the reference repair. The verifier accepted 132 of these edits. All but 2 of these either change nothing that the robot uses or still meet every stated requirement. The remaining 2 break a README requirement. In both, the robot keeps turning in place at the goal, because the check that it stops reads only its forward speed. On an earlier set a further gap appeared, because the hidden scenarios cover a limited range of start headings. The details are in Appendix § 10. § 6 reports how far a program that uses only the generator’s own files gets.
Results
The results for RQ1 to RQ3 are on ROBODX-HARD-100 (§ 3). We analyse its main run of 94 tasks, 44 singles with one injected fault and 50 pairs with two. The models are Claude Opus 5.5, GPT 6.1, Claude Sonnet 5.5, Claude Haiku 5.5 and the older GPT-5.5. Below, the first four are called Opus, GPT 6.1, Sonnet and Haiku. Repair agents use standard tools. Repair and diagnosis agents run in an agent loop that we wrote for this benchmark, the same for every model. The loop offers the tools as function calls and gives the agent no shell. Each model makes one attempt per task. Runs with a second tool set, the operating details and the statistics are in Appendix § 11.§ 1.
RQ1: Does a verified repair remove every injected fault?
A repair can pass every hidden check and still leave an injected fault in the code, and on pairs this happens often.
Almost every repair by Opus, GPT 6.1 and Sonnet is verified (Table 3). Opus’s repair is verified on 98 of all 100 tasks. On the 50 pairs, only 17 of Opus’s 49 verified repairs remove both injected edits. GPT 6.1 and Sonnet remove both in about half of theirs. The three models are verified on nearly the same number of tasks. GPT 6.1 and Sonnet often set a constant to a value other than the reference, which counts as removed. If we count only repairs that put the reference code back at both edits, Opus has 14 of 49, GPT 6.1 has 11 of 49 and Sonnet has 10 of 48 (Appendix § 11.§ 3). In the running example, Opus restored the heading gain and left the speed cap out. Its repair was verified because no hidden scenario fails when only the cap is missing.
| model, standard tools | singles (44) | pairs (50) | rate (94) | 95% CI | removed (pairs) | all 100 tasks |
| Haiku 5.5 | 42/44 | 27/50 | 0.73 | [0.54, 0.90] | 9/27 | 71/100 |
| Sonnet 5.5 | 44/44 | 48/50 | 0.98 | [0.93, 1.00] | 26/48 | 98/100 |
| GPT-5.5 | 42/44 | 43/50 | 0.90 | [0.82, 0.98] | 23/43 | 91/100 |
| GPT 6.1 | 42/44 | 49/50 | 0.97 | [0.92, 1.00] | 25/49 | 97/100 |
| Opus 5.5 | 44/44 | 49/50 | 0.99 | [0.96, 1.00] | 17/49 | 98/100 |
Table 3. Repair on ROBODX-HARD-100 with standard tools, one attempt per task. The first three columns count verified repairs on the 44 singles and 50 pairs we analyse and give the rate over those 94 tasks. The interval is a 95% bootstrap interval that resamples groups of tasks built from the same faults. “Both faults removed” counts the verified pair repairs that remove both injected edits. The last column covers all 100 tasks. Three models have a verified rate near 1.00 while far fewer of their repairs remove both faults.
Opus’s verified pair repairs leave 32 injected edits in place, and every one is a quiet fault. Of these, 18 are configuration constants whose correct values the README does not state. An edit of this kind left in place need not mean that the agent failed to find the cause. On pairs whose README states the correct form of every quiet fault, Opus removes both edits in 14 of 19 verified repairs.
In a later run without one sentence of the prompt, more verified repairs remove both edits. The repair prompt asks for minimal, root-cause fixes. In the run of Opus on the 50 pairs without that request, 24 of 50 verified repairs remove both edits, against 17 of 49 with it. Each of the 6 gained full repairs sets a gyro weight to a value other than the reference, and the number of repairs that put the reference code back at both edits does not rise. The constants that the README does not state are left in place as often as before (Appendix § 11.§ 4).
Takeaway (RQ1). A verified repair can leave a fault in the code. On the 50 two-fault tasks, 17 of Opus’s 49 verified repairs with standard tools remove both faults, and most faults left behind are configuration constants that the README does not state.
RQ2: Can agents name the cause without the code?
Opus often names the cause on single-fault tasks and rarely names both faults on two-fault tasks.
In the diagnosis track the agent does not see the code. It buys probes within the budget and then names the cause from the catalogue (§ 2). We ran this track for Opus, GPT-5.5 and Haiku, a choice of models that was not fixed in advance. The two reference procedures of § 3 show what the same probes allow.
Opus names the cause on 32/44 of the singles and both faults on 10/50 of the pairs (Table 4). Random probing with exact inference is right on 47/50 of the pairs. Probes picked at random usually hold enough information to name both faults. Random probing is right on fewer singles, 22/44. On many catalogues the initial observation does not single out the true hypothesis, and the choice of probes then matters more (Appendix § 12.§ 1). On the running example Opus named the low heading gain alone.
Figure 3 shows what the agents answered on the pairs. The most common wrong answer of Opus is one of the two faults alone, on 16 pairs. On 7 more it named both faults together with an extra hypothesis, which counts as wrong. On 14 it named a hypothesis that contains neither fault. GPT-5.5 named one of the two faults alone on 27 pairs, and Haiku named neither fault on 26.

Figure 3. What each agent answered on the 50 two-fault tasks when it chose its own probes (RQ2). Segments run left to right in the order of the legend, and only the first counts as correct. “Other answer with one of the two faults” is any other wrong answer that names one of the two faults. “Neither fault” includes the answer that nothing is wrong. The third category, one of the two faults alone, is the most common answer of Opus and GPT-5.5.
The setup may favour single-fault answers. The diagnosis prompt does not say that a cause may have two faults, so part of the gap on pairs may be a preference for answers with one fault (§ 6).
Takeaway (RQ2). Without the code, Opus names the cause on 32/44 of the single-fault tasks and names both faults on 10/50 of the two-fault tasks. A reference procedure that probes at random and is given the simulator’s output for every hypothesis is right on 47/50.
RQ3: What do agents lack when they miss the cause?
Our evidence suggests that agents lack the simulator’s output for each hypothesis, its signature (§ 2). When the signatures are given, every model names the cause on almost every task.
The choice of probes is a small part of the failure. In 46 of Opus’s 52 wrong answers in RQ2, the probes Opus had bought were already enough to single out the true hypothesis, as exact inference from the signatures shows (Appendix § 12.§ 4). We then removed probing. Every model is shown the same observations and must name the cause. Wherever the probes allow, only the true hypothesis has signatures equal to these observations (§ 3).
From the observations alone, the four models are right on 27 to 38 of 94 tasks (Table 4). On pairs they are right on 0 to 5 of 50. We then added the signature table, which lists what the simulator returns for every hypothesis on the same probes. For the running example the table would list each hypothesis’s outcome on the initial mission. With the low heading gain alone the robot finishes that mission after 20 s, and the true system needs 22 s. The table turns the task into a lookup, which a plain string comparison solves on 94 of 94 tasks. Given the table, the four models are right on 89 to 94 of 94 tasks, and on 48 to 50 of the 50 pairs.
The models do not supply the signatures themselves. Before seeing any observation, each model wrote a signature prediction for every hypothesis. Such a guess says, for example, how long the robot with only the low heading gain needs for the initial mission. These predictions rarely match the true signatures, and giving them to the model gives no detectable gain. Opus is right on 40 tasks with its own predictions and on 38 without them (Appendix § 9.§ 5).
The two controls of § 2, run for Opus on every task, agree with this reading. The first control gives Opus every observation that the probes can return, with no budget. Its share of correct answers moves from 0.45 in RQ2 to 0.54. The second control is a tool that returns the simulator’s output for any hypothesis on request. It makes Opus right on every task (Appendix § 12.§ 2). We read the diagnosis failure as lying in the step that needs the signatures, under a diagnosis prompt that does not say a cause may have two faults.
Takeaway (RQ3). Shown the same observations, with no probes to choose, Opus, GPT 6.1, Sonnet and Haiku are right on 27 to 38 of 94 tasks. The simulator’s output for each hypothesis raises them to 89 to 94, and their own predictions of that output give no detectable gain. We read the diagnosis failure as lying in the step that needs the simulator’s output, under a prompt that does not say a cause may have two faults.
RQ4: Do real bug-fix tasks separate the same agents?
Real bug-fix tasks separate the agents only in part. Haiku scores lowest, and the other three models are not ordered.
The tasks above are generated. As a check on real defects we ran the same repair agents on the real set. It has 97 tasks built from merged bug-fix pull requests of ArduPilot, an autopilot, and of Nav2 [74] and tf2, two packages of the Robot Operating System (ROS). tf2 is the successor of the tf transform library [36]. The agent gets the issue text and the repository just before the fix. A repair is graded by the test change of the same pull request, which the agent does not see (Appendix § 13.§ 1). The strict reading counts a task only if the repair was verified in the run. The lenient reading also counts a saved patch that failed in the run and was verified on one later re-grade at lower machine load.
Haiku, the weakest model on RoboDx-HARD-100 (Table 3), scores lowest under both readings (Table 5). Under the strict reading Haiku’s repair is verified on 0.76 of the tasks, against 0.91 for Opus. Under the lenient reading the rates are 0.84 and 0.94. A paired test with Holm correction separates Haiku from each of the three stronger models under the lenient reading, and from Opus alone under the strict reading (Appendix § 13.§ 2). The three stronger models lie within three tasks of each other under either reading, so the real set does not order them (Appendix § 13.§ 2).
| model, standard tools | strict reading | lenient reading | ||||
| verified | rate | 95% CI | verified | rate | ||
| Opus 5.5 | 88/97 | 0.91 | [0.83, 0.95] | 91/97 | 0.94 | |
| GPT 6.1 | 85/97 | 0.88 | [0.80, 0.93] | 93/97 | 0.96 | |
| Sonnet 5.5 | 85/97 | 0.88 | [0.80, 0.93] | 92/97 | 0.95 | |
| Haiku 5.5 | 74/97 | 0.76 | [0.67, 0.84] | 81/97 | 0.84 |
Table 5. Repair on the 97 tasks of the real set with standard tools, one attempt per task. The strict reading counts a task only if its repair was verified in the run. The lenient reading also counts a saved patch that was verified on one later re-grade. Haiku is lowest under both readings, and the other three models lie within a few tasks of each other.
The real set differs from RoboDx-HARD-100 in how repairs are graded, in which files the agent may edit and in the mix of tasks. The stronger models score lower on it than on RoboDx-HARD-100. Because of these differences, the lower scores do not show a loss of skill. Separately, the pull requests are public, so a model may have seen a fix in training, and our check for copied fixes cannot exclude that (Appendix § 13.§ 3).
Takeaway (RQ4). Real bug-fix tasks separate the agents only in part. On 97 real tasks Haiku scores lowest whether or not one re-grade is allowed, and the other three models are not ordered.
Related work
This section compares RoboDx with work on test-graded benchmarks, robot software, fault injection and diagnosis.
Software-engineering benchmarks graded by tests. SWE-bench and its descendants grade an agent’s patch by the repository’s own tests [52, 121]. Audits find these verdicts often wrong [110, 116, 5]. Program-repair research had found the same problem earlier [92, 82]. To our knowledge no version of SWE-bench covers a robotics repository. A robot whose code passes every unit test can still drive off its route. RoboDx grades a repair by how the simulated robot drives in hidden scenarios.
Robot software and its testing. Bug studies catalogue the faults that occur in real robot software [102, 20]. Detectors find classes of these faults, such as mismatched physical units [75]. Simulation and fuzzing check how robots behave [103, 61]. The closest prior work, RoboBugBench, grades LLM repairs of 33 real defects in a ROS 2 navigation stack in a physics simulator [104]. RoboBugBench uses real defects. RoboDx injects its faults, so it controls each one. It also adds a diagnosis track that asks for the cause without the code.
Fault injection and mutation. Several works build tasks from synthetic bugs in real repositories that break an existing test [114, 93]. One of them, SWE-smith, already combines several bugs in one task. Mutation testing injects small edits to assess a test suite [54]. Higher-order mutation combines several edits in one program [51]. RoboDx injects known faults, so RQ1 can check a verified repair for an injected edit left behind.
Diagnosis and reference procedures. Root-cause benchmarks for agents mostly score how well an agent analyses telemetry that it is given [111, 80]. None of them compares the agent with a reference procedure. Model-based diagnosis chooses each next measurement by expected information gain [27, 28]. To our knowledge no root-cause benchmark for agents combines priced probes, scoring against an exact-inference reference, two-fault causes, and controls that separate probe choice from signature prediction. Our diagnosis track scores whether coding agents find the cause of code-level faults against such a reference (RQ2, RQ3). Appendix § 15 gives the full survey.
Limitations
The limits below bear on the takeaways of § 4, and Appendix § 16 lists every limit.
Simulated robot, generated faults. RQ1 to RQ3 hold for generated faults in simulated navigation software. The real set of RQ4 is a companion check. It does not validate the generator.
The repair prompt, the README and the agent loop. The repair prompt asks for minimal fixes. A later run without that request removes both faults more often. Each gained full repair sets a gyro weight to a value other than the reference (Section 4.1). Most faults left behind are constants whose correct values the README does not state, so leaving one in place need not mean a missed cause. The agents run in our own agent loop. In a check with a shell agent on 20 of the 50 pairs, Opus’s repair is verified on all 20 and removes both faults on 8. In our loop on the same pairs, 6 of 19 verified repairs remove both. The shell agent’s two extra full repairs set a gyro weight to a value other than the reference (Appendix § 11.§ 5).
Small numbers. With one attempt per task and few fault combinations, several intervals are wide. The models come from two vendors. A difference of a few tasks between two runs can be noise.
The diagnosis setup. The agent picks from a closed list, and the prompt never says that a cause may have two faults. The low scores on two-fault tasks in RQ2 and in RQ3 without the signature table cannot be separated from a preference for single-fault answers.
What the generator’s files give away. A rule that only reads the generator configuration is right on 0.79 of the main run, averaged over its random tie-breaks (Appendix § 12.§ 1). No agent that chooses its own probes scores as high. A program that undoes the edits it recognises from the operator library repairs 0.84 of the main run (Appendix § 10). A model that has seen the generator’s configuration and operators should be tested on new fault combinations and held-out operators.
Conclusion
On generated robot navigation software, the repairs of Opus, GPT 6.1 and Sonnet pass every check on almost every task and often leave the quiet fault in place. Asked for the cause without the code, models rarely name both faults unless they are given the simulator’s output for each hypothesis. A benchmark for robot repair should report, beside the verified repair rate, whether a verified repair removes every injected fault and whether the agent can name the cause without the code. RoboDoX reports both. For those who build diagnosis agents, our results suggest giving the agent a simulator it can query about its hypotheses. On the 97 tasks from real bug-fix pull requests, the weakest model on the generated tasks also scores lowest under both the strict and the lenient reading, and the other three models are not ordered.
The running example: details
This section gives the issue report, the reference repair and the catalogue of the running example of § 2, task layered_multi_component_11017.
Robotics terms.
A differential-drive robot steers by the speeds of its two driven wheels.
Odometry turns wheel-encoder ticks into an estimate of distance and heading change.
The pose is the robot’s position plus its heading.
The mission frame is the coordinate frame of the waypoints. It may be shifted and rotated relative to the robot’s start.
The watchdog stops the motors when velocity commands stop arriving.
Issue report. The README that the agent reads holds the issue report below. The generator wrote it from observed symptoms, and an audit found no word in it that names an injected fault (Appendix § 9).
| Issue report (agent-visible, verbatim) |
| The robot leaves the planned route and ends far from the goal. Progress is made at first, then the trajectory deviates sharply. In some runs we instead see the following: waypoints are reached in the robot’s own estimate, yet the physical positions visited do not match the mission. Changing the control or sensor update rates changes how strongly it manifests. Find the root cause in the software stack and fix it. |
Table 7.
Reference repair. The reference repair is the generator’s own correct fix. It has two one-line edits. The first restores the cap on the speed command in the motion controller. The second restores the heading gain in the configuration from 0.2 to 2.0.
--- a/mbot/tracking.py
- v = self.kp_distance * dist_goal
+ v = min(self.v_max, self.kp_distance * dist_goal)
--- a/mbot/config/defaults.json
- "kp_heading": 0.2,
+ "kp_heading": 2.0,
Catalogue. The diagnosis agent gets the twelve hypotheses below. Each software hypothesis carries the id of the operator that injects it. K_kp_low is the low heading gain, and K_v_unclipped is the missing speed cap. Entries marked [plant] are faults of the simulated robot itself, the plant of Appendix B.1. The true entry is their composite, and the agent’s list does not mark it.
Each software operator id appears three times, once as a single and twice inside composites. Every composite has two components. For this reason the agent cannot find the true entry by counting how often a component appears or how many components a hypothesis has. The full menu of 27 probes can tell all twelve hypotheses apart.
Benchmark details
This section supports § 3 with the details of the robot, the tasks, the hidden checks and the scoring.
The simulated robot and its software
The plant is the deterministic simulation of the robot, its sensors and its network. The robot has two driven wheels with speed and acceleration limits. Its sensors are wheel encoders, a noisy gyro for turn rate, a compass and an intermittent position fix. A simulated network link can delay, reorder and drop messages. The plant records the true pose and the applied wheel commands, and the software cannot write to this record.
Two correct software stacks can drive the plant. In the layered stack, components exchange messages by topic through a rate-limited executor. This stack estimates the pose by fusing wheel motion with the gyro, and it corrects the heading with the compass. It steers toward a point ahead on the route. The callback stack hands each message straight to its handler. It also fuses wheel motion with the gyro and steers along an arc to a point ahead, a method called pure pursuit.
Operators and task generation
An operator is a small edit to one module that still compiles and injects one fault. Each operator belongs to one mechanism family, the kind of mechanism it breaks. Table 6 lists the four families with example operators.
| Hypothesis catalogue (balanced, with the true entry unmarked) |
K_kp_low [software, configuration]: heading gain configured too low. K_v_unclipped [software, motion controller]: linear speed command no longer limited to the configured maximum. |
T_watchdog_unit [software, actuator interface]: watchdog compares elapsed microseconds against a threshold scaled by 1000. V_arc_displacement [software, kinematics]: arc displacement re-implemented incorrectly. K_kp_low+K_v_unclipped, K_kp_low+V_arc_displacement, T_watchdog_unit+K_v_unclipped, T_watchdog_unit+V_arc_displacement [software, composite]. |
P_sluggish_motors [plant]: motor controllers limit wheel acceleration to 6 rad/s. |
P_wheel_radius_miscalibrated [plant]: both wheels 10% smaller than configured. |
P_encoder_noise [plant]: encoders report noisy tick counts. no_fault [none]: the system has no defect. |
Table 6. Hypothesis catalogue (balanced, with the true entry unmarked)
A masking operator changes a configuration constant. It is harmless alone and lets another fault show, so the robot fails only when both faults are present. One masking operator weakens the compass correction, so the software trusts the compass too little. The robot still drives correctly with a correct gyro and fails only when the gyro is also faulty.
Generation. For each candidate task the generator samples a software stack, a seed and one to three operators. It also picks one of several correct forms for each of 13 functions that the operators do not touch. Every public identifier is then renamed. The 4 public and 21 hidden scenarios are sampled independently of the fault. The issue report is written from observed symptoms. An audit rejects any report that names the operator, its target or an identifier from the edit.
Acceptance. A candidate becomes a task only if every condition below holds. The correct renamed stack passes every hidden scenario. The faulty stack fails a behavioural requirement on a hidden
| family | ops | example operators |
| spatial | 45 | S_compose_order: SE(2) composition translates before rotating<br>S_odom_halfbase: Odometry heading increment divides by twice the wheel separation |
| temporal | 37 | T_watchdog_unit: Watchdog compares elapsed microseconds against a threshold scaled by 1000<br>T_dt_arrival: Integration interval measured from arrival time instead of measurement stamps |
| communication | 26 | B_C_twist_key: Controller emits angular velocity under a field the motors do not read<br>C_executor_slow: Executor spins at 20 Hz |
| control | 35 | K_v_unclipped: Linear speed command is no longer limited to the configured maximum<br>BQ_rated_speed: Cruise speed above the platform’s rated 0.6 m/s (sampled) |
caption: Table 6: Mechanism families, the kinds of mechanism an operator breaks, with two example operators each. Column “ops” counts each family’s operators, 143 in all. SE(2) composition chains a shift and a rotation in the plane. Odometry is the wheel-measured motion. The watchdog stops the motors when commands stop. The executor delivers messages.
:::
scenario, or the metamorphic check, without raising an exception. The failure repeats on a second run. Renaming changes neither stack’s scenario verdicts. The reference repair is non-empty. In a pair, some hidden scenario fails with both faults present.
Selection of RoboDx-Hard-100. Apart from the eight replacement tasks (Appendix G.2), RoboDx-Hard-100 was selected by structural criteria fixed before any model was run on it. Pairs had to be dominated by interacting scenarios, which fail only when both faults are present. Singles had to fail no public scenario or need a rating stated in the README for their fix. A rating is a rated limit such as the rated speed. A script applied our thresholds to the 877 candidate pairs of sweeps that ran no model. The generator configuration lists the 11 operator sets it kept.
The first generator run’s acceptance conditions rejected every candidate of two of these sets. A separate generator run produced accepted pairs for one of the two. The six supplementary pairs come from that run and are reported apart from the main run.
Hidden checks
The verifier applies the agent’s edits to a clean copy of the faulty code and swaps in its own simulator. The unit tests are the public tests plus hidden property tests for the geometry and kinematics modules. On each of the 21 hidden scenarios, the verifier evaluates the behavioural requirements of tbl:7 from the plant’s record. The scenarios vary the mission frame, sensor noise, network delay, control rate and robot geometry, and some cut the network link. The thresholds come from the verifier’s own copy of the correct configuration. Edits to the software’s copy of the limits leave them unchanged.
The metamorphic check [17] has 3 pairs per task. Each pair takes a noise-free copy of one hidden scenario without a link cut and runs its mission twice, once as given and once with the whole world shifted and rotated. The second path, moved back, must match the first within 0.02 m. A repair is verified if every hidden scenario meets every requirement, every unit test passes, and the metamorphic check and the static configuration check hold.
Repair and diagnosis scoring
Repair. A verified pair repair counts as a full repair only if the injected text of both edits is gone from the patched module. An injected edit is README-determined if it changes code logic whose correct form the README describes, or a constant whose value or bound the README states. A pair is README-determined if all its quiet faults are. The running example is README-determined, because its README states the rated speed that the missing cap breaks.
Probes. A component’s status report or the statistics of one message stream cost 1. A sensor’s history or a mission under one of ten conditions, such as a long straight, cost 2. A test-bench drive on fixed commands costs 3. Observations are rounded to two significant figures and compared for exact equality.
Correct answers. Let be the set of hypotheses that give the same observation as the true hypothesis under every probe. An answer is task-correct if the set of hypotheses it names equals . No answer gets partial credit.
| group | requirement | rule (computed from plant truth and the verifier’s healthy limits) |
| protocol | no software error | the stack raised no exception |
| mission | mission completed | waypoints visited in order within 0.30 m, last one before s; skipped on link-cut scenarios |
| mission | path tracking | max distance to the current segment m until completion or link cut |
| mission | final position / stopped at goal | final distance m and final speed m/s |
| estimation | position / heading consistency | max error of the software’s logged estimates against interpolated truth m and rad (layered) or 0.4 rad (callback); fewer than six estimates fails |
| estimation | stamps monotonic / estimate rate | strictly increasing stamps; at least half of the expected odometry-rate count |
| safety | no plant violations | no non-finite or over-limit wheel command, robot stays in the workspace |
| configuration | ratings | configured maximum linear velocity (cruise speed) m/s and command timeout ms, read statically from the patched package |
| safety | linear / wheel speed limit | true speed rated; commanded wheel speed |
| safety | safety stop after link cut | stationary () by s, having moved before the cut |
| timing | command rate / actuation alive | logged commands per second within control rate; plant-side wheel updates control rate |
| metamorphic | SE(2) equivariance | on noise-free copies of the scenario, the shifted and rotated mission’s true trajectory, moved back, matches the base trajectory within 0.02 m |
caption: Hidden behavioural requirements, evaluated from the plant’s record of true motion. Thresholds are fixed per task from the correct stack, whose completion time is . The configuration row is the static check, read once from the patched package. The SE(2) row is the metamorphic check, evaluated on its 3 pairs of a mission and a shifted and rotated copy. The other rows are evaluated on every hidden scenario.
:::
Budgeted probing and reference procedures. In budgeted probing the agent picks its own probes, with a summed price of at most 12, and then answers. The EIG oracle keeps the set of hypotheses whose signatures equal all observations so far, under a uniform prior. At each step it buys the probe that maximises
where is entropy. It stops when one hypothesis remains or no probe has positive gain. Random probing with exact inference buys probes uniformly at random, with one seeded draw per task, and updates in the same way.
The direct-prediction study and the configuration-lookup rule
The direct-prediction study removes probing. For each of the 94 main-run tasks, is the set of probes that the EIG oracle bought. is empty when the initial mission alone singles out the true hypothesis. The target is the set of hypotheses that give the true observations on the initial mission and on . Every prompt holds the README, the catalogue and the observations of the fault-free system on the same probes. No prompt shows code.
Reference-probe observations. The prompt adds the observations of the faulty system under diagnosis on these probes, and the model names the cause.
Signature table. The prompt also adds the simulator’s observation for every hypothesis on the same probes, as one static block of text.
Own signature predictions. Before seeing any true observation, the model predicts every hypothesis’s observation on these probes,
no_faultexcepted. A later prompt adds these predictions to the reference-probe observations as the model’s earlier estimates.
The plain string comparison calls no model. It keeps every hypothesis whose observations in the signature table, written as text, equal the true observations. It is scored by the task-correct rule.
Configuration lookup sends no probe. It reads only the agent-visible catalogue and the generator configuration. If the catalogue holds a composite whose operator set is one of the configuration’s 11 operator sets, it answers that composite. Otherwise it answers a software single whose operator the configuration lists as preferred, or any software single if none is listed. It breaks ties uniformly at random, and a pair catalogue can hold two configured composites. Its reported score is the expected task-correct score over these random choices.
Validation of the verifier
This section supports the check of the verifier in § 3 and the last limit of § 6.
Stored controls. Stored controls are repairs whose verdict is known in advance. They cover the 296 generated tasks of RoboDx-HARD-100 and of four earlier sets (Appendix § 14.§ 1). Each task’s reference repair, the generator’s own correct fix, is checked twice. Five wrong repairs are applied to each task’s faulty code. They are a patch that changes nothing, a patch that deletes tests, a loosening of the limits, an edit to the wrong subsystem and a random edit of a number. The verifier verified 592 of 592 reference-repair checks. It rejected 1480 of 1480 wrong repairs (Table 8). An equivalent repair is a different correct fix and should be verified. An overfit repair is correct only under public conditions and should be rejected. Repairs of both kinds exist for a few operators only. Some stored wrong-subsystem and random edits changed no file. These controls graded the unchanged faulty code.
| set | reference | noop | test deletion | loosen limits | wrong subsystem | random edit | equivalent | overfit | mismatches |
| pilot (v1) | 60/60 | 30/30 | 30/30 | 30/30 | 30/30 | 30/30 | 5/5 | 1/1 | 0 |
| pilot v2 | 76/76 | 38/38 | 38/38 | 38/38 | 38/38 | 38/38 | – | – | 0 |
| compositions-100 | 200/200 | 100/100 | 100/100 | 100/100 | 100/100 | 100/100 | 2/2 | – | 0 |
| interaction/spec | 56/56 | 28/28 | 28/28 | 28/28 | 28/28 | 28/28 | – | – | 0 |
| RoboDx-HARD-100 main run | 188/188 | 94/94 | 94/94 | 94/94 | 94/94 | 94/94 | 14/14 | 12/12 | 0 |
| RoboDx-HARD-100 supplementary pairs | 12/12 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 | – | – | 0 |
Table 8. Stored controls per set. Each cell counts controls with the expected verdict over controls run. The reference repair is checked twice per task. The next five columns are wrong repairs of the faulty code. The text defines the equivalent and overfit repairs. Rows pilot (v1) and pilot v2 are the first and second pilots. Every cell should show zero mismatches.
Each of the five stored wrong repairs of the faulty code keeps the injected fault. The stored controls show that a repair which leaves the fault in place is rejected. They do not show that a wrong change to repaired code is rejected.
Wrong edits on repaired code. On the main run we also combined the loosened-limits, wrong-subsystem and random edits with the reference repair (Table 9). Each wrong edit was made to the faulty code as in the stored controls, and the reference repair was then applied on top of it. The loosened-limits edit was never verified. We sorted the 132 verified wrong-subsystem and random edits by inspection. Of these, 54 change no file and 64 change only code the robot does not use, such as a status counter. Another 12 change the robot’s behaviour but meet every requirement the README states. The other 2 are false accepts, wrong edits that break a README requirement and are verified.
| set | loosened limits | wrong subsystem | random edit | no change | inert | within spec | false accept |
| RoboDx-HARD-100 main run | 0/93 | 62/94 | 70/92 | 54 | 64 | 12 | 2 |
| RoboDx-HARD-100 and four earlier sets | 0/288 | 166/296 | 212/290 | 140 | 190 | 44 | 4 |
Table 9. Wrong edits combined with the reference repair, one grading each. The first three columns give verified over graded cases. Cases where the reference repair did not apply on top of the edit, because the edit overlaps the repair, are not graded. The last four columns sort the verified wrong-subsystem and random edits by inspection. The last column counts the false accepts.
In both false accepts on the main run, the edit makes the controller command a turn rate once the mission is finished. The robot then spins in place at the goal. The README asks the robot to stop there, but the check that it has stopped reads only its forward speed. The same gap gave 1 false accept on compositions-100. A second gap appeared on the first pilot, where 1 edit disabled the branch of the controller that turns the robot in place. The hidden scenarios never reach that branch, because the generator samples start headings in a limited range. The README asks for correct behaviour from any start pose in the supported ranges and does not state those ranges.
Operator-library recogniser. The recogniser holds the operator library. It searches a task’s code for the replacement text of every operator, with identifiers as wildcards, and reverts each match. Its repair is verified on 0.84 of the main-run tasks. The recogniser and configuration lookup (Appendix § 9.§ 5) measure strategies that use the generator. They are not an upper bound on a model trained on the operator library.
Other checks. One hidden test was replaced after it rejected a correct repair (Appendix § 14.§ 2). A nearest-neighbour classifier that predicts a task’s fault label from its sources was right on 0.27 of the first pilot, against 0.20 for always guessing the most common label. The pilot has five labels, the four families and a multi-component label, with six of its 30 tasks each. Grading the same submissions again reproduced every verdict.
Results in detail: repair
This section supports the opening of § 4 (Results) and RQ1 (Section 4.1).
Run setup
All models were called through one proxy under their aliases, so model snapshots are not pinned. No request set a temperature, a seed, or a reasoning or thinking parameter. The stored result rows record reasoning tokens only for GPT-5.5, non-zero in 297 of its 297 rows. For the other models they record none, which can mean that none were used or that none were reported.
A repair agent works with one of two tool sets. With standard tools it can list, read, search and edit files, run the public tests, simulate the four public scenarios and submit. Robotics tools add the diagnosis probes, which measure the running robot, as tools. Repair grading stays the same.
Each model makes one attempt per task. An attempt is capped at 25 turns, 300 s and 2 USD at list prices. GPT 6.1 has no entry in our price table, so its cap uses the prices of GPT-5.5. An attempt ends when the agent calls submit, reaches a cap or stops. The verifier grades the working copy as the agent left it, however the attempt ended.
Intervals are 95% percentile bootstrap intervals over 2000 resamples of operator sets. An operator set is the operators a task is built from, one for a single and two for a pair. The intervals cover resampling of the observed operator sets. They do not cover new fault mechanisms or repeated attempts. Where every operator set has the same value the bootstrap collapses. tbl:10 then shows, with a dagger, the Wilson interval at the sample size corrected for clustering. Whether a pair is fully repaired depends almost entirely on its operator set. A paired test that treats generated tasks as independent, such as the exact McNemar test, overstates the evidence.
Full repair table
tbl:10 gives every repair run on ROBODX-HARD-100 with both tool sets, beside three rows without a model. The recogniser row is the operator-library recogniser of Appendix § 10. Adding the robotics tools changes the verified repair rate by +0.01 for Opus and by −0.01 for Haiku. Neither change is significant under an exact McNemar test, with for Opus and for Haiku. The supplementary column gives each run’s rate on the six supplementary pairs. The no-op row shows that the unit tests alone accept the unrepaired repository on 35 of the 94 main-run tasks.
Which edits survive
An injected edit still present after a verified repair is a surviving edit. Opus’s verified pair repairs with standard tools leave 32 surviving edits. Of these, 18 come from masking operators (Appendix § 9.§ 2). One masking operator lowers the executor rate, how often the software’s message loop runs. The other weakens how strongly the compass corrects the heading estimate. The README states neither constant, and the configuration that the agent sees holds only the injected value. The other 14 surviving edits are edits that every evaluated check tolerates, such as the removed speed cap and a changed gyro weight.
tbl:11 splits the surviving edits of every run by operator. Whether an edit survives depends almost entirely on its operator. Over all ten runs, the executor-rate edit survives in 114/115 of the verified repairs of pairs that contain it. The compass-correction edit survives in 60/60 of such repairs. The speed-cap edit survives in 38/38, although the README of every speed-cap pair states the rated
| agent | singles (44) | pairs (50) | main run (94) | behaviour-pass | clustered 95% CI | full repair (pairs) | supplementary (6) | all tasks (100) |
| no-op (software tests pass) | 24/44 | 11/50 | 35/94 | – | – | – | 0/6 | – |
| operator-library recogniser | 35/44 | 44/50 | 0.84 | – | – | 44/44 | 1.00 | – |
| reference repair | 44/44 | 50/50 | 1.00 | – | – | – | 1.00 | – |
| Haiku 5.5, standard tools | 42/44 | 27/50 | 0.73 | 0.90 | [0.54, 0.90] | 9/27 | 0.33 | 71/100 |
| Haiku 5.5, robotics tools | 43/44 | 25/50 | 0.72 | 0.90 | [0.52, 0.90] | 9/25 | 0.50 | 71/100 |
| Sonnet 5.5, standard tools | 44/44 | 48/50 | 0.98 | 0.98 | [0.93, 1.00] | 26/48 | 1.00 | 98/100 |
| Sonnet 5.5, robotics tools | 44/44 | 49/50 | 0.99 | 0.99 | [0.97, 1.00] | 27/49 | 1.00 | 99/100 |
| GPT-5.5, standard tools | 42/44 | 43/50 | 0.90 | 0.90 | [0.82, 0.98] | 23/43 | 1.00 | 91/100 |
| GPT-5.5, robotics tools | 42/44 | 45/50 | 0.93 | 0.93 | [0.82, 0.99] | 26/45 | 1.00 | 93/100 |
| GPT 6.1, standard tools | 42/44 | 49/50 | 0.97 | 0.98 | [0.92, 1.00] | 25/49 | 1.00 | 97/100 |
| GPT 6.1, robotics tools | 43/44 | 49/50 | 0.98 | 0.99 | [0.94, 1.00] | 25/49 | 1.00 | 98/100 |
| Opus 5.5, standard tools | 44/44 | 49/50 | 0.99 | 0.99 | [0.96, 1.00] | 17/49 | 0.83 | 98/100 |
| Opus 5.5, robotics tools | 44/44 | 50/50 | 1.00 | 1.00 | [0.96, 1.00]† | 21/50 | 1.00 | 100/100 |
caption: Repair on ROBODX-HARD-100. Singles and pairs count verified repairs. Main run is the verified repair rate (VRR) over 94. The 95% CI is over operator sets. Behaviour-pass uses hidden scenarios only. Full repair counts pair repairs removing both edits. All tasks counts verified repairs over all 100 tasks. VRR is near 1.00 for most runs, and full-repair counts are low.
:::
| run | full / verified | clustered 95% CI | executor rate | compass corr. | speed cap | gyro (layered) | gyro (callback) | heading gain | strict | lenient |
| Haiku 5.5, standard tools | 9/27 | [0.03, 0.76] | 12 | 6 | 0 | 0 | 0 | 0 | 8/8 | 9/9 |
| Haiku 5.5, robotics tools | 9/25 | [0.00, 0.82] | 10 | 6 | 0 | 0 | 0 | 0 | 9/9 | 9/9 |
| Sonnet 5.5, standard tools | 26/48 | [0.22, 0.89] | 12 | 6 | 4 | 0 | 0 | 0 | 15/19 | 26/30 |
| Sonnet 5.5, robotics tools | 27/49 | [0.24, 0.89] | 11 | 6 | 5 | 0 | 0 | 0 | 15/19 | 26/30 |
| GPT-5.5, standard tools | 23/43 | [0.21, 0.87] | 10 | 6 | 3 | 1 | 0 | 1 | 14/16 | 23/26 |
| GPT-5.5, robotics tools | 26/45 | [0.24, 0.91] | 11 | 6 | 2 | 1 | 0 | 1 | 15/17 | 26/28 |
| GPT 6.1, standard tools | 25/49 | [0.18, 0.88] | 12 | 6 | 6 | 0 | 0 | 0 | 14/19 | 25/30 |
| GPT 6.1, robotics tools | 25/49 | [0.18, 0.88] | 12 | 6 | 6 | 0 | 0 | 0 | 14/19 | 25/30 |
| Opus 5.5, standard tools | 17/49 | [0.09, 0.66] | 12 | 6 | 6 | 5 | 3 | 0 | 14/19 | 17/30 |
| Opus 5.5, robotics tools | 21/50 | [0.16, 0.74] | 12 | 6 | 6 | 3 | 2 | 0 | 15/20 | 21/31 |
| all ten runs, surviving / verified instances | – | – | 114/115 | 60/60 | 38/38 | 10/106 | 5/45 | 2/38 | – | – |
caption: Full repair on the 50 main-run pairs, by run. Columns give full over verified repairs, the clustered 95% CI, surviving edits per operator, and full repairs on README-determined pairs under two rules. The last row counts surviving over verified instances in ten runs. The text discusses the first three operator columns.
:::
speed that the cap enforces. The differences between models come from the two gyro-weight edits. GPT 6.1 and Sonnet often set a gyro weight to a value other than the reference, which counts as removed. If we count only verified pair repairs whose code at both edits equals the reference, Opus has 14 of 49, GPT 6.1 has 11 of 49 and Sonnet has 10 of 48.
Under the strict rule of Appendix § 9.§ 4, 20 of the 50 main-run pairs are README-determined. The lenient rule, shown in the table’s lenient column, also counts the gyro weights and finds 31 such pairs. It counts them because the visible code fuses gyro and odometry headings with weights that sum to one, so a gyro weight above one gives odometry a negative weight. Which rule fits is a judgement, so the table shows both.
Some verified single-fault repairs also keep the injected text, 5 of Opus’s 44 verified singles with standard tools, mostly from operators that inject the same edit at several sites. A stricter rule counts a verified single as keeping the edit only if the function that the operator mutated is still identical to its faulty version after the repair. Under it Opus keeps the injected executor rate on 2 verified singles with standard tools and 2 with robotics tools.
The run without the request for minimal fixes
The repair prompt (Appendix § 17.§ 2) contains the sentence “Prefer minimal, root-cause fixes over symptom suppression.” We re-ran Opus with standard tools on the 50 main-run pairs with that sentence removed and everything else unchanged. The new run verifies 50 of 50 pairs and fully repairs 24 of them. The original run fully repairs 17 of its 49 verified pairs. On the 49 pairs verified in both runs, 6 gain a full repair and 0 lose one. The one pair that only the new run verifies is also fully repaired.
Each of the 6 gained full repairs removes a gyro-weight edit, and none of them restores the reference value. If we count only repairs that put the reference text back at every injected edit, the original run has 14 of 49 and the new run 14 of 50. Without the sentence the masking edits survive 18 times and the speed-cap edit 6 times, as often as with it. An exact McNemar test on the gained and lost pairs gives , unclustered. All gains come from operator sets that contain a gyro-weight edit, so this test overstates the evidence. Each task has one attempt in each run. The new run took place about a month after the original repair runs. A change in the served model between the two runs is not ruled out.
The run with a shell agent
Every other repair agent in the paper runs in our own agent loop, which offers a fixed set of tools as function calls and no shell (Appendix § 11.§ 1). To check whether this loop explains the surviving edits, we ran Opus under mini-swe-agent, version 2.2.7, a shell-only agent from the authors of SWE-agent [113]. The agent works in a shell inside a container built from the sandbox image of the original runs, with no network. It can run the public tests and the simulator, and it can write its own scripts and scenarios. Its prompt keeps the text of the repair prompt, including the request for minimal fixes. An attempt may take up to 75 steps and 3 USD. The tasks are 20 of the 50 main-run pairs, drawn at random with a fixed seed. Each task has one attempt, and the same verifier grades it.
Under the shell agent Opus’s repair is verified on all 20 pairs and removes both injected edits on 8. In the original run on the same 20 pairs, 19 repairs are verified and 6 remove both. On each of the 12 pairs where the shell agent leaves an edit in place, the original run leaves the same edit. The shell agent’s two additional full repairs both change a gyro-weight constant. In each the agent tries several values with a script of its own and keeps one that differs from the reference value. If we count only repairs that put the reference text back at every injected edit, the two runs agree, with 6 of 20 for the shell agent and 6 of 19 for the original run. The agent itself submitted 10 of the 12 attempts that leave an edit in place, and the other 2 ended at the cost limit. In all 12 the line that holds the edit appears in the agent’s shell output, and no later command of the agent refers to it. The limits on an attempt do not explain these edits. Twenty pairs with one attempt each cannot show that the two loops give the same rate. The check gives no sign that our loop causes the surviving edits. As in the run without the request for minimal fixes (Appendix § 11.§ 4), the original run is about a month older, and a change in the served model is not ruled out.
Results in detail: diagnosis
This section supports RQ2 and RQ3 (Sections 4.2 and 4.3) and the last two limits of § 6. All results are on the 94 main-run tasks of ROBODX-HARD-100 unless a sentence says otherwise.
Full diagnosis table
Table 12 is the full version of Table 4. It adds a row for configuration lookup and a column for the six supplementary pairs.
| agent | singles (44) | pairs (50) | main run (94) | supplementary (6) |
| EIG oracle | 44/44 | 50/50 | 1.00 | 1.00 |
| random probing, exact inference | 22/44 | 47/50 | 0.73 | 1.00 |
| configuration lookup, no probes (expected) | 25.9/44 | 48.5/50 | 0.79 | 6.0/6 |
| Opus 5.5, budgeted probing | 32/44 | 10/50 | 0.45 | 0.67 |
| GPT-5.5, budgeted probing | 34/44 | 2/50 | 0.38 | – |
| Haiku 5.5, budgeted probing | 10/44 | 0/50 | 0.11 | 0.17 |
| direct prediction, no probing | ||||
| Opus 5.5, reference-probe observations | – | 3/50 | 38/94 | – |
| GPT 6.1, reference-probe observations | – | 5/50 | 36/94 | – |
| Sonnet 5.5, reference-probe observations | – | 2/50 | 30/94 | – |
| Haiku 5.5, reference-probe observations | – | 0/50 | 27/94 | – |
| Opus 5.5, signature table | – | 50/50 | 94/94 | – |
| GPT 6.1, signature table | – | 50/50 | 93/94 | – |
| Sonnet 5.5, signature table | – | 50/50 | 93/94 | – |
| Haiku 5.5, signature table | – | 48/50 | 89/94 | – |
Table 12. Diagnosis on ROBODX-HARD-100. Entries count task-correct answers (Appendix § 9.§ 4) out of 44 singles, 50 pairs and six supplementary pairs, and a main-run rate over 94. Above the rule, policies choose probes, and EIG is expected information gain. Below it, every model gets the same observations. The pairs column carries the RQ2 comparison.
Random probing with exact inference is right on fewer singles than Opus. In 60 of the 100 catalogues of ROBODX-HARD-100, supplementary pairs included, the initial observation does not single out the true hypothesis. On such a single, random probing often spends its budget before it picks the probe that tells the hypotheses apart.
In expectation, configuration lookup (rule in Appendix § 9.§ 5) is right on 25.9 of 44 singles and on 48.5 of 50 pairs. Its expected rate over the main run is 0.79. On pairs and over the main run, this score is above random probing and above every agent that chooses its own probes.
The verifier compares observations with exact equality. If a numeric observation may differ from a signature by up to 10%, the full menu of probes can still single out the true hypothesis in 89 of the main-run catalogues, against 90 under exact comparison. Opus’s correct answers in budgeted probing then go from 42 to 43, and its gap to the reference procedures stays in place. The tolerant count of Table 14, 77, is a different quantity. It uses only the initial mission and the EIG oracle’s probes, as in the direct-prediction study, and counts tasks on which the same 10% rule applied to the simulator’s signatures is task-correct.
| model | observations | + own predictions | + signature table | worksheet | cells within 10 % | categorical cells | truth kept (10 %) |
| Claude Opus 5.5 | 38 (3) | 40 (7) | 94 (50) | 35 (4) | 0.15 | 0.71 | 0/94 |
| GPT 6.1 | 36 (5) | 39 (7) | 93 (50) | 30 (4) | 0.18 | 0.62 | 2/94 |
| Claude Sonnet 5.5 | 30 (2) | 28 (2) | 93 (50) | 30 (1) | 0.17 | 0.67 | 1/94 |
| Claude Haiku 5.5 | 27 (0) | 30 (1) | 89 (48) | 28 (0) | 0.14 | 0.66 | 1/94 |
| plain string comparison | 94 (50) | tolerant elimination: 10 % 77, categorical only 19 |
Table 14. Direct prediction on the 94 main-run tasks of ROBODX-HARD-100 (50 pairs). Diagnosis columns count task-correct answers, pairs in parentheses. The worksheet arm asks the model to mark each hypothesis consistent or not. The last three columns score each model’s own signature predictions against the simulator. The signature-table column gives the rates quoted in Section 4.3.
Probing also differs in cost. On average the EIG oracle spends 0.12 of the probe budget, and Opus spends 0.79 of it.
Two controls for Opus
The two controls of § 2 were run for Opus on every main-run task (Table 13). The full observation table gives Opus the true system’s observation under every probe, with no budget, and asks for one answer. With it Opus is task-correct on 0.54 of the main run, against 0.45 with budgeted probing. 12 misses under budgeted probing become correct, and 3 correct answers become wrong. The difference between the two rates is small. Whether it is significant depends on the test once tasks are clustered by operator set (Appendix § 11.§ 1).
| Opus 5.5 run, and what the agent is given | RoboDx-HARD-100 main run (94) |
| budgeted probing: the probes it picks within the budget | 0.45 |
| full observation table: every probe’s observation of the true system, no budget, one answer | 0.54 |
| forward model: budgeted probing plus the signature of any catalogue hypothesis on request | 1.00 |
Table 13. Diagnosis controls for Opus on the 94 main-run tasks of RoboDx-HARD-100. Entries are task-correct rates (Appendix B.4). Each row says what the agent is given. The forward model reaches 1.00, and the full observation table is near budgeted probing.
The forward model keeps the budget and adds a tool, predict(hypothesis, action), where the action is a probe. The tool returns the signature of any catalogue hypothesis under any probe. With it Opus is task-correct on 1.00 of the main run. 52 misses under budgeted probing become correct, and no correct answer becomes wrong. On the running example, the tool shows that with the low heading gain alone the robot completes the mission given in rotated coordinates. The true system, as Opus observed with its fifth probe, circles through 24 rad and does not finish (Table 2). With the forward model, Opus named both faults (§ 2).
The direct-prediction study
Table 14 gives the results of the direct-prediction study of Section 4.3. Its bottom row scores the signature table with no model, by exact comparison, by comparison within 10% on numeric fields, and by comparison on categorical fields only. Its truth-kept column counts tasks where elimination with the model’s own predictions within 10% keeps the true hypothesis. In the worksheet arm the model gets the reference-probe observations and must mark every hypothesis consistent or inconsistent, citing the deciding field, before it answers. Without the signature table, the four models name both faults on 0 to 5 of the 50 pairs. Adding the table raises this to 48 to 50. Only 0.14 to 0.18 of a model’s predicted observations match the signature on every field within 10%.
Given its own signature predictions beside the observations, Opus wins 6 tasks and loses 4 against the observations alone. An exact McNemar test finds no detectable gain for any of the four models ( for Opus, for GPT 6.1, for Sonnet, for Haiku).
The plain string comparison is task-correct on 94 of 94 tasks, so the signature table turns diagnosis into a lookup. Compared on categorical fields alone, such as whether a mission completes, the signatures single out the true hypothesis on 19 of 94 tasks. On pairs they do so on 4 of 50.
Failure audit
We audited every failed submission of every reported run against the verifier’s ground truth. For diagnosis, we replayed each agent’s probes against the verifier’s table of observations. The audit found no error in what the diagnosis agent is given. Every catalogue contains the true hypothesis, and every probe observation shown to Opus equals the verifier’s entry.
Table 15 sorts the wrong answers into classes. Of Opus’s 52 wrong answers, 37 contradict an observation Opus itself made. In 18 of these, the answer predicts a different mission outcome or state field, for example a completed mission where Opus saw it fail. 10 answers are hedges. 5 answers are consistent but incomplete. They agree with all the evidence and name only part of the hypotheses it cannot tell apart.
| set | run | correct | answer refuted by own evidence — categorical | answer refuted by own evidence — numeric only | hedged (truth + extra) | consistent but incomplete | truth unique given evidence | |
| compositions-100 | Opus, probing | 100 | 48 | 9 | 16 | 11 | 16 | 36 |
| compositions-100 | Haiku, probing | 100 | 18 | 14 | 41 | 5 | 22 | 45 |
| compositions-100 | Opus, full observation table | 100 | 53 | 18 | 20 | 0 | 9 | 37 |
| interaction/spec | Opus, probing | 28 | 10 | 7 | 7 | 2 | 2 | 16 |
| interaction/spec | Haiku, probing | 28 | 6 | 6 | 12 | 1 | 3 | 18 |
| interaction/spec | Opus, full observation table | 28 | 12 | 9 | 6 | 0 | 1 | 14 |
| ROBODX-HARD-100 | Opus, probing | 94 | 42 | 18 | 19 | 10 | 5 | 46 |
| ROBODX-HARD-100 | Haiku, probing | 94 | 10 | 33 | 36 | 1 | 14 | 61 |
| ROBODX-HARD-100 | GPT-5.5, probing | 94 | 36 | 25 | 26 | 1 | 6 | 52 |
| ROBODX-HARD-100 | Opus, full observation table | 94 | 51 | 21 | 21 | 0 | 1 | 42 |
Table 15. Wrong diagnoses classified by the agent’s own evidence. Refuted answers contradict an observation the agent made, by a categorical field or by numbers only. Hedged answers add an excluded hypothesis to the true hypothesis. Consistent but incomplete answers agree with all the evidence and name only part of the hypotheses it cannot tell apart. The last column counts wrong answers made after the evidence singled out the true hypothesis.
In 46 of the 52 wrong answers, the probes Opus had bought were already enough to single out the true hypothesis. This class holds every hedge and all but one refuted answer. Exact inference on the probes Opus chose is task-correct on 0.94 of the main run. Opus loses accuracy after it has chosen its probes. For Haiku, most wrong answers also come after the evidence singles out the true hypothesis.
An unverified repair counts as leaving its fault in place if every hidden requirement it fails also fails without repair, or if the mutated function fails its hidden property test. Every unverified Opus repair on every set leaves its fault in place or is a regression, a failed requirement that the original defect did not break. The audit found one defect in the hidden checks, a test that rejected a correct repair (Appendix § 14.§ 2).
The real set
This section supports RQ4 (Section 4.4) and the real set of § 3.
| repository | candidate PRs | fail-to-pass gate run | gate passed | in the real set |
| ArduPilot | 56 | 128 | 67 | 54 |
| Nav2 | 108 | 108 | 41 | 30 |
| tf2 | – | 21 | 14 | 13 |
Table 16. Mining of the real set by repository. The first column counts the curated candidate PRs. The other columns count tasks, and a backported fix gives several tasks per PR. ArduPilot has no backports, so its tasks are PRs. Its gate ran on the 56 curated candidates plus 72 PRs from a wider pool of merges that touch tests, 128 PRs in all. The tasks of the real set are those that passed the gate less those the quality audit removed. tf2 has no separate candidate list.
How the real set was built
The real set has 97 tasks from three open-source robot projects. Of these, 54 come from the ArduPilot autopilot, 30 from Nav2 [74], the navigation stack of the Robot Operating System (ROS), and 13 from tf2, the ROS library that tracks coordinate frames and the successor of tf [36]. Each task comes from a merged bug-fix pull request (PR). The agent gets an issue text and the repository at the commit just before the fix. It may read the whole repository and may edit only the directories that the fix touched. A backport is a task that applies an existing fix to another ROS 2 distribution, a separate release line. Of the tasks, 14 are backports (7 in Nav2 and 7 in tf2).
Gates. A mined PR became a task only if its own test change fails at the commit before the fix and passes at the fix commit. The fix itself had to be verified and an empty patch rejected. For ArduPilot a fresh run had to reproduce the flip from failing to passing. A quality audit then removed 25 of the 122 tasks that passed these gates. Most were removed for an unfair hidden test or for a change that is not a behavioural repair.
Issue texts. On 86 of the 97 tasks the issue text is a rewrite by Claude Opus 5.5 that describes only the symptom. On the rest it is the PR body or a linked issue. The rewriting model saw the PR, the linked issue and the failing test output. It also saw the names of the hidden tests and the diffs. It was told not to name the fix, the files it touches or the hidden tests. Model judges then checked each text for actionable symptoms and for leaks of the fix, with disagreements adjudicated by a further model pass. No human labelled the texts.
The grader. The grader applies the agent’s patch to a clean checkout of the commit before the fix. It takes the hidden test files from the fix commit, so an edit to a test cannot make it pass. It then builds the code and runs the PR’s own test change. Every test that the fix turns from failing to passing must pass, and a fixed set of regression tests must not fail. ArduPilot tasks run a software-in-the-loop autotest, which drives the code against a vehicle simulator. Nav2 and tf2 tasks run unit tests built with the colcon tool. The agent loop and tools are those of the generated tasks (Appendix § 11.§ 1), with one attempt per task.
Results
Table 17 gives each model’s rates under both readings. The strict reading counts a task only if its repair was verified in the run. The lenient reading also counts a failed result whose saved patch was verified on one later re-grade at lower machine load. The 9 failed results without a saved patch were not re-graded. Of 47 re-graded results, 25 were verified. Of the 21 re-graded results that had failed the regression tests, 19 were verified. We attribute these passes to load on the shared machine by exclusion, since the tests then passed in isolation. We later ran every set five times in isolation at the fix commit. None of the 122 hidden-test instances changed verdict, and 31 of 2959 regression-test instances did.
The paired test is an exact McNemar test on the tasks where only one of two models is verified. Under the strict reading, after a Holm correction for three tests, it separates Haiku from Opus alone (adjusted for Opus and for GPT 6.1 and Sonnet). Under the lenient reading it separates Haiku from each of the three stronger models, with largest adjusted . On the tasks that are not backports, the strict rates are 0.87 to 0.90 for the stronger models and 0.76 for Haiku. The Wilson 95% intervals treat tasks as independent, although each backport shares its fix with another task. Neither the strict nor the lenient rate is a bound on correctness.
| model | main run (94), std. tools | verified | strict | 95% CI | lenient | non-backport | ArduPilot | Nav2 | tf2 |
| Claude Opus 5.5 | 0.99 | 88/97 | 0.91 | [0.83, 0.95] | 91/97 (0.94; [0.87, 0.97]) | 0.90 | 48/54 | 27/30 | 13/13 |
| GPT 6.1 | 0.97 | 85/97 | 0.88 | [0.80, 0.93] | 93/97 (0.96; [0.90, 0.98]) | 0.87 | 50/54 | 22/30 | 13/13 |
| Claude Sonnet 5.5 | 0.98 | 85/97 | 0.88 | [0.80, 0.93] | 92/97 (0.95; [0.88, 0.98]) | 0.87 | 46/54 | 26/30 | 13/13 |
| Claude Haiku 5.5 | 0.73 | 74/97 | 0.76 | [0.67, 0.84] | 81/97 (0.84; [0.75, 0.90]) | 0.76 | 39/54 | 23/30 | 12/13 |
Table 17. Verified repair on the 97 real-set tasks with standard tools. Column ROBODX-HARD-100 repeats the main-run rate. Strict is the share verified in the run, with a 95% interval. Lenient adds one re-grade of the saved patch. Non-backport is the strict rate on 83 tasks. Haiku is lowest under both readings.
What limits the reading
Copied fixes. Because the PRs are public, a model may have seen a fix in training. We measured the share of a fix’s distinctive added lines that a verified patch reproduces word for word. The mean share is 0.33 for Haiku and up to 0.42 for Opus. The exact reproductions are one-line fixes with a single correct edit. For every model the share is lower on PRs merged before 2026 than on PRs merged in 2026 (Table 18). No model’s training cutoff is recorded, so this split does not show that the 2026 PRs are unseen. It also cannot tell no recall apart from recall in both periods. The comparison has no test and is confounded with the size of the fix.
| model | before 2026, mean | before 2026, | 2026, mean | 2026, |
| Opus 5.5 | 0.33 | 30 | 0.47 | 52 |
| GPT 6.1 | 0.21 | 27 | 0.46 | 54 |
| Sonnet 5.5 | 0.34 | 28 | 0.40 | 52 |
| Haiku 5.5 | 0.27 | 21 | 0.36 | 47 |
Table 18. Mean share of a PR fix’s distinctive added lines that a verified patch reproduces, by the year the PR was merged. counts the verified patches scored. No test or interval.
The editable scope. We re-ran 20 non-backport tasks that Opus had already verified with the whole repository editable. Opus verifies 20 of them and GPT 6.1 19. Sonnet verifies 18 and Haiku 17. Removing the limit changes little on these tasks. The tasks were drawn from Opus’s successes, so the re-run does not measure the average effect of the limit. A second re-run on 20 non-backport tasks drawn at random found no difference between the two scopes that an exact McNemar test finds significant. Twenty tasks can rule out only a large effect.
Run-to-run variance. The second re-run above also ran its 20 random tasks for every model with the editable scope of the reported runs, limited to the fix’s directories. These fresh attempts are the only repeats under the reported condition. They were graded by the final grader, which re-runs a failing test once in place, and with more builds running at once, so they are not a pure repeat. Their verdicts differ from the strict reading on 7 of 80 model-task results and from the lenient reading on 5. The whole-repository runs of the two re-runs share some tasks, and on these 3 of 28 results differ.
Comparison with the generated tasks. The three stronger models score 0.08 to 0.10 lower on the real set than on ROBODX-HARD-100 under the strict reading. The two sets differ in the grader, the editable scope and the mix of tasks. The editable scope, the size of the fix and the hidden test may each affect the level of the scores. The lower rates cannot be read as a loss of skill.
Earlier sets and changes after the first results
This section supports § 2 and § 4. It describes the four earlier sets, one of which holds the running example, and lists the changes made after the first model results.
Earlier generated sets
The same generator built four sets before RoboDx-Hard-100. On a first pilot of 30 tasks, 24 with one fault and 6 with two, the operator-library recogniser of Appendix § 10 repaired 0.97 of the tasks. This result led us to weight the later sets toward tasks with several faults. A second pilot of 38 tasks followed changes to the operators that made them harder to recognise. Compositions-100 has 30 singles, 55 pairs and 15 tasks with three faults. Interaction/spec has 16 singles whose fix needs a rating stated in the README, and 12 pairs.
The running example is a pair of interaction/spec, and RoboDx-Hard-100 holds pairs built from the same two operators. The running example is one of 5 interaction/spec pairs in which one fault alone already fails a hidden scenario. In the other 7, the interaction-only pairs, both faults are quiet, and neither fault alone fails a hidden scenario. With standard tools Opus’s repair is verified on 0.96 of interaction/spec (Table 19), including 11 of the set’s 12 pairs. In diagnosis Opus names both faults on 2/12 pairs. The EIG oracle and random probing are each right on 12/12. Each of Opus’s 2 correct pair answers is on an interaction-only pair.
| agent | tools | compositions-100 () VRR | compositions-100 () SW | compositions-100 () SW∧¬V | interaction/spec () VRR | interaction/spec () SW | interaction/spec () SW∧¬V |
| no-op (unchanged repo) | – | 0.00 | 0.08 | 8 | 0.00 | 0.57 | 16 |
| operator-library recogniser | – | 0.85 | 0.88 | 3 | 0.79 | 0.79 | 0 |
| reference repair | – | 1.00 | 1.00 | 0 | 1.00 | 1.00 | 0 |
| Claude Opus 5.5 | standard tools (repaired runtime) | 0.91 | 1.00 | 9 | 0.96 | 0.96 | 0 |
| Claude Opus 5.5 | robotics tools (repaired runtime) | 0.97 | 1.00 | 3 | 0.96 | 1.00 | 1 |
| Claude Haiku 5.5 | standard tools (repaired runtime) | 0.86 | 0.92 | 6 | 0.68 | 0.79 | 3 |
| Claude Haiku 5.5 | robotics tools (repaired runtime) | 0.91 | 0.97 | 6 | 0.64 | 0.79 | 4 |
Table 19. Repair on compositions-100 and interaction/spec, one attempt per task. VRR is the verified repair rate. SW is the share of repairs that pass every unit test. SW∧¬V counts repairs that pass every unit test and fail the hidden checks. Repaired runtime marks runs made after the robotics-tool runtime was repaired (Table 20). The Haiku rows come from that separate run on the repaired runtime. They do not use Haiku’s three repeated attempts of Appendix § 17.§ 3. The text discusses the Opus rows.
Changes after the first results
Table 20 lists the changes made after a model had been run that affect a reported number. Changes to file names, metadata and documentation came after every reported run and change no input of one. A configuration check added to the verifier changed 0 of the 3307 stored verified repairs. A hidden test of pose integration was also rewritten, and it had rejected no model submission.
| what changed | why | reported number it affects |
| diagnosis catalogues re-balanced, every diagnosis condition re-run | their make-up gave the true composite away | every diagnosis number |
| eight ROBODx-HARD-100 tasks replaced by tasks with logic faults | their fix value was printed in the README | every run on ROBODx-HARD-100 |
| issue texts of 70 main-run tasks re-rendered, affected repair rows re-run | a model-judge audit found symptom terms that did not match the behaviour | repair of Opus, Haiku and GPT-5.5, Haiku’s change with robotics tools |
| lookahead hidden test replaced by property checks, 10 failed submissions re-graded, 6 of them verified | it rejected an equally correct ordering of the formula for the lookahead point, the route point the controller steers toward | repair verdicts on one task |
| robotics-tool runtime repaired, affected sets re-run | it failed 34% of simulation requests | robotics-tools rates in Table 19 |
| capped attempts graded from the working copy | they had counted as not submitted | 7 Opus repair rows |
| check for a surviving edit made rename- and number-aware, configuration keys looked up through the identifier map | the old check matched canonical text only, and 5–8 edits per run were unresolved | every full-repair rate |
| 4 Opus robotics-tools rows re-graded, later run again | their grading failed on a full disk | Opus’s robotics-tools rate on ROBODx-HARD-100 |
| ties of the reference procedures scored by the verifier’s rule, no longer as 1/ | one rule for all rows | reference-procedure rates |
| list-valued fields of refuted answers typed element by element | one shared typing rule | refuted answers by type |
| real-set texts rewritten as symptom reports, 13 ArduPilot tasks excluded, Nav2, tf2 and three models added | re-validation, a quality audit and adjudication | every real-set number |
| lenient reading, each failed real-set row re-graded once at lower load | the regression tests proved load-sensitive | the lenient reading |
| two fail-to-pass tests removed | they pass at the bug commit in every run | real-set verdicts |
| four real-set texts rewritten and re-run, re-grade counts recomputed | a further model audit | real-set verified and re-grade counts |
Table 20. Changes made after a model had been run that affect a reported number. Rows are grouped into task inputs and grading of ROBODx-HARD-100, scoring and accounting, and the real set. Superseded result files are kept.
The selection rules of RoboDx-Hard-100 were fixed before the first model run on it. Every later change to the protocols or the reference procedures is a row of Table 20. The masking operators (Appendix § 9.§ 2) were written after agents on the earlier sets had been seen to leave one of two faults in place. They supply most of the edits that survive Opus’s verified pair repairs. The full-repair analysis, the diagnosis controls, the direct-prediction study, the tolerance replays of the reference procedures and the repair-prompt ablation were designed after the first results and are post hoc. The harder tier of real tasks (Appendix § 16) is post hoc as well. The choice of models was not fixed in advance for the diagnosis track (Section 4.2) or for the real set.
Related work in full
This section supports § 5 with the full survey of related work. Table 21 sets RoboDx and its real set against the nearest benchmarks, with cells taken from the cited papers.
| benchmark | defects | host code | behaviour-graded in simulation | hidden oracle | root cause scored | metered probes |
| RoboBugBench [104] | 33 real | real ROS 2 | yes | yes | no | no |
| SWE-smith / PDB [114, 15] | generated | real repos | no | unit tests | edit precision | no |
| DDBench [112] | 60 real | real | no | reproducer | no | context arm |
| Active-SWE [66] | real | real | no | unit tests | hunk match | unbudgeted |
| ROS Help Desk [58] | runtime-injected | real ROS | no | rubric | yes | unmetered |
| CaRE [46] | configuration | real | partial | no | yes | |
| RoboFuzz / PGFuzz [64, 61] | found by fuzzing | real | yes | policy oracles | no | no |
| FixDrive / DrPlanner [97, 68] | violations found in simulation | Apollo strategy DSL / planner heuristic and cost code, parameters | yes | scenario oracle | no | no |
| Multi-SWE-bench (C/C++) [120] | real | real repos | no | unit tests | no | no |
| Cloud-OpsBench / OpenRCA [109, 111] | injected cloud faults | services | no | label match | yes | step cap, no oracle |
| RoboDx | generated, mechanism-typed | generated, renamed | yes (kinematic unicycle plant) | behavioural + metamorphic | greedy EIG reference, exact inference | budgeted, EIG reference |
| RoboDx real set | 97 fix PRs (ArduPilot, Nav2, tf2) | real | SITL autotests for ArduPilot; gtest for Nav2/tf2 | fixing PR’s tests | no | no |
Table 21. How ROBODX and its real set differ from the nearest benchmarks, from the cited papers. No prior row combines generated code-level defects, hidden checks of simulated robot behaviour, exact root-cause scoring and metered probing. The real set is graded by the tests that each fixing pull request (PR) adds. SITL is software in the loop, the autopilot run against a vehicle simulator. gtest is the GoogleTest unit-test framework.
Software-engineering benchmarks graded by tests. SWE-bench and its descendants grade an agent’s patch by the repository’s own tests [52, 121, 77, 48, 9]. A multilingual version adds tasks in C and C++ [120]. Audits find these verdicts often wrong [110, 116, 5]. Program-repair research had found this earlier [92, 82].
HWE-Bench grades repairs of hardware bugs with testbenches, the simulation harnesses of hardware design [26]. DDBench compares agents given only the symptom with agents given extra context [112]. In Active-SWE the agent gets no issue report [66]. Interactive debugging is studied in its own environments and methods [118, 55]. One work debugs multi-agent systems by intervening on their traces [73].
To our knowledge no version of SWE-bench covers a robotics repository. None of these benchmarks asks the agent to name the cause. RoboDx grades a repair by how the simulated robot drives in hidden scenarios and adds a diagnosis track.
Robot software and its testing. Bug studies catalogue the faults of real robot software. ROBUST lists 221 bugs in software for the Robot Operating System (ROS) [102]. Other studies cover bugs in autonomous vehicles and drone autopilots [39, 107] and bugs in how ROS components interact [20]. Further studies cover fix patterns in two driving stacks, misconfigured ROS systems and drone flight logs with anomalies [19, 14, 108]. We wrote the operators ourselves, without sampling them from the studied bugs, and then mapped them to the taxonomies of ROBUST, the interaction-bug study and the vehicle and drone studies.
Detectors find classes of these faults, such as mismatched physical units, wrong reference frames and misconfigured software architecture [75, 56, 57, 101]. One tool infers component models of ROS systems for such analysis [32]. One tool learns a model of normal behaviour from the telemetry of the ArduPilot autopilot and uses it to judge runs [4]. Others repair misconfigured robot swarms or compare versions of one autonomous system [53, 115]. A study lists the challenges of testing robots [2]. Robot repositories test their control code only superficially [23].
Property-based testing, which checks a stated property on many random inputs, has been proposed for ROS [90]. Control invariants derived from vehicle dynamics detect attacks at run time [21]. Fuzzers feed robotic vehicles and ROS systems many generated inputs and flag unstable control, broken physical laws or violated safety policies [65, 64, 61]. Other work generates stressful trajectories or lets agents test autopilots in simulation [45, 33]. Studies of ArduPilot and of navigation software find that most bugs can be reproduced in simulation [103, 94]. Developers report that simulation is hard to use for testing [3]. Metamorphic relations state how an output must change when the input changes in a known way [17]. They have been used to test autonomous systems [123, 69, 100, 1, 8, 105]. In several works agents write new control code that is scored by running it, in simulation or against task metrics [72, 34, 37, 91, 106].
The closest prior work, RoboBugBench, grades LLM repairs of 33 real defects in a ROS 2 navigation stack in a physics simulator. It reports repair rates of 24–36% [104]. PGPatch repairs logic bugs in robotic vehicles from policy formulas, and PatchVerif measures how a patch changes the vehicle’s physical state [62, 63]. FixDrive and DrPlanner repair driving behaviour and motion planners for automated vehicles [97, 68]. Our hidden unit tests are property tests. Our metamorphic check requires the robot’s path to shift and rotate with the mission when the whole mission is shifted and rotated in the plane. RoboBugBench uses real defects. RoboDx injects its faults, so it controls each one. None of these works checks whether the agent knows which fault caused the misbehaviour.
Fault injection and mutation. Several works build debugging tasks from synthetic bugs in existing code [114, 93, 98, 15]. SWE-smith and BugPilot keep bugs that break an existing test of a real repository. One of them, SWE-smith, already combines several bugs in one task.
Synthetic bug sets are judged by how realistic they are and by whether training on them helps on real tasks [81, 16]. SWE-at-Scale reports the same solvers on synthesised and real tasks. It does not compare the two rankings. Mutation testing injects small edits to assess a test suite [54]. Higher-order mutation combines several edits in one program [51]. Robot-specific mutation operators also exist [31]. Mutation-based artificial bugs are also used to benchmark program repair [76]. RoboDx injects known faults into a generated navigation stack with renamed modules and symbols. RQ1 can then check a verified repair for an injected edit left behind. The verifier also marks the hidden scenarios that fail only when both of a task’s faults are present.
Diagnosis and reference procedures. Model-based diagnosis compares what a system shows with what a model of it predicts [86, 25, 96]. Its classical methods choose each next measurement by expected information gain [27, 28]. Later work formalised this choice and extended it to unreliable tests [79, 83]. More recent work makes sequential diagnosis scale [88, 87]. Tables of the symptoms of each fault are older [11]. The methods have been applied to robot control software and ROS systems [95, 119], and fault detection and diagnosis for robots has been surveyed [59]. Greedy choice of measurements is provably close to optimal under adaptive submodularity [41]. We do not show that our objective meets those conditions and use the greedy policy only as a reference. Our requirement that the probes tell the true hypothesis apart is an analogue of diagnosability in discrete-event systems [89], and we do not reduce our setting to it.
Bayesian experimental design [84] is now applied to how LLMs gather information [22, 47]. Agents that gather information have been scored by expected information gain or against an exact Bayesian reference, in games, supply chains and experimental design [42, 30, 38]. These settings involve no code faults.
Most root-cause benchmarks for agents score the analysis of telemetry that the agent is given [111, 35, 80, 13]. Others have the agent work on a running system with injected faults [49, 24, 18]. One replays faults through diagnostic interfaces that the agent can query [109]. These benchmarks do not compare the agent with a reference procedure. Robot work searches driving systems for root causes, diagnoses faults injected at run time, ranks configuration options and scores root-cause text from issue tickets [70, 58, 46, 44]. To our knowledge no benchmark combines priced probes scored against an exact-inference reference, scored two-fault causes, and controls that separate probe choice from signature prediction. Our diagnosis track scores coding agents on code-level faults against such a reference.
Benchmark hygiene. Curation changes repair scores. About a third of the SWE-bench successes of SWE-agent with GPT-4 had a problem statement that leaked the solution [7]. Rewriting problem statements as realistic user requests lowers scores and can reorder models [60]. Hints that point to the faulty code raise repair rates [6, 99]. ImpossibleBench measures how often agents exploit the tests [122]. Tests strengthened against adversarial patches lower the reported success [117].
Agents retrieve upstream solutions or recall memorised code [71], and models show memorisation of benchmark repositories [67, 29, 85]. Spaghetti Architect generates new code tasks with checked reference outputs [50]. SWE-rebench V2 certifies tasks with model judges [10]. A review of benchmark practice gives recommendations for construct validity [12]. How reliably benchmarks rank systems is itself studied [43, 78]. We found no work that tests with a paired analysis whether models rank the same on generated and real tasks. Our real set also relies on model judges, and a further model pass adjudicates every disagreement (Appendix § 13).
Limitations in full
This section lists every limit of the study, under the heads of Section 6 and three more.
Simulated robot, generated faults. The simulated robot is a kinematic unicycle with no contact, perception, 3D, ROS or actuator lag. Faults that make a dynamic robot oscillate cannot be posed. The two programs under test follow waypoints with one robot. The scores do not cover manipulation or perception code. We wrote the operators and mapped them to four robot-bug taxonomies afterwards [102, 20, 39, 107]. Several categories have no operator, and some message-bus operators have no ROS 2 counterpart. The generator keeps only faults the simulator can expose. The operator mix does not show how often each bug occurs. Templates write the issue reports.
The verifier. A verified repair can keep an injected edit in a function that no hidden property test covers. One hidden test rejected an equally correct repair, and the checks do not show that every correct alternative is verified. A verified repair can leave the robot turning in place at the goal (§ 3) or fail from a start heading the hidden scenarios do not cover.
The repair prompt, the README and the agent loop. The full-repair rate counts verified pair repairs in which the injected text of every edit is gone. An edit changed to a value other than the reference counts as removed. The rate is not a defect rate. It is higher in a run a month later without the request for minimal fixes, where every gained full repair sets a gyro weight to a value other than the reference (Appendix D.4). A change in the served model is not ruled out. Most edits left in place are configuration constants that the README neither states nor bounds, so they need not mean a missed cause. The agents run in our own agent loop. A check with a shell agent on 20 of the 50 pairs gives no sign that this loop causes the edits left in place (Appendix D.5).
Small numbers. Models of two vendors, one attempt per task and few operator sets make the intervals, clustered by operator set, wide. Verdicts change between repeated attempts, so a difference of a few tasks can be run-to-run variance. Paired tests on the generated sets ignore the clustering, and their -values overstate the evidence. Each model is compared under its provider’s default settings, and the stored records cannot show whether a model reasoned by default.
The diagnosis setup. The catalogue is closed. Each operator appears in it equally often, yet its size still tells the agent how many faults to expect. The true cause is never a robot fault or no_fault, the answer that nothing is wrong. The reference procedures ignore the size cue and compare coarsened signatures by exact equality. A lexical baseline that matches hypothesis descriptions to the initial observation is task-correct on 10/94 main-run tasks. The prompt never says that a cause may have two faults. The low pair scores are not separated from a preference for single-fault answers.
The forward-model control bounds what an agent could do with a forward model. It does not measure unaided reasoning. Because it gives several aids at once, it cannot say which step Opus lacked. The controls ran in full for Opus only. Haiku ran two of them on two earlier sets. In the direct-prediction study, pair signatures identify the true hypothesis mostly through numeric fields that the README may not determine. Every arm sees the EIG oracle’s probes.
The real set. The real set covers three repositories, fixes whose pull request adds a test, four models and one agent scaffold. The result holds for these 97 tasks only. Models rewrote and judged its issue texts with no human check. The editable scope names the directory of the fix. The real set is a companion check. It does not validate the generator or stand alone as a benchmark of real debugging. On the real set only the weakest model stands apart. Under the strict reading a paired test separates that model from Opus alone. A harder tier of 5 real tasks with larger fixes, mined after the first results, does not show the same pattern, with 2/5 verified for Opus, 5/5 for GPT 6.1, 4/5 for Sonnet and 3/5 for Haiku. Each of the 14 backports shares its fix pull request with another task. The fixes are public and no training cutoff is recorded, so the share of fix lines a patch reproduces does not bound contamination (Section 4.4).
Choices made after the first results. Several analyses and the masking operators were designed after the first results, so their results should be read as post hoc (Appendix G.2). The choice of models was not fixed in advance. Two inexpensive models stopped after the first pilot, and GPT-5.5 did not run on the final real set. The real set’s tasks, issue texts and choice of models changed after earlier runs of Opus and GPT-5.5 on its ArduPilot tasks. The eight replacement tasks of RoboDx-Hard-100 meet neither structural selection criterion and lack the canary string that lets training corpora be checked.
What the generator’s files give away. Only a model whose training data predate 2026-10-13, the day the main run of RoboDx-Hard-100 was generated, is certain not to have seen the set. A new seed keeps the same operator combinations, so new combinations need new sweeps that run no model. Configuration lookup already beats random probing and every budgeted agent on pairs. The existing sweeps hold no further pair outside the 11 configured combinations that meets the selection rule. We have not verified that regenerating RoboDx-Hard-100 reproduces it bit for bit. The stored task files are the reference copy.
An adversary that compares a task’s code with the template sources, and the nearest-neighbour classifier, were not run on RoboDx-Hard-100. Comparing a task’s configuration with the template configuration is a shortcut for configuration faults that we have not tested. The recogniser and configuration lookup give no upper bound for a model trained on the generator. The splits hold out operator combinations only. A model exposed to the tasks or the generator, or a claim about unseen mechanisms, needs new combinations and held-out operators. Scores do not show that a deployed robot is safe. Unattended use would need a completion signal, a patch explanation and review before merging. In the real-set task files we redacted mentions of people and an e-mail address after every reported run. Stored run records and mining caches still hold the original text.
Datasheet, prompts and reproducibility
This section supports § 3 and Section 4.4. It gives the datasheet of RoboDx-Hard-100, the agents’ prompts and tools, and how much repeated runs agree.
Datasheet
The datasheet follows the template of Gebru et al. [40].
Motivation. We built RoboDx-Hard-100 to evaluate how coding agents repair and diagnose faults in robot navigation software. Whether such software is correct depends on how the robot behaves in closed loop, and unit tests alone do not decide it. There was no external funding specific to the dataset.
Composition. RoboDx-Hard-100 has 100 tasks. Of these, 70 use the layered publish/subscribe architecture and 30 the callback architecture. 44 tasks are singles and 56 are pairs, built from 30 distinct operators. Of the singles, 6 need a rating stated in the README and 30 fail no public scenario. The other eight singles are the replacement tasks, which meet neither criterion (Appendix § 14.§ 2). The singles fall into four mechanism families. Spatial, temporal and control faults account for 14, 14 and 14 singles. Communication faults account for 2.
Each task is a directory. Its task.json holds agent-visible metadata and the issue report. Its diagnosis.json holds the catalogue, the probe menu, the budget and the initial observation. The faulty repository with renamed modules and symbols, the simulator, four public scenarios, public tests and the README are in its agent/ folder. A separate verifier/ bundle holds the hidden checks, the reference patch and the ground truth, and it must never be mounted into an agent’s environment. Every agent-visible repository holds a canary file, CANARY.txt, for contamination detection, except the eight tasks of the third generator run. The set holds no data about people and no offensive content.
Collection process. Three runs of the generator built the set, each with its own configuration and seed. The first run gave 94 tasks, the second gave 6 supplementary pairs, and the third gave 8 tasks that replaced 8 tasks of the first run. The three runs drew 209 candidate tasks in all, and Appendix § 9 gives the gates that accepted them. These candidate tasks differ from the 877 candidate pairs of Appendix § 9.§ 2, which come from earlier sweeps that ran no model and from which a script chose the operator sets of the generator configuration. Each run’s manifest records its configuration and seed. No human annotated any task.
Preprocessing and labelling. The task builder derives the labels from simulation and stores them in the verifier bundle. The labels are the fault operators, target modules, interaction classes, symptom classes, exposure conditions and hidden failure fractions. Issue reports are filled into templates from observed symptoms and were checked for leaked fault details.
Uses. RoboDx-Hard-100 is meant for evaluating repair and diagnosis agents and for studying disagreement between checks. The generator can also produce training data, with the limit on comparability stated in § 6. The set must not be used to judge the safety of a deployed robot.
Files and labels. Each task has an agent-visible part, with task.json, diagnosis.json and agent/, and a verifier bundle that is kept apart for evaluators. Each task id names its fault family, and each task.json carries the fault families and difficulty metadata, so the agent-visible files reveal some labels. Issue texts are filled from templates and 97 of the 100 are distinct. No word in an issue text names an injected fault, but within the set the issue text identifies the operator set for 96 tasks. The hash suffix of a task id also identifies the operator set. We keep the message-level trajectories of every reported repair run on RoboDx-Hard-100 and the probe log and answer of every diagnosis attempt.
Personal data. The generated tasks hold no personal data. In the real-set task files, 9 mentions of people in 6 files and an e-mail address with a name in 1 file were redacted. The real set’s run records and mining caches hold text from the upstream repositories, which may include personal data.
Prompts and tools
The system prompts of the repair agent, the diagnosis agent and the control that gives the full observation table are printed below as the agents received them.
# Repair (both tool sets)
You are an expert robotics software engineer debugging a differential-drive robot’s navigation stack.
You are given an issue report and a repository. Only files under the editable paths may be changed; ‘sim/‘
is the read-only hardware/physics simulation and ‘tests/‘, ‘scenarios/‘ are read-only. Your fix must
make the robot behave correctly for ANY robot geometry, start pose, mission frame, sensor/control rate
and network condition within the supported ranges, not just for the public scenarios. Prefer minimal,
root-cause fixes over symptom suppression. Do not loosen limits or disable checks to make tests pass.
When you are confident, call submit_solution.
# Diagnosis (budgeted probing)
You are an expert robotics engineer diagnosing a robot that misbehaves. You cannot read the source code.
You can run diagnostic probes (each has a cost; you have a limited budget) and must then submit a
structured diagnosis choosing the root cause from the hypothesis catalog. Observations are deterministic
simulations rounded to two significant figures: the same fault always produces exactly the same
observation, so any difference between an observation and what a hypothesis would produce, however small,
is real. If several hypotheses remain indistinguishable with the available probes, list all of them as
root_cause (a list) rather than guessing one.
# Diagnosis, full-observation-table control
You are an expert robotics engineer diagnosing a robot that misbehaves. You cannot read the source code.
You are given the COMPLETE set of diagnostic observations (every available probe has already been run)
and a catalog of hypotheses. Infer the root cause. If several hypotheses are genuinely indistinguishable
given the observations, list them all as root_cause; otherwise commit to one.
The two tool sets of the repair agent are described in Appendix D.1. A diagnosis agent can only buy probes with probe and answer with submit_diagnosis. The control that gives the full observation table offers only submit_diagnosis. The forward-model control adds the following text to the user message.
You also have predict(hypothesis, action): a simulator of the hypothesised fault that returns what the probe would show if that hypothesis were true (free). Compare predictions with real probe results to discriminate hypotheses.
The diagnosis user message holds the README as the system description, the initial observation, the catalogue and the probe menu with its costs and budget. Every request carried the model alias, the messages, the tool schemas with tool_choice set to auto, an output-token cap and a user field for attribution. The real-set prompt and the prompts of the direct-prediction study are not printed here.
Run-to-run variance
Some repeats come from earlier sets (Appendix G.1). Haiku’s three repair attempts on compositions-100 give identical verdicts on 88 of 100 tasks with standard tools. With robotics tools the count is 87 of 100. Opus’s three diagnosis attempts on interaction/spec give the same task-correct verdict on 24 of 28 tasks, and their answers agree on fewer tasks.
On RoboDx-HARD-100, attempts by Opus, GPT-5.5 and Haiku were re-run on the 70 main-run tasks whose issue texts were re-rendered. The input of these attempts changed, so their verdict changes bound sampling variance from above. Opus with standard tools went from verified to not verified on 0 of the 70 tasks and the other way on 2. Haiku with robotics tools, the run that changed most, went from verified to not verified on 10 tasks and the other way on 2. Sonnet and GPT 6.1 have no repeated attempts on RoboDx-HARD-100.
References
- [1]Jubril Gbolahan Adigun, Linus Eisele, and Michael Felderer. Metamorphic Testing in Autonomous System Simulations, 2022. URL https://arxiv.org/abs/2209.11031.
- [2]Afsoon Afzal, Claire Le Goues, Michael Hilton, and Christopher S. Timperley. A study on challenges of testing robotic systems. In IEEE 13th International Conference on Software Testing, Validation and Verification (ICST), 2020.DOI
- [3]Afsoon Afzal, Deborah S. Katz, Claire Le Goues, and Christopher S. Timperley. Simulation for robotics test automation: Developer perspectives. In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST), pages 263–274. IEEE, 2021. doi: 10.1109/ICST49551.2021.00036.DOI
- [4]Afsoon Afzal, Claire Le Goues, and Christopher Steven Timperley. Mithra: Anomaly detection as an oracle for cyberphysical systems. IEEE Transactions on Software Engineering, 48(11): 4535–4552, 2022.DOI
- [5]Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar, and Martin Hirzel. Investigating Test Overfitting on SWE-bench, 2025. URL https://arxiv.org/abs/2511.16858.
- [6]Mohammad Nour Al Awad and Sergey Ivanov. Loc2Repair: A framework for evaluating the impact of file-level issue localization in repo-level LLM repair, 2026. URL https://arxiv.org/abs/2606.30963.
- [7]Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. SWE-Bench+: Enhanced coding benchmark for LLMs, 2024. URL https://arxiv.org/abs/2410.06992.
- [8]Jon Ayerdi, Asier Iriarte, Pablo Valle, Ibai Roman, Miren Illarramendi, and Aitor Arrieta. MarMot: Metamorphic Runtime Monitoring of Autonomous Driving Systems, 2024. URL https://arxiv.org/abs/2310.07414. doi:10.1145/3678171.
- [9]Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei Andriushchenko, Maria Trofimova, Daria Litvintseva, and Boris Yangel. SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents, 2025. URL https://arxiv.org/abs/2505.20411.
- [10]Ibragim Badertdinov, Maksim Nekrashevich, Anton Shevtsov, and Alexander Golubev. SWE-rebench V2: Language-agnostic SWE task collection at scale, 2026. URL https://arxiv.org/abs/2602.23866. ICML 2026.
- [11]John W. Bandler and Abdel E. Salama. Fault diagnosis of analog circuits. Proceedings of the IEEE, 73(8), 1985.DOI
- [12]Andrew M. Bean, Ryan Othniel Kearns, Angelika Romanou, Franziska Sofia Hafner, Harry Mayne, Jan Batzner, Negar Foroutan, Chris Schmitz, Karolina Korgul, Hunar Batra, Oishi Deb, Emma Beharry, Cornelius Emde, Thomas Foster, Anna Gausen, María Grandury, Simeng Han, Valentin Hofmann, Lujain Ibrahim, Hazel Kim, Hannah Rose Kirk, Fangru Lin, Gabrielle Kaili-May Liu, Lennart Luettgau, Jabez Magomere, Jonathan Ryström, Anna Sotnikova, Yushi Yang, Yilun Zhao, Adel Bibi, Antoine Bosselut, Ronald Clark, Arman Cohan, Jakob Foerster, Yarin Gal, Scott A. Hale, Inioluwa Deborah Raji, Christopher Summerfield, Philip H. S. Torr, Cozmin Ududec, Luc Rocher, and Adam Mahdi. Measuring what matters: Construct validity in large language model benchmarks, 2025. URL https://arxiv.org/abs/2511.04703. NeurIPS 2025 Datasets and Benchmarks.
- [13]Yuanhong Cai, Xiaohui Nie, Kanglin Yin, Changhua Pei, Yongqian Sun, Shenglin Zhang, Haibin Liu, Guiyang Liu, Xidao Wen, Fang Situ, and Dan Pei. A Multi-Dataset Benchmark for Evaluating LLM Agents in Microservice Failure Diagnosis, 2026. URL https://arxiv.org/abs/2606.29193.
- [14]Paulo Canelas, Bradley Schmerl, Alcides Fonseca, and Christopher S. Timperley. Understanding misconfigurations in ROS: An empirical study and current approaches, 2024. URL https://arxiv.org/abs/2407.19292. ISSTA 2024.
- [15]Miaose Chai, Wang Bill Zhu, Shangshang Wang, Yejia Liu, Song Bian, Honghua Dong, Willie Neiswanger, and Robin Jia. Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?, 2026. URL https://arxiv.org/abs/2604.17338.
- [16]Silin Chen, Zihan Zhang, Xiaodong Gu, Jiale Huang, Yuling Shi, Han Li, Zhao Shan, Chengcheng Wan, Zhensu Sun, and Haibing Guan. SWE-at-Scale: Scalable issue synthesis from historical bug fixes, 2026. URL https://arxiv.org/abs/2610.21359.
- [17]T. Y. Chen, S. C. Cheung, and S. M. Yiu. Metamorphic testing: a new approach for generating next test cases. Technical Report HKUST-CS98-01, Department of Computer Science, Hong Kong University of Science and Technology, 1998.
- [18]Yinfang Chen, Manish Shetty, Gagan Somashekar, Minghua Ma, Yogesh Simmhan, Jonathan Mace, Chetan Bansal, Rujia Wang, and Saravan Rajmohan. AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds, 2025. URL https://arxiv.org/ abs/2501.06706.arxiv.org/abs/2501.06706
- [19]Yuntianyi Chen, Yuqi Huai, Yirui He, Shilong Li, Changnam Hong, Qi Alfred Chen, and Joshua Garcia. A comprehensive study of bug-fix patterns in autonomous driving systems, 2025. URL https://arxiv.org/abs/2502.01937. FSE 2025.
- [20]Zhixiang Chen, Zhuangbin Chen, Xingjie Cai, Wei Li, and Zibin Zheng. An Empirical Study of Interaction Bugs in ROS-based Software, 2025. URL https://arxiv.org/abs/2507. 10235.
- [21]Hongjun Choi, Wen-Chuan Lee, Yousra Aafer, Fan Fei, Zhan Tu, Xiangyu Zhang, Dongyan Xu, and Xinyan Deng. Detecting attacks against robotic vehicles: A control invariant approach. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 801–816, 2018. doi: 10.1145/3243734.3243752.DOI
- [22]Deepro Choudhury, Sinead Williamson, Adam Goliński, Ning Miao, Freddie Bickford Smith, Michael Kirchhof, Yizhe Zhang, and Tom Rainforth. BED-LLM: Intelligent Information Gathering with LLMs and Bayesian Experimental Design, 2025. URL https://arxiv.org/ abs/2508.21184.arxiv.org/abs/2508.21184
- [23]Nils Chur, Thorsten Berger, Einar Broch Johnsen, and Andrzej Wąsowski. Beyond the control equations: An artifact study of implementation quality in robot control software, 2026. URL https://arxiv.org/abs/2602.04799.
- [24]Jackson Clark, Yiming Su, Saad Mohammad Rafid Pial, Yifang Tian, Lily Gniedziejko, Hans- Arno Jacobsen, Yinfang Chen, and Tianyin Xu. SREGym: A live benchmark for AI SRE agents with high-fidelity failure scenarios, 2026. URL https://arxiv.org/abs/2605.07161.
- [25]Luca Console and Pietro Torasso. A spectrum of logical definitions of model-based diagnosis. Computational Intelligence, 7(3), 1991.
- [26]Fan Cui, Hongyuan Hou, Zizhang Luo, Chenyun Yin, and Yun Liang. HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks, 2026. URL https://arxiv.org/abs/2604.14709.
- [27]Johan de Kleer and Brian C. Williams. Diagnosing multiple faults. Artificial Intelligence, 32 (1):97–130, 1987.DOI
- [28]Johan de Kleer and Brian C. Williams. Diagnosis with behavioral modes. In Proceedings of the 11th International Joint Conference on Artificial Intelligence (IJCAI-89), pages 1324–1330, Detroit, MI, 1989. Morgan Kaufmann. URL https://www.ijcai.org/Proceedings/ 89-2/Papers/076.pdf.
- [29]Milan De Koning, Ali Asgari, Pouria Derakhshanfar, and Annibale Panichella. A Metamorphic Testing Approach to Diagnosing Memorization in LLM-Based Program Repair, 2026. URL https://arxiv.org/abs/2604.21579.
- [30]Sagar Deb and Ashwanth Krishnan. STOCKTAKE: Measuring the gap between perception and action in LLM agents with a fair oracle, 2026. URL https://arxiv.org/abs/2607.13618.
- [31]Marcela Gonçalves dos Santos, Sylvain Hallé, and Fábio Petrillo. Mutation Testing for Industrial Robotic Systems, 2025. URL https://arxiv.org/abs/2511.14432.
- [32]Tobias Dürschmid, Christopher S. Timperley, David Garlan, and Claire Le Goues. ROSInfer: Statically inferring behavioral component models for ROS-based robotics systems. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering (ICSE), pages 144:1–144:13, 2024. doi: 10.1145/3597503.3639206.DOI
- [33]Venkata Sai Aswath Duvvuru, Bohan Zhang, Michael Vierhauser, and Ankit Agrawal. LLM-agents driven automated simulation and analysis of small uncrowded aerial systems, 2025. URL https://arxiv.org/abs/2501.11864. ICSE 2025.
- [34]Karim Elmaaroufi, Justin Svegliato, Sarunas Kalade, Graham Schelle, Sanjit A. Seshia, and Matei Zaharia. RHO: Your Coding Agent is Secretly a Roboticist, 2026. URL https://arxiv.org/abs/2606.16458.
- [35]Aoyang Fang, Yifan Yang, Jin’ao Shang, Qisheng Lu, Junjielung Xu, Rui Wang, Songhan Zhang, Yuzhong Zhang, Boxi Yu, and Pinjia He. OpenRCA 2.0: From outcome labels to causal process supervision, 2026. URL https://arxiv.org/abs/2606.27154.
- [36]Tully Foote. tf: The transform library. In IEEE Conference on Technologies for Practical Robot Applications (TePRA), 2013, doi: 10.1109/TePRA.2013.6556373.DOI
- [37]Letian Fu, Justin Yu, Karim El-Refai, Ethan Kou, Haoru Xue, Huang Huang, Wenli Xiao, Guanzhi Wang, Dantong Niu, Fei-Fei Li, Guanyа Shi, Jiajun Wu, Shankar Sastry, Yuke Zhu, Ken Goldberg, and Linxi "Jim" Fan. CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation, 2026. URL https://arxiv.org/abs/2603.22435.
- [38]Kanishk Gandhi, Michael Y. Li, Lyle Goodyear, Agam Bhatia, Louise Li, Aditi Bhaskar, Mohammed Zaman, and Noah D. Goodman. BoxingGym: Benchmarking progress in automated experimental design and model discovery, 2025. URL https://arxiv.org/abs/2501.01540.
- [39]Joshua Garcia, Yang Feng, Junjie Shen, Sumaya Almanee, Yuan Xia, and Qi Alfred Chen. A comprehensive study of autonomous vehicle bugs. In Proceedings of the 42nd International Conference on Software Engineering (ICSE), 2020.DOI
- [40]Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. Datasheets for Datasets, 2021. URL https://arxiv.org/abs/1803.09010. Communications of the ACM 64(12):86–92.
- [41]Daniel Golovin and Andreas Krause. Adaptive Submodularity: Theory and Applications in Active Learning and Stochastic Optimization, 2011. URL https://arxiv.org/abs/1003.3967. Journal of Artificial Intelligence Research 42:427–486.
- [42]Gabriel Grand, Valerio Pepe, Jacob Andreas, and Joshua B. Tenenbaum. Shoot first, ask questions later? building rational agents that explore and act like people, 2025. URL https://arxiv.org/abs/2510.20886. ICLR 2026.
- [43]Michael Hardy, Ruhana Azam, Anka Reuel, Mykel Kochenderfer, and Sanmi Koyejo. Agent evaluation reliability: More tasks won’t (always) fix an agent leaderboard, 2026. URL https://arxiv.org/abs/2610.00651.
- [44]Jordis Emilia Herrmann, Aswath Mandakath Gopinath, Mikael Norrlöf, and Mark Niklas Müller. Diagnosing Robotics Systems Issues with Large Language Models, 2024. URL https://arxiv.org/abs/2410.09084.
- [45]Carl Hildebrandt, Sebastian Elbaum, Nicola Bezzo, and Matthew B. Dwyer. Feasible and stressful trajectory generation for mobile robots. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), pages 349–362, 2020. doi: 10.1145/3395363.3397387.DOI
- [46]Md Abir Hossen, Sonam Kharade, Bradley Schmerl, Javier Cámara, Jason M. O’Kane, Ellen C. Czaplinski, Katherine A. DZurilla, David Garlan, and Pooyan Jamshidi. CaRE: Finding Root Causes of Configuration Issues in Highly-Configurable Robots, 2023. URL https://arxiv.org/abs/2301.07690.
- [47]Zhiyuan Hu, Chumin Liu, Xidong Feng, Yilun Zhao, See-Kiong Ng, Anh Tuan Luu, Junxian He, Pang Wei Koh, and Bryan Hooi. Uncertainty of thoughts: Uncertainty-aware planning enhances information seeking in large language models, 2024. URL https://arxiv.org/abs/2402.03271. NeurIPS 2024.
- [48]Naman Jain, Jaskirat Singh, Manish Shetty, Liang Zheng, Koushik Sen, and Ion Stoica. R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents, 2025. URL https://arxiv.org/abs/2504.07164.
- [49]Saurabh Jha, Rohan Arora, Yuji Watanabe, Takumi Yanagawa, Yinfang Chen, Jackson Clark, Bhavya Bhavya, Mudit Verma, Harshit Kumar, Hirokuni Kitahara, Noah Zheutlin, Saki Takano, Divya Pathak, Felix George, Xinbo Wu, Bekir O. Turkkan, Gerard Vanloo, Michael Nidd, Ting Dai, Oishik Chatterjee, Pranjal Gupta, Suranjana Samanta, Pooja Aggarwal, Rong Lee, PavanKumar Murali, Jae-wook Ahn, Debanjana Kar, Ameet Rahane, Carlos Fonseca, Amit Paradkar, Yu Deng, Pratibha Moogi, Prateeti Mohapatra, Naoki Abe, Chandrasekhar Narayanaswami, Tianyin Xu, Lav R. Varshney, Ruchi Mahindru, Anca Sailer, Laura Shwartz, Daby Sow, Nicholas C. M. Fuller, and Ruchir Puri. TBench: Evaluating AI agents across diverse real-world IT automation tasks, 2025. URL https://arxiv.org/abs/2502.05352. ICML 2025.
- [50]Yuxiang Ji. Spaghetti Architect: A Controlled Code Benchmark Generator, and What Its Own Comments Do to Model Scores, 2026. URL https://arxiv.org/abs/2607.18642.
- [51]Yue Jia and Mark Harman. Higher order mutation testing. Information and Software Technology, 51(10), 2009.DOI
- [52]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, 2024. URL https://arxiv.org/abs/2310.06770. ICLR 2024.
- [53]Chijung Jung, Ali Ahad, Jinho Jung, Sebastian Elbaum, and Yonghwi Kwon. Swarmbug: debugging configuration bugs in swarm robotics. In Proceedings of the 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pages 868–880, 2021. doi: 10.1145/3468264.3468601.DOI
- [54]René Just, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, and Gordon Fraser. Are mutants a valid substitute for real faults in software testing? In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE), 2014.DOI
- [55]Sungmin Kang, Bei Chen, Shin Yoo, and Jian-Guang Lou. Explainable Automated Debugging via Large Language Model-driven Scientific Debugging, 2023. URL https://arxiv.org/abs/2304.02195.
- [56]Sayali Kate, John-Paul Ore, Xiangyu Zhang, Sebastian Elbaum, and Zhaogui Xu. Phys: probabilistic physical unit assignment and inconsistency detection. In Proceedings of the 26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pages 563–573, 2018. doi: 10.1145/3236024.3236035.DOI
- [57]Sayali Kate, Michael Chinn, Hongjun Choi, Xiangyu Zhang, and Sebastian Elbaum. PhysFrame: Type checking physical frames of reference for robotic systems. In Proceedings of the 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pages 45–56, 2021. doi: 10.1145/3468264.3468608.DOI
- [58]Kavindie Katuwandeniya and Samith Rajapaksha Jayasekara Widhanapathirana. ROS Help Desk: GenAI Powered, User-Centric Framework for ROS Error Diagnosis and Debugging, 2025. URL https://arxiv.org/abs/2507.07846.
- [59]Eliahu Khalastchi and Meir Kalech. On fault detection and diagnosis in robotic systems. ACM Computing Surveys, 51(1):9:1–9:24, 2018. doi: 10.1145/3146389.DOI
- [60]Gyuheong Kim, Hyojung Gwon, Jeonghyeon Kim, Kyuhong Shim, and Sunjae Lee. Real-SWE: A compositional evaluation of coding agents under realistic user requests, 2026. URL https://arxiv.org/abs/2608.27831.
- [61]Hyungsub Kim, Muslum Ozgur Ozmen, Antonio Bianchi, Z. Berkay Celik, and Dongyan Xu. PGFUZZ: Policy-guided fuzzing for robotic vehicles. In Network and Distributed System Security Symposium (NDSS), 2021.DOI
- [62]Hyungsub Kim, Muslum Ozgur Ozmen, Z. Berkay Celik, Antonio Bianchi, and Dongyan Xu. PGPatch: Policy-guided logic bug patching for robotic vehicles. In IEEE Symposium on Security and Privacy (S&P), 2022.DOI
- [63]Hyungsub Kim, Muslum Ozgur Ozmen, Z. Berkay Celik, Antonio Bianchi, and Dongyan Xu. PatchVerif: Discovering faulty patches in robotic vehicles. In 32nd USENIX Security Symposium, 2023.
- [64]Seulbae Kim and Taesoo Kim. RoboFuzz: Fuzzing robotic systems over Robot Operating System (ROS) for finding correctness bugs. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), 2022.DOI
- [65]Taegyu Kim, Chung Hwan Kim, Junghwan Rhee, Fan Fei, Zhan Tu, Gregory Walkup, Xiangyu Zhang, Xinyan Deng, and Dongyan Xu. RVFuzzer: Finding input validation bugs in robotic vehicles through control-guided testing. In 28th USENIX Security Symposium (USENIX Security 19), pages 425–442, Santa Clara, CA, 2019. USENIX Association. ISBN 978-1-939133-06-9. URL https://www.usenix.org/conference/usenixsecurity19/presentation/kim.
- [66]Haobin Li, Ping Deng, Weizhong Qian, Liang Jiang, Zhenyu Huang, Mouxing Yang, and Xi Peng. Active-SWE: Benchmarking Coding Agents for Proactive Bug Fixing without Issue Reports, 2026. URL https://arxiv.org/abs/2608.04682.
- [67]Shanchao Liang, Spandan Garg, and Roshanak Zilouchian Moghaddam. The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason, 2025. URL https://arxiv.org/abs/2506.12286.
- [68]Yuanfei Lin, Chenran Li, Mingyu Ding, Masayoshi Tomizuka, Wei Zhan, and Matthias Althoff. DrPlanner: Diagnosis and repair of motion planners for automated vehicles using large language models, 2024. URL https://arxiv.org/abs/2403.07470. IEEE Robotics and Automation Letters.
- [69]Mikael Lindvall, Adam Porter, Gudjon Magnusson, and Christoph Schulze. Metamorphic model-based testing of autonomous systems. In 2017 IEEE/ACM 2nd International Workshop on Metamorphic Testing (MET), pages 35–41, 2017. doi: 10.1109/MET.2017.6.DOI
- [70]You Lu, Dingji Wang, Kun Zhang, Bihuan Chen, Jiyan Zhang, and Xin Peng. Poirot: Automatic root cause analysis of safety violations in ADS simulation testing via hypothetical reasoning. Proceedings of the ACM on Software Engineering, 3(ISSTA):2021–2043, 2026. doi: 10.1145/3832179.DOI
- [71]Nikolai Ludwig, Wasi Uddin Ahmad, Somshubra Majumdar, and Boris Ginsburg. Shortcutting the Fix: Identifying and Categorizing Agentic Exploits in Software Engineering Benchmarks, 2026. URL https://arxiv.org/abs/2609.06780.
- [72]Haitong Ma, Chenxiao Gao, Rushi Qiang, Bo Dai, and Na Li. RLE-Bench: A Qualifying Exam for Coding Agents as Robot Learning Engineers, 2026. URL https://arxiv.org/abs/2609.34210.
- [73]Ming Ma, Jue Zhang, Fangkai Yang, Yu Kang, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems, 2025. URL https://arxiv.org/abs/2512.06749.
- [74]Steve Macenski, Francisco Martín, Ruffin White, and Jonatan Ginés Clavero. The Marathon 2: A navigation system. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020.
- [75]John-Paul Ore, Carrick Detweiler, and Sebastian Elbaum. Phriky-Units: a lightweight, annotation-free physical unit inconsistency detection tool. In Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), 2017.DOI
- [76]Yicheng Ouyang, Jun Yang, and Lingming Zhang. Benchmarking automated program repair: An extensive study on both real-world and artificial bugs. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), 2024.DOI
- [77]Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training Software Engineering Agents and Verifiers with SWE-Gym, 2025. URL https://arxiv.org/abs/2412.21139. ICML 2025.
- [78]Dhaval C. Patel, Kaoutar El Maghraoui, Shuxin Lin, Yusheng Li, Tianjun Feng, Chun-Yi Tsai, Yihan Sun, Wei Alexander Xin, Akshat Bhandari, Tanisha Rathod, Aaron Fan, Sanskriti Vijay Shejwal, Tomas Pasiecznik, Sagar Chethan Kumar, Tanmay Agarwal, Rohith Kanathur, Sam Colman, Amaan Sheikh, Dev Bahl, Ann Li, Krish Veera, Alimurtaza Mustafa Merchant, Shambhawi Baswaraj Bhure, Sajal Kumar Goyla, Chengrui Li, Kirthana Natarajan, Rui Li, Thomas Ajai, Rujing Li, Vivek G. Iyer, Sanjai Vijayakumar, Yitong Bai, Ayal Yakobe, Darief Maes, Yassine Jebbouri, Tianyang Xu, Thai Quoc On, Vera Mazeva, Winston Li, Yuval Shemla, Yeshitha Bhuvanesh, Rushin Bhatt, Siddharth Chethan Gowda, Alisha Vinod, Caroline Cahill, Shriya Aishani Rachakonda, Yunfeng Chen, Aryaman Agrawal, Aman Upganlawar, Mao Le Jonathan Ang, Yubin Sally Go, Madhav Rajkondawar, Yang-Jung Chen, Trisha Maturi, Ananya Kapoor, Andrew Li, Shrey Arora, Mana Abbaszadeh, Shen Li, Charles Xu, and Byeolah Kwon. Beyond static leaderboards: Predictive validity for the evaluation of LLM agents, 2026. URL https://arxiv.org/abs/2606.19704.
- [79]Krishna R. Pattipati and Mark G. Alexandridis. Application of heuristic search and information theory to sequential fault diagnosis. IEEE Transactions on Systems, Man, and Cybernetics, 20 (4):872–887, 1990. doi: 10.1109/21.105086.DOI
- [80]Luan Pham, Hongyu Zhang, Huong Ha, Flora Salim, and Xiuzhen Zhang. RCAEval: A benchmark for root cause analysis of microservice systems with telemetry data, 2024. URL https://arxiv.org/abs/2412.17015.
- [81]Minh V. T. Pham, Huy N. Phan, Hoang N. Phan, Cuong Le Chi, Tien N. Nguyen, and Nghi D. Q. Bui. SWE-Synth: Synthesizing verifiable bug-fix data to enable large language models in resolving real-world bugs, 2025. URL https://arxiv.org/abs/2504.14757. ICSE 2026.
- [82]Zichao Qi, Fan Long, Sara Achour, and Martin Rinard. An analysis of patch plausibility and correctness for generate-and-validate patch generation systems. In Proceedings of the 2015 International Symposium on Software Testing and Analysis (ISSTA), 2015.DOI
- [83]Vijaya Raghavan, Mojdeh Shakeri, and Krishna Pattipati. Test sequencing algorithms with unreliable tests. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 29(4):347–357, 1999. doi: 10.1109/3468.769753.
- [84]Tom Rainforth, Adam Foster, Desi R. Ivanova, and Freddie Bickford Smith. Modern Bayesian experimental design. Statistical Science, 39(1), 2024.arxiv.org/abs/2302.14545
- [85]Daniel Ramos, Claudia Mamede, Kush Jain, Paulo Canelas, Catarina Gamboa, and Claire Le Goues. Are large language models memorizing bug benchmarks?, 2024. URL https://arxiv.org/abs/2411.13323.
- [86]Raymond Reiter. A theory of diagnosis from first principles. Artificial Intelligence, 32(1):57–95, 1987.
- [87]Patrick Rodler. DynamicHS: Streamlining Reiter’s Hitting-Set Tree for Sequential Diagnosis, 2020. URL https://arxiv.org/abs/2012.11078.
- [88]Patrick Rodler, Wolfgang Schmid, and Konstantin Schekotihin. A Generally Applicable, Highly Scalable Measurement Computation and Optimization Approach to Sequential Model-Based Diagnosis, 2017. URL https://arxiv.org/abs/1711.05508.
- [89]Meera Sampath, Raja Sengupta, Stéphane Lafortune, Kasim Sinnamohideen, and Demosthenis Teneketzis. Diagnosability of discrete-event systems. IEEE Transactions on Automatic Control, 40(9):1555–1575, 1995.
- [90]André Santos, Alcino Cunha, and Nuno Macedo. Property-based testing for the Robot Operating System. In Proceedings of the 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation (A-TEST), pages 56–62, 2018. doi: 10.1145/3278186.3278195.
- [91]Zeyu Shen, Haoxiang You, Yilang Liu, Zhicheng Zheng, Lihan Zha, Kashu Yamazaki, Ming-tong Zhang, Suning Huang, Jiankai Sun, Qianzong Chen, Lucy He, Kaiyuan Liu, Haoran Chang, Katerina Fragkiadaki, Dhruv Shah, Mac Schwager, Peter Henderson, Ian Abraham, and Canwen Xu. EmbodiedSWE: Coding Agents for Long Horizon Dexterous Robotics, 2026. URL https://arxiv.org/abs/2609.27308.
- [92]Edward K. Smith, Earl T. Barr, Claire Le Goues, and Yuriy Brun. Is the cure worse than the disease? Overfitting in automated program repair. In Proceedings of the 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE), 2015.
- [93]Atharv Sonwane, Isadora White, Hyunji Lee, Matheus Pereira, Lucas Caccia, Minseon Kim, Zhengyan Shi, Chinmay Singh, Alessandro Sordoni, Marc-Alexandre Côté, and Xingdi Yuan. BugPilot: Complex Bug Generation for Efficient Learning of SWE Skills, 2025. URL https://arxiv.org/abs/2510.19898.
- [94]Thierry Sotiropoulos, Hélène Waeselynck, Jérémie Guiochet, and Félix Ingrand. Can robot navigation bugs be found in simulation? An exploratory study. In 2017 IEEE International Conference on Software Quality, Reliability and Security (QRS), pages 150–159. IEEE, 2017. doi: 10.1109/QRS.2017.25.DOI
- [95]Gerald Steinbauer and Franz Wotawa. Detecting and locating faults in the control software of autonomous mobile robots. In Proceedings of the 19th International Joint Conference on Artificial Intelligence (IJCAI), page 1742, 2005.
- [96]Peter Struss and Oskar Dressler. “Physical negation”: integrating fault models into the general diagnostic engine. In Proceedings of the 11th International Joint Conference on Artificial Intelligence (IJCAI), 1989.
- [97]Yang Sun, Christopher M. Poskitt, Kun Wang, and Jun Sun. FixDrive: Automatically repairing autonomous vehicle driving behaviour for $0.08 per violation, 2025. URL https://arxiv.org/abs/2502.08260. ICSE 2025.
- [98]Balázs Szalontai, Ábel Szauter, Balázs Márton, Péter Verebics, Balázs Pintér, and Tibor Gregorics. Diff-Based Code Corruption using LLMs for Large-Scale Bugfix Benchmarking, 2026. URL https://arxiv.org/abs/2606.29088.
- [99]Hovhannes Tamoyan, Sean Narenthiran, Erik Arakelyan, Mira Mezini, and Boris Ginsburg. SHERLOC: Structured diagnostic localization for code repair agents, 2026. URL https://arxiv.org/abs/2606.24820. EMNLP 2026.
- [100]Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars, 2018. URL https://arxiv.org/abs/1708.08559. ICSE 2018.
- [101]Christopher S. Timperley, Tobias Dürschmid, Bradley Schmerl, David Garlan, and Claire Le Goues. ROSDiscover: statically detecting run-time architecture misconfigurations in robotics systems. In IEEE 19th International Conference on Software Architecture (ICSA), 2022.DOI
- [102]Christopher S. Timperley, Gijs van der Hoorn, André Santos, Harshavardhan Deshpande, and Andrzej Wasowski. ROBUST: 221 Bugs in the Robot Operating System, 2024. URL https://arxiv.org/abs/2404.03629.
- [103]Christopher Steven Timperley, Afsoon Afzal, Deborah S. Katz, Jam Marcos Hernandez, and Claire Le Goues. Crashing simulated planes is cheap: Can simulation detect robotics bugs early? In IEEE International Conference on Software Testing, Verification and Validation (ICST), 2018.DOI
- [104]Alexey Tuzikov. RoboBugBench: Benchmarking automated repair of real robot software under an audited behavioral oracle. arXiv preprint arXiv:2610.16369, 2026. Verify-Agents workshop (“Who Verifies the Agents?”, NeurIPS 2026.arxiv.org/abs/2610.16369
- [105]Pablo Valle, Sergio Segura, Shaukat Ali, and Aitor Arrieta. Metamorphic Testing of Vision-Language Action-Enabled Robots, 2026. URL https://arxiv.org/abs/2602.22579.
- [106]Noah van der Vleuten, Anthony Flores, Shray Mathur, Max Rakitin, Thomas Hopkins, Kevin G. Yager, and Esther H. R. Tsai. EnvTrace: Simulation-Based Semantic Evaluation of LLM Code via Execution Trace Alignment – Demonstrated at Synchrotron Beamlines, 2025. URL https://arxiv.org/abs/2511.09964.
- [107]Dinghua Wang, Shuqing Li, Guanping Xiao, Yepang Liu, and Yulei Sui. An exploratory study of autopilot software bugs in unmanned aerial vehicles. In Proceedings of the 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), 2021.DOI
- [108]Dinghua Wang, Shuqing Li, Guanping Xiao, Yepang Liu, Yulei Sui, Pinjia He, and Michael R. Lyu. An exploratory investigation of log anomalies in unmanned aerial vehicles. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE), 2024. doi: 10.1145/3597503.3639186.DOI
- [109]Yilun Wang, Guangba Yu, Haiyu Huang, Yujie Huang, Zirui Wang, Pengfei Chen, and Michael R. Lyu. Cloud-OpBench: A reproducible benchmark for agentic root cause analysis in cloud systems, 2026. URL https://arxiv.org/abs/2603.00468.
- [110]You Wang, Michael Pradel, and Zhongxin Liu. Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study, 2025. URL https://arxiv.org/abs/2503.15223.
- [111]Junjielong Xu, Qinan Zhang, Zhiqing Zhong, Shilin He, Chaoyun Zhang, Qingwei Lin, Dan Pei, Pinjia He, Dongmei Zhang, and Qi Zhang. OpenRCA: Can large language models locate the root cause of software failures? In International Conference on Learning Representations (ICLR), 2025. URL https://openreview.net/forum?id=M4qNIzQYpd.
- [112]Yibo Yan, Huijuan Wang, Junzhou He, Yizhuo Liang, Shaoyu Wang, Huanchen Sun, and Seo Jin Park. DDBench: A Benchmark for Agentic Debugging on Distributed Systems, 2026. URL https://arxiv.org/abs/2608.14863.
- [113]John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. arXiv preprint arXiv:2405.15793, 2024.arxiv.org/abs/2405.15793
- [114]John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. SWE-smith: Scaling Data for Software Engineering Agents, 2025. URL https://arxiv.org/abs/2504.21798.
- [115]Hossein Yousefizadeh, Shenghui Gu, Lionel C. Briand, and Ali Nasr. Constrained co-evolutionary metamorphic differential testing for autonomous systems with an interpretability approach, 2025. URL https://arxiv.org/abs/2509.16478. ACM TOSEM.
- [116]Boxi Yu, Yuxuan Zhu, Pinjia He, and Daniel Kang. UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench, 2025. URL https://arxiv.org/abs/2506.09289.
- [117]Boxi Yu, Yang Cao, Yuzhong Zhang, Liting Lin, Junjielong Xu, Zhiqing Zhong, Qinghua Xu, Guancheng Wang, Jialun Cao, Shing-Chi Cheung, Pinjia He, and Lionel Briand. SWE-ABS: Adversarial benchmark strengthening exposes inflated success rates on test-based benchmark, 2026. URL https://arxiv.org/abs/2603.00520.
- [118]Xingdi Yuan, Morgane M Moss, Charbel El Feghali, Chinmay Singh, Darya Moldavskaya, Drew MacPhee, Lucas Caccia, Matheus Pereira, Minseon Kim, Alessandro Sordoni, and Marc-Alexandre Côté. debug-gym: A Text-Based Environment for Interactive Debugging, 2025. URL https://arxiv.org/abs/2503.21557.
- [119]Safdar Zaman, Gerald Steinbauer, Johannes Maurer, Peter Lepej, and Suzana Uran. An integrated model-based diagnosis and repair architecture for ROS-based robot systems. In 2013 IEEE International Conference on Robotics and Automation (ICRA), pages 482–489. IEEE, 2013. doi: 10.1109/ICRA.2013.6630618.DOI
- [120]Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, and Liang Xiang. Multi-SWE-bench: A multilingual benchmark for issue resolving, 2025. URL https://arxiv.org/abs/2504.02605. NeurIPS 2025 Datasets and Benchmarks.
- [121]Pujun Zheng, Zixin Shang, Shufan Jiang, Wenhui Tian, Dongsheng Zhu, Zerun Ma, Dingbo Yuan, and Qi Zhang. SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents, 2026. URL https://arxiv.org/abs/2609.08149.
- [122]Ziqian Zhong, Aditi Raghunathan, and Nicholas Carlini. ImpossibleBench: Measuring LLMs’ propensity of exploiting test cases, 2025. URL https://arxiv.org/abs/2510.20270.
- [123]Zhi Quan Zhou and Liqun Sun. Metamorphic testing of driverless cars. Communications of the ACM, 62(3):61–67, 2019.DOI