WorksComputer Science
StepBench: Diagnosing Real-World Procedural Understanding in AI Agents
Abstract
Evaluating an agent's planning ability remains challenging for open-domain tasks, where verifying procedures through execution is often impractical. Existing evaluations typically assess only the final plan and do not distinguish procedural errors such as missing, incorrect, or misordered actions. We introduce STEPBENCH, a benchmark of 6,885 verified procedures crawled from wikiHow across 81 open-domain categories. From each procedure, we derive 5 diagnostic tasks: four with deterministic answers involving withheld, incorrect, or shuffled steps, and one requiring next-step generation. We additionally evaluate full-plan generation to examine how diagnostic performance correlates with end-to-end plan validity. Using the same procedures across tasks lets us directly compare different planning capabilities. For generation, STEPEVAL identifies critical failures that prevent goal completion. Across 14 configurations of 12 models, we find substantial differences across planning capabilities, with step ordering proving the most difficult: the strongest model correctly orders only 63.0% of procedures, dropping to 33.1% for those with nine or more steps. Incorrect-step detection also predicts judged plan validity across models, suggesting a lower-cost alternative for evaluating generation quality. On STEPBENCH-Val, comprising 214 procedures published in 2026, comprehension scores and rankings remain stable, suggesting limited influence from memorisation.
[GitHub] StepBench 🤗 StepBench 🤗 StepEval
Introduction
Large Language Model (LLM)-based agents are on behalf of humans to perform long-horizon tasks, with access to tools and actions that go beyond text generation. These tools and actuators that can link AI systems beyond cyber-digital spaces will emerge, enabling LLM agents to tackle open-domain cyberphysical tasks (tasks that require manipulating both digital and physical environments). AI agents capable of goal-based planning and tool calling ([26, 11]) on physical, open-domain tasks lead to synergistic teaming with humans (who have access to physical environments) at present and, in time, take autonomy over executing and completing these tasks. Having a deep understanding of task decomposition, sequencing and gap identification in open-domain tasks plays a foundational role in creating reliable agents that can succeed in both knowledge-based and (much broader) cyber-physical tasks.
Open-domain task-planning benchmarks primarily focus on generating task decompositions ([26, 11] (Zhang et al. 2021)) or predicting inter-step temporal dependencies ([14] (Zhang et al. 2020a; Yuan et al. 2023a)). Fine-grained error analysis exists in formal ([33, 12, 13]) and embodied ([16]) environments, where an executor supplies ground truth, but is harder to obtain for open-domain written procedures.
Our goal is to diagnose the capabilities underlying open-domain agent planning without executing the proposed steps. For open-domain tasks, generated plans generally cannot be physically executed or formally verified at benchmark scale, while evaluating only the final plan or task outcome provides little insight into which planning capabilities models struggle with. We aim to identify these strengths and weaknesses through targeted diagnostic tasks and examine how they relate to end-to-end plan generation performance.
We introduce STEPBENCH, comprising 5 tasks derived from 6,885 curated wikiHow procedures across 81 categories. STEPBENCH’s 5 diagnostic tasks probe distinct aspects of procedural understanding: last-step selection, missing-step recovery, incorrect-step detection, reordering, and next-step generation. The source articles have documented editorial provenance, with 96.1% co-authored by subject experts, and distractors are retrieved from other procedures to create plausible errors. Using the same procedures across tasks enables direct comparison of different aspects of agentic planning ability. For next-step and full-plan generation, STEPEVAL adopts the critical-failure formulation of Chang et al. [3], judging whether an output contains an error severe enough to prevent goal completion while allowing valid alternative plans.
Across 14 configurations of 12 models, we find that performance varies substantially across capabilities. Models with similar incorrect-step detection scores can differ markedly in reordering. Even when all correct steps are provided, the strongest configuration recovers the reference order for only 63.0% of procedures, falling to 33.1% for procedures with nine or more steps. Reordering accuracy also correlates strongly with judged full-plan validity (), suggesting that diagnostic tasks can provide a lower-cost signal of plan quality.
Our main contributions are as follows:
We introduce STEPBENCH, a planning diagnosis benchmark suite for evaluating plan recovery, error detection, ordering, and generation on 6,885 open-domain procedures. We additionally evaluate on 214 procedures published in 2026 (§ 3–§ 4).
We apply critical-failure evaluation to next-step and full-plan generation through STEPEVAL, an 8B judge distilled from GPT-5 and validated against human annotations (§ 5).
We evaluate 12 models across 14 configurations and find substantial differences across planning abilities. Diagnostic performance also correlates with judged full-plan validity, providing a lower-cost signal of generation quality (§ 7).
Related Work
Task Decomposition for Planning. Language models can decompose high-level goals into executable steps, evaluated against task-management logs and wikiHow (Zhang et al., 2021), represented as task graphs (Yuan et al., 2023a), or incorporated into tool-use pipelines (Shen et al., [26]; Huang et al., [11]). Evaluating only the final plan, however, can obscure distinct failure modes: necessary actions may be missing, incorrect actions introduced, or correct actions placed in the wrong order. Several benchmarks separate such planning abilities when a simulator, formal domain, or annotated workflow provides an oracle (Li et al., [16]; Kokel et al., [12, 13]; Qiao et al., [22]). Embodied Agent Interface, for example, executes model-generated action sequences in household simulators and uses the resulting state to label failures as missing, additional, or misordered steps (Li et al., [16]), while ACPBench uses planning domain definition language (McDermott, [19]) to automatically generate planning questions with verifiable ground-truth answers, covering capabilities such as action applicability and plan validity (Kokel et al., [12]). STEPBENCH instead evaluates open-domain procedures that cannot be checked against a simulator or formal model. Its tasks are derived from the same underlying procedures, enabling direct comparisons between component procedural skills and full-plan generation.
Procedural Reasoning and Plan Verification. Related benchmarks capture individual components of this profile. Zhang et al. (2020a) introduce wikiHow goal–step and pairwise temporal-ordering tasks with semantically similar negatives, while other procedural benchmarks test temporal relations or step ordering [14, 1, 25]. Formal planning benchmarks verify complete plans through execution [34, 33], and reasoning benchmarks localise errors in mathematical solutions or long chains of thought [10] (Zheng et al. 2025). STEPBENCH brings these abilities together for open-domain procedures, relating error localisation and full-order recovery to generation for the same goals. This joint evaluation is relevant to agents that must audit as well as generate plans, consistent with prior findings that language models can struggle to identify errors in their own outputs [32, 27].
Evaluating Generated Plans. Open-ended plan evaluation has typically relied on surface overlap [26, 2] (Zhang et al. 2021), with recent benchmarks increasingly using LLM judges [11, 3]. To evaluate next-step and full-plan generation, we follow the critical-failure formulation of Chang et al. [3] and distil an 8B judge from GPT-5’s judgments, validating it against human annotations § 5. Unlike evaluations focused solely on final generated plans, STEPBENCH probes distinct components of procedural planning through targeted diagnostic tasks. Appendix § 10 provides a detailed benchmark comparison and further discussion of procedural corpora and LLM-judge reliability.
StepBench: Data Curation
STEPBENCH is built from wikiHow procedures that pass a two-stage filtering and curation pipeline (see Figure 1). We crawl 3,637 wikiHow articles across 81 categories. wikiHow suits this purpose because its articles are written as explicitly ordered imperative steps, span diverse open-domain tasks, and carry an inspectable editorial history. 96.1% of articles behind the final benchmark are co-authored by a credentialed subject expert, and a substantial fraction has additionally been fact-checked, expert-approved, or physically tested. Further details are provided in Table 5 and Figure 3 in Appendix § 12. While these features do not prove every procedure correct, they are evidence that the procedures were written for real users and reviewed through an accountable process.
Figure 1. Constructing STEPBENCH. wikiHow methods are crawled into an initial pool, filtered by GPT-4.1 for whether they describe a genuine procedural task, and the selected procedures (each a task query, category, and ordered steps) instantiate five diagnostic tasks: missing-step selection (T1–T2), erroneous-step detection and reordering (T3–T4), and next-step generation (T5). Full-plan generation is additionally evaluated to examine how diagnostic performance correlates with end-to-end plan validity.
Figure 3. STEPBENCH corpus. Left: procedures per category, top 15 of 81; the distribution is long-tailed because how-to content is. Centre: procedures are short — a median of 4 steps — which is what makes exact-match reordering a demanding but not hopeless task. Right: editorial provenance of the source articles.
| Scale | Provenance of source articles | ||
| Procedures | 6,885 | Distinct articles | 2,920 |
| Reference steps | 30,975 | Expert co-authored | 2,806 (96.1%) |
| Categories | 81 | Editorially reviewed | 1,707 (58.5%) |
| Steps per procedure | 4.5 | Fact checked | 990 |
| median (range) | 4 (1–28) | Expert approved | 658 |
| Words per step | 7.27 | Tested | 59 |
| Words per task title | 10.56 | Median readership | 244,916 views |
| Largest / smallest category | 286 / 2 | Cumulative readership | 1.49 B views |
| Median category size | 68 | Median helpfulness | 82% |
Table 5. STEPBENCH at a glance. Left: benchmark scale. Right: provenance of the wikiHow articles the procedures are drawn from. wikiHow attaches an authorship record to every article; the great majority of STEPBENCH’s sources were co-authored with a credentialed subject expert, and a further editorial process fact-checks, expert-approves, or physically tests a large subset. Readership and helpfulness ratings are reader-supplied signals that these are procedures people actually follow.
wikiHow articles often contain multiple task queries and methods, each with its own ordered sequence of steps. We treat each method as an independent procedure, yielding 10,305 candidate procedures from the initial crawl. To ensure procedural quality, we use GPT-4.1 (OpenAI, [20]) to filter and refine candidates in two stages (prompts are included in Appendix § 11.§ 1). (1) Task filtering retains only candidates that describe a well-defined procedural task with a specific objective achieved through an ordered sequence of steps. We exclude, for example, collections of advice or reflective prompts presented in a step-like format. This leaves 6,944 of the original 10,305 candidates. (2) Step curation applies two checks to each remaining procedure: a necessity check removes steps not required to complete the task (e.g., encouragements, asides, and optional variations), while an order check reorders the remaining steps where necessary to form an executable sequence. This yields 6,885 curated procedures, each represented by its category, task query, and ordered reference steps. Only 59 candidates are discarded during step curation. This limited attrition may partly reflect the expert co-authorship and editorial review of the source articles described above. All tasks draw from this shared pool, enabling direct cross-task comparison (§7.2).
Human validation. We assess both task filtering and step curation validity on 200 instances each, with 3 independent annotators per instance. Filtering decisions agree with the human majority on 82.0% of instances, and 81.5% of curated procedures are judged correct by a majority. Sampling protocol, instructions, and inter-annotator agreement are in Appendix § 12.§ 3.
STEPBENCH-Val. The main corpus was crawled by 2025-04-10, before the release of some evaluated models, raising the risk of training-data contamination. To account for this, we construct a separate validation set from wikiHow articles created in 2026, after the release of the 11 evaluated models (§6). We also evaluate GPT-5.6-sol on this set, but do not establish that these articles postdate its training or release. Applying the same two-stage pipeline yields 214 procedures (1,034 steps from 94 articles), spanning 27 of the 81 categories. This set is excluded from the main StepBench benchmark and its headline results. We report the corresponding evaluation results in Appendix E.6.
The StepBench Task Suite
Agents can fail at procedural tasks in different ways: identifying the final step, detecting missing or incorrect steps, or recovering the correct order. These require distinct capabilities that no single task can fully assess. We therefore define five complementary diagnostic tasks over the same 6,885 verified procedures, which together constitute our benchmark. By evaluating these tasks on a shared set of procedures, we can attribute performance differences to the capabilities being tested rather than variations in the underlying data. We further examine how performance on these diagnostic tasks correlates with full plan generation, assessing whether they serve as effective indicators of agents’ procedural capabilities. Figure 4 in Appendix § 12 illustrates the five diagnostic tasks and the procedure generation task using a single example.
Figure 4. The five STEPBENCH diagnostic tasks and full-plan generation on a shared procedure (“perform a glute bridge”). T1–T2 withhold a step for recovery from candidates; T3–T4 introduce incorrect or shuffled steps for detection or reordering; and T5 requires generating the next step. Full-plan generation is shown separately as an end-to-end evaluation setting. Circles indicate each step’s state: given, withheld, incorrect, shuffled, or generated.
Task Definitions
T1: Last-step selection. Given the task query, all steps except the last, and four candidate final steps, the model selects the correct one.
T2: Missing-step recovery. Similar to T1, but the withheld step can occur at any position, with a gap marking its location in the sequence. The model must identify the missing step using both preceding and following context.
T3: Incorrect-step detection. A step at a random position is replaced with a retrieved in-category step (§ 4.2) from another procedure. The model must identify its index without knowing where the replacement occurred, testing its ability to verify a complete, superficially coherent plan against the task goal.
T4: Step reordering. The reference steps are shuffled and renumbered by their shuffled positions, and the model must recover the original ordering.
Since all steps are present, this isolates ordering knowledge from content knowledge.
T5: Next-step generation. Given the task goal and a partial sequence of steps, the model generates the next step without candidate options. We randomly select where to stop the sequence, with at least one preceding and one remaining step, covering early, middle, and late continuations. Unlike T2, the model must generate the step without candidate options.
Full plan generation. Given only the task goal, the model generates the complete ordered procedure. We examine how performance on the diagnostic tasks correlates with full plan generation, assessing the extent to which the capabilities measured by our benchmark are predictive of procedural generation performance.
Distractor Retrieval
T1, T2, and T3 require plausible distractor steps, which we retrieve from other procedures in StepBench. We embed all steps using Qwen3-Embedding-8B ((Zhang et al., 2025)) and rank candidates by cosine similarity. For T3, we replace one reference step with an in-category distractor to prevent detection based on topic similarity alone. For T1 and T2, we pair the correct step with three distractors, one from each of the following strategies:
Step-similar. The most similar step to the withheld step, drawn from another task. These are near-misses involving the same action at a different granularity or with different parameters, making them the hardest distractors and testing procedural detail rather than topic alone.
In-category. The step most similar to the procedure as a whole, drawn from another procedure in the same category. The distractor is topically relevant but belongs to a different task.
Out-of-category. The same retrieval strategy, but restricted to a different category. These are easier to reject and serve as a basic check that the model is engaging with the task.
StepEval: Detecting Critical Failures
Four of the five diagnostic tasks (T1–T4) have known ground-truth answers and are evaluated using accuracy, with T4 additionally reporting normalised edit distance. Evaluating next-step (T5) and full-plan generation is less straightforward: valid procedures may differ substantially from the reference, making text-overlap metrics insufficient, while human evaluation is costly and difficult to scale ([2]). We use STEPEVAL valid-plan rate (VPR; § 5.1) as our primary metric, and additionally report ROUGE (Lin [17]) and baseline-rescaled BERTScore (Zhang et al. (2020b)). Following the critical-failure formulation of Chang et al. [3], STEPEVAL assesses whether a generated procedure contains any failure severe enough to prevent task completion or render the instructions unusable. We also validate the judge’s per-instance labels against human annotations and distil a lower-cost judge for large-scale evaluation.
StepEval Protocol
Following the formulation of Chang et al. [3], we define a critical failure as an omission, extraneous or counterproductive action, contradiction, severe vagueness, or other error that prevents task completion or renders the instructions unusable. Given the task query, reference steps, and generated plan, failures are classified into three families: contradictions with the goal or reference steps; structural faults, including internal inconsistency, incoherence, and severe vagueness; and missing or extraneous actions, such as omitted essential steps, counterproductive additions, or unwarranted repetition. The reference steps help establish essential actions and their dependencies, but differences in wording, granularity, emphasis, or ordering, as well as harmless additions, are accepted if the necessary actions are preserved. The judge returns free-text reasoning and a list of failures, each linked to the relevant reference and candidate steps. Full guidelines and prompts are provided in Appendix § 13.
We derive a binary failure label from the judge’s output and compute the valid-plan rate (VPR) as the proportion of plans with no critical failures:
where is the task goal, the reference procedure, the generated plan, and the judge. Table 1 shows representative critical failure examples.
| Task query and reference steps | Model-generated steps | STEPEVAL |
| Query: [Cakes] How to make cake flour from all-purpose flour? Reference: 1. Measure one level cup of all-purpose flour. 2. Remove two tablespoons of flour. 3. Replace them with two tablespoons of cornstarch. 4. Sift the flour and cornstarch together five times. | 1. Measure one cup of all-purpose flour. 2. Sift into a bowl to remove lumps. 3. Sift again through a fine-mesh sieve. 4. Repeat sifting two or three more times until powdery and smooth. 5. Store the cake flour in an airtight container. | Missing critical action. The plan omits the cornstarch substitution. Repeated sifting leaves the ingredient composition unchanged and does not achieve the conversion described by the reference. |
| Query: [Computers and Electronics] How to blur your background in Zoom using the mobile app? Reference: 1. Open the Zoom app. 2. Join a meeting. 3. Tap the three-dot menu. 4. Tap Virtual Background or Background and Filters. 5. Tap Blur. | 1. Open the Zoom mobile app. 2. Join or start a meeting. 3. Tap More or the three-dot icon. 4. Select Virtual Background. 5. Choose an available background or upload your own. 6. Select the background to apply it. 7. Adjust it through the Background option as needed. | Failure to achieve the stated goal. The plan reaches the relevant menu but instructs the user to apply a replacement background, omitting the essential action of selecting Blur. |
Query: [Computers] How to check your computer uptime on Linux? Reference: 1. Open the terminal. 2. Type uptime -p and press Enter. | 1. Open the terminal. 2. Type uptime and press Enter. 3. Read the displayed system uptime. | No critical failure. Omitting -p changes the output format, but the command still provides the requested uptime information. |
Table 1. Examples of critical-failure evaluation with STEPEVAL on full-plan generation. Each example pairs a task query and reference steps with a plan generated by Llama 3.1 8B Instruct and the judgment of the STEPEVAL. The examples show an omitted essential action, failure to satisfy the task goal, and an acceptable variation. Steps and explanations are abbreviated for readability, and bold text highlights the relevant differences.
Human validation. We validate STEPEVAL on 200 generated plans assessed by three human annotators, who identify critical failures using the task query, reference steps, and generated plan. GPT-5 agrees with the human majority on 85.5% of examples, including 87.1% of has_failure and 83.8% of no_failure cases. Annotation details and inter-annotator agreement are reported in Appendix § 13.§ 5.
Cost-Efficient Planning Judge at Scale
GPT-5 (OpenAI, [21]) achieves 85.5% agreement with human annotations, but its cost makes large-scale evaluation impractical. We therefore distil GPT-5’s judgements into STEPEVAL, a smaller Qwen3-8B judge designed for cost-efficient evaluation. We collect 82K GPT-5 annotations on plans generated by diverse models and fine-tune Qwen3-8B for three epochs. On the human-labelled evaluation set, STEPEVAL achieves 92% agreement with GPT-5 and 82.5% with the human majority, including 80.2% on has_failure and 84.8% on no_failure cases. These results suggest that the distilled judge retains strong agreement across both labels while reducing evaluation costs. Further training details are provided in Appendix § 13.§ 3.
Experimental Setup
Models. We evaluate 12 open-weight and proprietary models spanning an order of magnitude in parameter size. Evaluated models include Qwen3-4B, Qwen3-8B, Qwen3-14B, Qwen3-32B, and Qwen3-4B-Instruct-2507 (Qwen Team, 2025), Llama-3.1-8B-Instruct (Grattafiori et al., [9]), Gemma-3-27B-IT (Gemma Team, 2025), and GLM-4-9B (GLM Team, [7]), as well as proprietary models GPT-5.6 Sol, Gemini 3 Flash (Google DeepMind, [8]), Command A, and its reasoning-tuned variant Command A Reasoning (Cohere, [4]). All 14 configurations have results on all diagnostic tasks and full-plan generation on both datasets. We use the fine-tuned 8B STEPEVAL judge for next-step and full-plan VPR throughout the results. GPT-5 is used only as a reference in judge training and validation. Further model details are provided in Appendix § 16.
Evaluation. We evaluate each configuration on both StepBench and StepBench-Val (§ 3), using identical prompts, decoding settings, parsers, and judges. First four diagnostic tasks (T1–T4) are scored by exact match, with normalised edit distance additionally reported for T4. For next-step (T5) and full-plan generation, we use STEPEVAL valid-plan rate (§ 5.1) as our primary metric, and additionally report ROUGE-1/2/L and baseline-rescaled BERTScore.
Results and Discussion
We examine what StepBench’s five diagnostic tasks reveal about planning capabilities and how they relate to full-plan generation. Table 2 reports results across all 14 configurations. Despite being substantially less costly to evaluate than full plans, with T1–T4 requiring no judge model at all, the diagnostic scores are strongly associated with full-plan validity while also exposing capability differences that full-plan evaluation alone can obscure.
| Model | T1 Acc. | T2 Acc. | T3 Acc. | T4 Acc. | T4 NED | T5 VPR | Full plan VPR |
| Proprietary | |||||||
| GPT-5.6-sol | 75.7 | 73.6 | 66.5 | 63.0 | 80.6 | 79.7 | 92.1 |
| Gemini 3 Flash | 78.8 | 78.2 | 71.1 | 58.7 | 73.4 | 77.0 | 84.5 |
| Gemini 3 Flash | 77.4 | 77.1 | 69.8 | 59.4 | 76.6 | 77.7 | 83.0 |
| Command A Reasoning | 67.9 | 43.9 | 60.5 | 51.8 | 74.1 | 70.0 | 82.0 |
| Command A | 65.5 | 59.1 | 61.2 | 44.3 | 70.0 | 70.7 | 67.4 |
| Open weight | |||||||
| Qwen3 32B think | 65.4 | 43.6 | 58.4 | 49.6 | 72.4 | 73.3 | 75.7 |
| Qwen3 32B | 62.7 | 30.5 | 57.3 | 48.6 | 72.4 | 69.7 | 73.8 |
| Gemma 3 27B | 62.0 | 47.2 | 57.4 | 47.1 | 71.5 | 68.3 | 73.5 |
| Qwen3 14B | 53.9 | 24.9 | 56.9 | 45.3 | 70.3 | 71.5 | 66.2 |
| GLM-4 9B | 54.3 | 28.3 | 53.9 | 38.3 | 65.5 | 65.9 | 60.6 |
| Qwen3 8B | 28.2 | 23.5 | 51.5 | 39.8 | 66.5 | 67.4 | 55.0 |
| Llama 3.1 8B | 21.6 | 26.3 | 48.0 | 16.0 | 45.9 | 50.4 | 53.6 |
| Qwen3 4B Instruct | 42.4 | 28.2 | 49.9 | 35.2 | 62.8 | 68.2 | 57.3 |
| Qwen3 4B | 25.4 | 28.2 | 49.7 | 38.0 | 65.2 | 64.1 | 11.2 |
| Random baseline | 25.0 | 25.0 | – | – | – | – | – |
Table 2. Performance on the five diagnostic tasks and end-to-end full-plan generation (%). T1–T4 report exact-match accuracy. T4 additionally reports normalised edit distance (NED), with higher values indicating closer agreement with the reference order. T5 and full-plan generation are evaluated using STEPEVAL valid-plan rate (VPR). Best scores within each column are bold; second-best scores are <u>underlined</u>.
Planning Beyond Aggregate Scores
Table 2 shows that similar full-plan validity can hide differences in specific planning capabilities. Command A and Qwen3 14B have similar full-plan VPR (67.4% and 66.2%), but differ in missing-step recovery (59.1% and 24.9%). Their reordering accuracy is much closer (44.3% and 45.3%). The diagnostic tasks therefore reveal a difference in recovering missing content that full-plan VPR alone does not show.
Success on one diagnostic task also does not imply success on the others. Qwen3 4B and its Instruct-2507 variant have nearly identical incorrect-step detection accuracy (49.7% and 49.9%), but different full-plan VPR (11.2% and 57.3%). The original model has slightly higher reordering accuracy (38.0% versus 35.2%). These results support evaluating recovery, detection, ordering, and continuation separately. Each task tests a specific capability, while full-plan generation requires producing both the actions and their sequence.
Correlation with Full-Plan Validity
Section 7.1 shows that the diagnostics expose capability differences that full-plan VPR hides. Figure 2 shows how each diagnostic task relates to full-plan VPR across all 14 configurations. All five tasks are positively associated with full-plan validity. Reordering has the strongest correlation (), followed by last-step selection

Figure 2. Correlations between the five diagnostic tasks and end-to-end plan validity. Left: Spearman correlations with full-plan VPR across all 14 configurations, using accuracy for T1–T4 and VPR for T5. Right: reordering accuracy versus full-plan VPR, with each point representing one configuration.
(), incorrect-step detection (), next-step VPR (), and missing-step recovery (). Thus, models that perform well on the diagnostic tasks also tend to generate more valid full plans over the same procedure pool.
These associations suggest that the diagnostic tasks provide a lightweight signal of planning ability without requiring full-plan generation and costly judge evaluation. In particular, T1–T4 have known answers and require no judge model. The correlations should nevertheless be interpreted at the model level: strong performance on a diagnostic task does not imply valid full-plan generation for every procedure on which the model succeeds in that task. Taken together, the diagnostics complement full-plan evaluation by capturing overall planning performance while revealing capability differences that full-plan VPR can obscure.
Procedure Length Effects
Table 2 shows that the best reordering accuracy is 63.0%, even though all required steps are supplied. This is below the best accuracy on selection, recovery, and detection. Figure 5 in Appendix § 15.§ 1 shows how performance changes with procedure length. For GPT-5.6-sol, exact-order accuracy falls from 86.8% at two steps to 33.1% at nine or more steps, while normalised edit distance falls from 86.8 to 76.6. The model often produces an order close to the reference, but recovering the complete sequence becomes harder as the length increases. Exact reference agreement is a strict diagnostic: a different order is not necessarily an invalid plan when some actions can be exchanged.
Figure 5. Scores by procedure length for eight configurations. Left: incorrect-step detection accuracy. Centre: exact-order accuracy. Right: normalised edit distance, where higher scores indicate closer reference agreement. Gemini 3 Flash uses ; Qwen3 32B uses non-thinking mode.
Length affects detection differently. Gemini 3 Flash’s incorrect-step detection accuracy rises from 68.5% at four steps to 80.3% at nine or more, whereas Qwen3 4B falls from 51.1% at three steps to 42.5% at seven or eight. Longer procedures therefore do not make every diagnostic task harder in the same way, and aggregate scores can hide these differences.
Effects of Scale and Reasoning
Table 2 shows that increasing model size changes some capabilities more than others. From Qwen3 4B to Qwen3 32B without thinking, reordering accuracy rises from 38.0% to 48.6%, while full-plan VPR rises from 11.2% to 73.8%. Incorrect-step detection changes less, from 49.7% to 57.3%.
Reasoning comparisons also show uneven changes. Qwen3 32B gains +1.0 points on reordering and +1.9 on full-plan VPR with thinking enabled. Command A Reasoning improves on Command A by +7.4 and +14.6 points, respectively. However, missing-step recovery improves by 13.2 points for Qwen3 32B and falls by 15.2 points for Command A Reasoning. These comparisons show why gains in full-plan generation should not be taken to imply gains in every diagnostic capability. The Command A comparison also involves different model variants, so it does not isolate the effect of reasoning alone.
Performance on StepBench-Val
Table 3 shows performance on STEPBENCH-Val after matching category composition and procedure length. Selection, recovery, and detection change by at most 1.4 points on average, while reordering and next-step VPR decrease by 8.3 and 7.5 points, respectively. Full-plan VPR decreases by 16.2 points. Overall, the diagnostic tasks show varying sensitivity to the dataset change, while remaining comparatively stable.
| T1 | T2 | T3 | T4 | T5 | Full plan | |
| Change | -1.4 | -0.5 | -0.6 | -8.3 | -7.5 | -16.2 |
Table 3. Mean score change on STEPBENCH-Val relative to STEPBENCH (percentage points), after matching categories and reweighting procedure lengths. The five diagnostic tasks use accuracy (T1–T4) or VPR (T5), while full-plan generation uses VPR. Results are averaged over all 14 configurations.
Model rankings are also strongly preserved across datasets, with rank correlations ranging from 0.77 to 0.99. This consistency suggests that the diagnostic patterns observed on STEPBENCH, although it does not by itself rule out memorisation because the sets differ in construction and GPT-5.6-sol’s training chronology is unverified. Appendix § 15.§ 6 provides per-configuration comparisons and additional controls.
Conclusion
We introduce STEPBENCH, a benchmark for diagnosing open-domain procedural planning through five complementary tasks over 6,885 curated procedures. Existing open-domain benchmarks primarily evaluate final generated plans ([26, 11] (Zhang et al. 2021)), while fine-grained error analysis has largely focused on formal or embodied settings with executable plans ([33, 12, 16]). STEPBENCH bridges this gap by probing distinct aspects of procedural understanding, enabling fine-grained diagnosis of planning strengths and weaknesses without requiring execution.
Our evaluation of 12 models across 14 configurations reveals substantial differences in diagnostic profiles. Reordering remains particularly challenging even when all correct steps are provided: the strongest configuration recovers the reference order for only 63.0% of procedures, dropping to 33.1% for procedures with nine or more steps. Models with similar incorrect-step detection scores can also differ markedly in reordering, highlighting differences that aggregate planning scores may obscure.
All five diagnostic tasks are positively correlated with judged end-to-end plan validity, with reordering showing the strongest association (). These findings suggest that diagnostic tasks can provide lower-cost indicators of planning performance while revealing strengths and weaknesses that full-plan evaluation alone cannot capture. Evaluation on STEPBENCH-Val, comprising 214 procedures published in 2026, further supports the robustness of these findings, with relatively stable diagnostic performance and preserved model rankings across older and newer procedures.
Overall, STEPBENCH provides a framework for fine-grained planning diagnosis, connecting distinct procedural capabilities to end-to-end generation performance. We release the corpus, task suite, model outputs, judge annotations, and analysis code to support further research on diagnosing procedural planning in open-domain settings.
Limitations
STEPBENCH focuses on English-language procedures from wikiHow, primarily covering everyday tasks. Broader coverage of languages, sources, and specialised domains would strengthen its generality. The procedures are also relatively short, leaving longer and more complex planning scenarios for future evaluation.
STEPEVAL provides a scalable assessment of critical failures, but its judgements depend on reference quality and the interpretation of acceptable procedural variation. Additional references and expert assessments could improve coverage of alternative valid plans. Computational resources and API credit budgets also limited the number of models and configurations we could evaluate; expanding this coverage would further strengthen the empirical findings.
Acknowledgments
This research is supported by the Engineering and Physical Sciences Research Council [EP/S021566/1] and the EPSRC Fellowship titled “Task Based Information Retrieval” [EP/P024289/1]. This project also used the DGX Spark compute equipment provided by NVIDIA for the grant titled “Agentic Evaluation: Efficient and Reliable Evaluation for Agentic AI Systems”. This research is supported in part by the Cohere Labs Catalyst Grant Program.
Extended Related Work
Benchmark Comparison
Table 4 positions STEPBench against selected benchmarks most closely related to task decomposition or procedural comprehension. It is a map of task coverage, rather than a ranking of resources or a claim that the benchmarks solve identical problems. In particular, the source column records where procedures originate; it is not a measure of reference correctness. The coverage columns distinguish plan production, recovery of a withheld step, localisation of an inserted incorrect step, and ordering. They make explicit that STEPBench combines diagnostic comprehension and generation over the same procedure instances. We report separate task scores because similar overall performance can conceal different procedural strengths and weaknesses. Prior work shows that a few underlying capabilities explain much of the variation in benchmark scores across models [24]. STEPBench provides a more detailed procedural profile by evaluating missing-step recovery, incorrect-step detection, ordering, and generation over the same procedure instances.
| Benchmark | Source | Size | Gen. | Miss. | Err. | Ord. | Decomp. metric |
| Task decomposition | |||||||
| Zhang et al. (2021) | Wunderlist, wikiHow | 8,262 | ✓ | ✗ | ✗ | ◦ | ROUGE, BERTScore |
| TaskLAMA (Yuan et al., 2023a) | search logs, MSCT | 1,612 | ✓ | ✗ | ✗ | ◦ | ROUGE, Hungarian |
| TaskBench [26] | synth. tool graphs | 17,331 | ✓ | ✗ | ✗ | ✗ | ROUGE only |
| UltraTool [11] | expert queries | 5,824 | ✓ | ✗ | ✗ | ✗ | LLM judge (6 dim.) |
| Procedural comprehension | |||||||
| CaT-Bench [14] | recipes | 4,260 | ✗ | ✗ | ✗ | ◦ | accuracy, F1 |
| Anika and Miah [1] | recipes | — | ✗ | ✗ | ✗ | ✓ | Kendall τ, NED |
| proScript [25] | crowd | 6,400 | ✓ | ✗ | ✗ | ◦ | edge F1 |
| How2Bench [3] | open web | 7,000 | ✓ | ✗ | ✗ | ✗ | LLM judge (failures) |
| STEPBENCH (ours) | wikiHow, expert-prov. | 6,885 | ✓ | ✓ | ✓ | ✓ | accuracy, NED, judge |
Table 4. Selected benchmarks for task decomposition or procedural comprehension. Gen.: the model must produce step text. Miss.: the model must identify a step withheld from a given procedure. Err.: the model must localise an incorrect step injected into an otherwise correct procedure. Ord.: the model must recover or judge step order — ◦ marks benchmarks that predict pairwise dependencies over gold steps rather than requiring a full ordering. Among the task-decomposition benchmarks in this table, none evaluates recovery of missing or detection of incorrect steps, and none evaluates comprehension and generation over a shared procedure pool. Sizes are as reported by each paper; STEPBENCH is the count of curated procedures, each of which yields an instance of every task.
Procedural Understanding
Task-decomposition benchmarks generally measure whether a model can generate a suitable collection of steps for a goal. Zhang et al. (2021) use reference-overlap metrics for generated decompositions; TaskLAMA (Yuan et al., 2023a) evaluates generated task graphs and separately probes pairwise temporal dependencies; TaskBench [26] embeds decomposition in a tool-automation pipeline; and UltraTool [11] evaluates natural-language plans before tools are used. PLANET [15] collects multiple planning evaluations under one framework, further illustrating the breadth of behaviours subsumed by a single “planning” label. These benchmarks motivate evaluating planning before execution, but their plan-production scores do not localise a model’s failure mode. Benchmarks that do localise it restrict the setting to where an oracle can check each sub-skill. Embodied Agent Interface (Li et al. [16]) factors embodied decision making into goal interpretation, subgoal decomposition, action sequencing, and transition modelling, using simulators to type errors as missing, additional, or misordered steps; ACPBench (Kokel et al. [12]) poses atomic reasoning tasks over thirteen formal planning domains, with generative variants in ACPBench Hard (Kokel et al. [13]); and WorfBench (Qiao et al. [22]) scores workflows by subsequence and subgraph matching, finding models markedly weaker on graph-structured plans than linear ones. Each depends on a simulator, a formal domain, or an annotated graph. STEPBENCH obtains a comparable per-capability readout from controlled perturbations of verified references instead.
Several works isolate parts of procedural understanding. CaT-Bench (Lal et al. [14]) evaluates whether a step should precede another in recipe plans, and Anika and Miah [1] reconstruct shuffled recipe sequences. Most closely, Zhang et al. (2020a) derive wikiHow tasks for goal-step inference and pairwise temporal ordering, using semantic similarity to obtain challenging negative examples. STEPBENCH inherits this controlled-inference tradition but evaluates gaps and complete orders within procedures, adds plausible inserted faults, and connects all of these diagnostics to generation over the same goals. PARADISE (Uzunoglu et al. [31]) evaluates warnings and tips associated with procedures, Yuan et al. (2023b) distil script knowledge for constrained generation, and proScript (Sakaguchi et al. [25]) models partially ordered scripts. These tasks assess relations among supplied steps, but do not present a complete procedure with a topically plausible violation and ask the model to identify it.
Error localisation of that kind exists where the answer is formally checkable: ProcessBench (Zheng et al. (2025)) asks for the earliest erroneous step in expert-annotated mathematical solutions, and DeltaBench (He et al. [10]) for faulty sections in long reasoning traces. The capability matters because models audit their own output unreliably: self-critique degrades plans in closed domains rather than repairing them ([32, 27]), which argues for measuring detection as a capability of its own rather than assuming generation implies it. STEPBENCH supplies that diagnostic setting through incorrect-step detection and full-order recovery, while its missing-step tasks assess recovery with the surrounding procedural context visible.
Executable planning benchmarks provide another useful point of comparison. Valmeekam et al. [34] and PlanBench (Valmeekam et al., [33]) use formal action models that can determine whether a plan is valid. Their closed action spaces provide an execution oracle, whereas STEPBENCH evaluates open-domain procedures for which executing every candidate plan would require diverse real-world environments. The two settings therefore answer different questions: executable benchmarks assess success under a formal dynamics; STEPBENCH diagnoses how a model represents the content and order of a procedure without executing it.
References and Generation Scoring
Procedural corpora make different trade-offs among scale, domain, and source characteristics. wikiHow has supported summarisation, goal reasoning, and procedure discovery [18] (Koupaee and Wang 2018; Zhang et al. 2020a; Zhou et al. 2022); recipes and instructional videos provide narrower domains [29] (Yamakata et al. 2020); proScript uses crowd-authored scripts [25]; and How2Everything [3] mines web-scale procedures. Because STEPBENCH creates controlled faults from its references, it records editorial signals for every source article and validates its model-assisted curation, rather than treating source provenance as a guarantee of correctness.
For open-ended plan generation, ROUGE and BERTScore remain common points of comparison [26, 2] (Zhang et al. 2021). Judge-based protocols instead assess properties of a plan that need not match a single reference closely (Zheng et al., 2023). The judges themselves are now measured: JudgeBench [28] finds strong judges near chance on response pairs with objectively correct answers, Thakur et al. [30] show high raw agreement coexisting with weak chance-corrected alignment, and Dorner et al. [5] bound how much ground-truth labelling a judge can save once the judged models rival it. UltraTool evaluates six dimensions of a plan with an LLM judge [11]; How2Everything asks whether a plan contains a critical failure that prevents the goal being achieved [3]. STEPEVAL adopts the latter formulation for STEPBENCH’s generation tasks.
Prompts
All prompts are reproduced verbatim. Placeholders in braces are filled at runtime with the corresponding instance fields.
Data Construction Prompts
These two prompts implement the pipeline stages in § 3. All use GPT-4.1.
Stage 1: Task Filtering
[System]
You are an expert in wikiHow task
understanding and evaluation.
[User]
I have a method and its steps from wikiHow.
Please analyse the method title and its
steps and strictly evaluate whether they
constitute a well-defined task. A task is a
specific objective that can be completed by
following a structured sequence of steps.
Steps are the individual actions that, when
executed in order, should lead to the
successful completion of the task.
Specifically, determine if completing the
steps logically leads to accomplishing the
task described in the method title, and if
they can be modelled as a task and steps in
a general.
Here is the method and its steps: {method}.
Please strictly follow the instruction, and
output 'Yes' if it is logically a task and
'No' if it is not. Please do not provide
any extra information or assumptions.
Stage 2: Step Curation
[System]
You are an expert in analysing procedural
tasks.
[User]
Given a task title, the list of steps, and
the description of each step, evaluate them
based on the following criteria:
1. Necessity Check: Determine if each step
is STRICTLY required to accomplish the
task. If any step is unnecessary, REMOVE it.
2. Order Check: If the remaining steps are
not in the correct sequential order,
reorder them logically.
Here is the task title, the list of steps,
and the description of each step: {method}.
Please follow the following instructions
carefully:
- Remove the unnecessary steps from the
initial steps.
- If reordering is needed for the remaining
STRICTLY required steps, return the step
titles arranged logically ordered.
- Respond strictly in JSON format as
follows: {'Step 1': 'Title', 'Step 2':
'Title', ...}
- Please do not provide any details, just
provide the final STRICTLY logically
ordered steps in the provided JSON format.
Task Prompts
These are the prompts given to evaluated models, one per task (§4).
T1: Last-Step Selection
[System]
You are an expert in task understanding and decomposition.
[User]
You are given a task title, the initial steps required to accomplish the task, and a list of candidate options for the final step. The last step is missing, and your goal is to identify the correct final step from the given options. Choose the single correct missing last step and provide your answer strictly in the following JSON format: {"step_index": "Step Index"}.
Task Title: {method_title}
Task Steps:
{steps}
Options:
{options}
T2: Missing-Step Recovery
[System]
You are an expert in task understanding and decomposition.
[User]
You are given a task title, the initial steps required to accomplish the task, and a list of candidate options for the missing step. One of the steps is missing, and your goal is to identify the correct step from the given options. Choose the single correct missing step and provide your answer strictly in the following JSON format: {"step_index": "Step Index"}.
Task Title: {method_title}
Task Steps:
{steps}
Options:
{options}
T3: Incorrect-Step Detection
[System]
You are an expert in task understanding and decomposition.
[User]
You are given a task title and a list of steps to accomplish the task. Your goal is to identify the incorrect step from the given steps. Choose only one step and provide your final answer strictly in the following JSON format: {"step_index": "Step Index"}
Task Title: {method_title}
Task Steps:
{steps}
T4: Step Reordering
[System]
You are an expert in task understanding and decomposition.
[User]
You are given a task title and a list of steps to accomplish the task. Your goal is to determine the correct order of the steps. Return the step indices in the correct order as a list of integers. Provide your final answer strictly in the following JSON format: {"ordered_steps": [<integer indices in correct order>]}
Only return valid JSON. Do not include any explanations or additional text.
Task Title: {method_title}
Task Steps:
{steps}
T5: Next-Step Generation
[System]
You are an expert in task understanding and decomposition.
[User]
You are given a task and the steps completed so far. Your goal is to generate the NEXT step only. Return your answer strictly in the following JSON format:
{"title": "Next concrete action"}
Requirements:
- Generate exactly ONE next step.
- The step must logically follow the given steps.
- The step must be concrete and actionable (not vague).
- Do not repeat previous steps.
- Do not include explanations or additional text.
- Only return valid JSON.
Task Title:
{method_title}
Steps completed so far:
{input_context}
Full Plan Generation
[System]
You are an expert in task understanding and decomposition.
[User]
You are given a task query. Your goal is to decompose the request into a sequence of concrete steps required to accomplish it. Return your final answer strictly in the following JSON format, which is a list of ordered steps:
{"steps": [
{"index": 1, "title": "First concrete action"},
{"index": 2, "title": "Second concrete action"}
]}
Requirements:
- Steps must be logically ordered.
- Steps must be concrete and actionable (not vague).
- The plan must fully solve the USER REQUEST.
- Do not include explanations or additional text.
- Only return valid JSON.
Task Title:
{method_title}
Decoding. All runs use greedy decoding () unless a model’s provider requires otherwise. We add one sampled configuration, Gemini 3 Flash at , to check that our conclusions are not artefacts of greedy decoding. Models supporting extended reasoning are run in both modes where available (Qwen3 32B); Command A Reasoning and Gemini 3 Flash are evaluated in reasoning mode only, as their non-reasoning counterparts are separate models rather than a toggle.
Parsing. Model output is parsed by a per-task extractor that attempts strict JSON first, then a progressively more tolerant sequence: fenced-block stripping, JSON5, Python literal evaluation, and finally a regular-expression fallback for the expected fields. Outputs that survive none of these are recorded as unparsed and scored as failures.
Corpus Details
Datasheet
Motivation. STEPBench was created to evaluate whether language-model agents can understand, verify, and construct real-world task procedures. No existing resource measured comprehension and generation on a shared pool of procedures with a stated reference provenance.
Composition. 6,885 procedures, each a goal plus an ordered list of reference steps, derived from 2,920 wikiHow articles across 81 categories. 30,975 reference steps in total. Instances carry a category label and a stable identifier linking them across all 5 tasks. Text is English.
Collection. Articles were crawled from wikiHow in February 2025. The crawl retained article text, method structure, and the article’s editorial metadata (expert-authorship flag, editorial review tag, co-author count, reference count, view count, reader helpfulness rating). Processing is described in § 3.
Preprocessing. Two GPT-4.1 stages: task filtering and step curation (prompts in Appendix § 11.§ 1). Raw crawl, intermediate outputs, and final pool are all released, so any stage can be inspected or re-run.
Provenance Signals in Detail
wikiHow attaches two independent quality signals to articles. The expert co-author flag indicates the article was written with a named, credentialed subject expert. The editorial tag records a separate review process with three levels: Expert Approved (a subject expert reviewed the finished article), Fact Checked (staff verified factual claims against cited sources), and Tested (staff physically performed the procedure). Table 5 reports the distribution over STEPBench’s source articles. Neither signal is a correctness proof, and we do not use them to filter: every article that passes the pipeline enters the benchmark regardless of its tags. They are reported because they characterise the population the references are drawn from, and because that population differs materially from an unfiltered web crawl.
Human Validation
Stages 1 and 2 make the consequential decisions in our pipeline: Stage 1 determines which candidates qualify as procedural tasks, while Stage 2 determines the reference steps used to construct the benchmark. We therefore evaluate both stages against independent human judgements.
For each stage, we sample 200 instances and collect independent judgements from 3 annotators. Annotators receive the same instruction as the model and answer the corresponding evaluation question without seeing its decision. We construct the two samples differently to reflect what each validation is intended to measure. For Stage 1, we sample methods by including candidates that the Stage 1 filter accepts and rejects to evaluate whether the filtering criterion generalises beyond the procedures retained in the benchmark. For Stage 2, we sample directly from STEPBENCH, since the objective is to assess the quality of the reference procedures used in the final benchmark.
Task Filtering. Annotators judge whether each candidate constitutes a well-defined procedural task. Human annotators agree with one another on 84.5-87.0% of instances, while their majority judgment agrees with the model on 82.0%.
Step Curation. Annotators are shown the original and curated step sequences side by side and judge whether the curation preserves all necessary steps, removes unnecessary ones, and produces a logical ordering. Human annotators agree with one another on 84.5–89.5% of instances and judge 81.5% of the resulting curations to be correct.
StepEval Details
Judge Prompt and Codebook
The judge receives the goal, the reference procedure (labelled L1), and the candidate plan (labelled L2), and returns JSON with a free-text reasoning field and a list of critical_failures, each linked to the reference and candidate steps it concerns.
Failure taxonomy.
Contradiction of the goal — a step directly contradicts a condition stated in the goal.
Contradiction of the reference — a step diverges from a reference step in a way that prevents the goal being achieved.
Internal inconsistency — a step is inconsistent with another step in the same plan.
Incoherence — the plan has very low readability or logical flow; determinable without consulting the reference.
Severe vagueness — the plan omits so much essential detail that it is unusable as instructions.
Missing critical action — an essential reference step is absent with no equivalent or implied action.
Extraneous action — an added step is unnecessary, confusing, or counterproductive.
Redundant repetition — a step repeats an earlier one where the reference has no such repetition.
Acceptable variation. The codebook explicitly instructs the judge not to flag differences of tone, phrasing, or level of detail; differences of emphasis or ordering that do not change the outcome; additional steps that are neutral or practical; or omissions that are reasonably implied by another step. This is what distinguishes the protocol from generic preference judging, and it is why the protocol can credit a plan that is correct by a different route.
Aggregation. A plan is scored no_failure if the returned failure list is empty, has_failure otherwise. The valid-plan rate is the share scored no_failure. Following Chang et al. [3], we take “any failure” rather than “first failure” as the aggregation, which yields higher inter-annotator agreement in their validation.
Judge Configurations
GPT-5 (OpenAI, [21]): Reference judge used to evaluate selected models and provide annotations for distillation; it is not used for the final reported evaluation. Default temperature, with no top_p or top_k settings, in non-thinking mode.
STEPEVAL: Qwen3-8B fine-tuned for three epochs on GPT-5 annotations. This judge provides all final reported evaluations for both T5 and full-plan generation across all configurations and both datasets. Decoding settings: , top_p 0.95, and top_k 20.
Distillation Setup
Training data joins GPT-5 judge outputs back to their source instances using the same prompt-rendering code the judge harness uses at inference, so training and inference formats cannot drift. We fine-tune Qwen3-8B for 3 epochs, effective batch size 16, learning rate , cosine schedule with 3% warmup, bf16, gradient checkpointing, no sequence-length truncation.
StepEval Critical Failure
You are given a goal and two lists of steps, L1 and L2. L1 is one correct procedure that is guaranteed to achieve the goal. L2 is a candidate procedure whose correctness needs to be determined. Your task is to determine whether L2 has any **critical failures**, using the goal and L1 as the reference.
# Important Guidelines
## L1 as Reference
L1 reliably achieves the goal as written, but it may not be the only valid way to do so. Use it as a reliable reference, not the exclusive solution.
## Definition of Critical Failure
A **critical failure** is an issue that fundamentally prevents the goal from being achieved or makes L2 unusable as a set of followable instructions.
Critical failures can take several forms:
### Contradictions
- **Contradiction to the goal:** An L2 step directly contradicts a condition specified in the goal.
- **Contradiction to L1 steps:** An L2 step directly contradicts or significantly diverges from an L1 step, preventing the goal from being achieved.
### Logical or Structural Issues
- **Internal inconsistency:** An L2 step is inconsistent with another step within L2.
- **Incoherence:** L2 has very low readability or logical flow and is hard to follow. This doesn't require reading L1 to determine.
- **Severe vagueness:** As a whole, L2 lacks so much essential detail from L1 that it becomes basically unusable.
### Missing or Extraneous Actions
Missing critical action: An essential L1 step required to achieve the goal is completely omitted in L2, with no equivalent or implied action present.
Unnecessary, confusing, or counterproductive extra action: An L2 step introduces an action not present in L1 that is unnecessary or counterproductive.
Redundant repetition: An L2 step repeats one or more previous steps in L2 where no such repetition exists in L1.
These categories are not exhaustive. In practice, a single critical failure may span multiple categories.
Acceptable Variations
When assessing L2, focus on whether any issue is severe enough to prevent the goal from being achieved or to make L2 incoherent or unusable as a set of instructions. If not, the variation is acceptable.
Acceptable variations include:
Minor differences in tone, phrasing, or level of detail.
Differences in emphasis or ordering that do not affect the outcome.
Additional steps that are neutral or practical.
Reasonable implicit equivalence, where an omitted action is implied by another step.
Ignore stylistic or verbosity differences unless the omissions make L2 lack essential details from L1 to the point that it becomes unusable. In that case, treat it as a critical failure (severe vagueness).
External Knowledge
Base all decisions only on the provided Goal and L1. Minimize reliance on outside knowledge as much as possible.
Examples
Below are examples of what qualifies as a critical failure, as well as examples of what does not. To keep things concise, the L1 and L2 cases are shown in summarized form. Please read through them carefully to understand how to make the distinction.
Keep in mind that these examples are not an exhaustive list of all possible failures for each L2.
Examples of critical failures
Note: The following examples are not listing all failures present in each L2; it's only for demonstration purposes.
Goal: Prepare Indian-style red lentil dhal for 8 portions using an oven and skillet. Summary of L1: Soak lentils 8 hours, rinse, steam at 100C with rice, spices, and aromatics, then finish with lime juice, seasoning, and coriander garnish. Summary of L2: Soak lentils only 30 minutes, then fry onions, garlic, chili, cumin, and salt in ghee, add lentils with water, and simmer until soft.
Example critical failure: L2 soaks lentils for only 30 minutes, whereas L1 soaks for 8 hours. This is a critical difference in time.
Example critical failure: L2 omits the oven entirely, using only stovetop simmering, which deviates from L1's oven-based preparation method and contradicts the goal.
Goal: To construct a traditional wooden Jacob's Ladder toy using wood, ribbon, and small nails. Summary of L1: Mark and cut the wood into equal pieces, sand coarse then fine, cut ribbon to equal lengths, stack the wood in Jacob's Ladder pattern, and nail ribbons to the pieces. Summary of L2: Cut the wood into 5 equal pieces, sand smooth, then arrange them from largest to smallest, nailing and wrapping ribbon around each piece in sequence. Example critical failure: L2 contradicts itself; if the 5 pieces are of equal size, there is no largest or smallest.
Goal: To treat head lice by applying a tea tree oil and apple cider vinegar solution to the hair. Summary of L1: Mix tea tree oil with apple cider vinegar, wash hair, apply solution, cover 15 minutes, rinse, then comb with a fine-tooth comb. Summary of L2: Wash hair with shampoo, apply diluted tea tree oil–vinegar spray under a cap for 1 hour, comb, and repeat treatment over 2 weeks, wash hair with shampoo. Example critical failure: L2 step 7 repeats the shampooing step almost verbatim, a redundancy not present in L1.
Goal: Prepare an alkyl chloride from a primary or secondary alcohol using thionyl chloride to avoid acid and rearrangements. Summary of L1: Place alcohol in a flask, add thionyl chloride, reflux, cool, then separate and dry the alkyl chloride with a drying agent. Summary of L2: Add alcohol and thionyl chloride to a flask, then add the drying agent, attach condenser, reflux, cool, and filter off the drying agent. Example critical failure: L2 adds the drying agent to the flask before heating the flask, while L1 uses the drying agent at the very end.
Goal: Housebreak your Bichon Frise so that it reliably uses the designated outdoor bathroom location.
Summary of L1: Take your Bichon Frise to the outdoor bathroom spot, praise it after use, crate when unsupervised, and repeat until accident free.
Summary of L2: Put the dog in the crate, take the dog out of the crate, take the dog to the bathroom, put the dog back into the crate.
Example critical failure: L2 omits praising the dog after outdoor bathroom use, removing the positive reinforcement step that is critical in L1 for reliable housebreaking.
Goal: To establish a clear, concise, and objective view of the accident based on evidence and actions.
L1:
Establish specific snapshots of the accident based on evidence.
Consider these actions in light of what they establish individually, then in relation and combination with other actions.
Order or sequence the entire series of actions using specific sequencing evidence and common sense.
Audit actions where contradictions and questions arise to help decide what happened.
Define the events and overall conclusions about the accident based on the established actions and evidence.
L2:
Get the basic facts.
Do not make assumptions.
Separate the people from the problem.
Define the problem.
Do not judge.
Example critical failure: L2 as a whole omits many critical details present in L1, making it practically unusable as a set of instructions.
Examples of acceptable variations that do not count as failures
Goal: Prepare Ambrosia Fruit Dip using cream cheese, yogurt, vanilla extract, grated lemon rind, and Equal sweetener.
Summary of L1: Blend cream cheese and yogurt until smooth, add vanilla, lemon rind, and sweetener, mix well, and chill in refrigerator.
Summary of L2: Combine cream cheese, yogurt, vanilla, and sweetener, beat until smooth, add lemon rind, chill, then serve with fruit, enjoy, clean up, and store leftovers.
Acceptable variation: L2 last step (storing leftovers) is not in L1, but it is an extra practical step that is reasonable and does not hurt the process.
Goal: Prepare a package for shipping so that its contents arrive in good condition.
Summary of L1: Choose a strong box, wrap and cushion items, fill empty space, close and tape box, attach label , and remove old labels.
Summary of L2: Place items in box with cushioning, tape securely, attach and verify label, seal seams, mark fragile if needed, and send to shipping service.
Acceptable variation: L2 omits removing old labels, but this is not critical since it is reasonable to assume a new box without old labels.
Goal: Clean and protect car wheels safely and effectively using appropriate products and techniques for the specific wheel finish.
Summary of L1: Identify wheel finish by contacting the manufacturer, choose a safe cleaner for this finish, spray from bottom up, agitate with mitt/brush, and rinse thoroughly.
Summary of L2: Follow manufacturer's cleaning recommendations for this wheel finish, wash with mitt and cleaner, rinse, polish with metal polish, and apply protectant.
Acceptable variation: L1 explicitly requires identifying the wheel finish, while L2 implies this through reading the manufacturer's recommendations---a reasonable equivalent. This is not a critical omission.
Goal: Create distressed terra cotta pots as baby shower favors, each with an herb seed packet in a stamped muslin bag.
Summary of L1: Paint pots with a base coat, dry, add a second coat, dry overnight, sand for a distressed look, add pebbles, tie twine with a thank-you note, stamp "GROW" on muslin bags, insert herb seed packets, and place the bags next to each pot.
Summary of L2: Paint pots in a contrasting color and dry, lightly sand, add pebbles and soil, tie twine with a handwritten thank-you card, stamp and label muslin bags with the herb name, fill with seed packets, tie shut, and place the bag in each pot.
Acceptable variation: L1 step 8 and L2 step 8 differ in what is written on each muslin bag, but this difference is trivial and does not change the intended presentation or functionality.
Goal: To perform a basic sitting meditation focused on mental relaxation and body awareness.
Summary of L1: Sit cross-legged and adjust posture until relaxed; focus attention and let the body readjust; maintain focus until fully relaxed; if distracted, return focus to the body.
Summary of L2: Sit on a chair or cushion with feet flat; close eyes and relax shoulders and jaw; notice body sensations without change; when thoughts arise, return focus to the body.
Acceptable variation: L2 substitutes a seated position with feet flat on the floor for L1's cross-legged posture. Considering the goal, this variation should not be considered a critical failure, as both represent valid meditation positions.
Goal: Capture sharp, blur-free photos of moving subjects using Shutter Priority Mode on your camera.
L1:
1. Set your camera to Shutter Priority Mode.
2. Select an appropriate shutter speed for the action you want to freeze (e.g., 250 for moderate movement, 1000 for fast action).
3. If shooting in low light, increase the ISO setting to a higher value (e.g., 800 or higher) to allow for faster shutter speeds.
4. Use a lens with a wide aperture (low f-number) to let in more light.
5. Position yourself at an appropriate angle or level for the subject (e.g., get low to the ground for children or sports).
6. Take the photo by pressing the shutter button.
L2:
1. Set the camera's mode dial to Shutter Priority.
2. Set the ISO setting to a high value for adequate light sensitivity.
3. Select a wide aperture to capture as much light as possible.
4. Set the shutter speed to a fast value (at least 1/500 of a second) to freeze motion.
5. Use the camera's viewfinder to frame and focus on your subject.
6. Press the shutter button to take the photo.
Acceptable variation: L2 simplifies the process a bit but retains all essential actions, so it is not a critical failure. However, if it used vague terms like "appropriate value" instead of specifying "high" or "fast," it would be a critical failure, as the instructions would no longer be useful.
# Input data
Goal:
{goal}
L1:
{reference_steps}
L2:
{steps}
# Output format
To ensure transparency, provide clear reasoning in the "reasoning" field. This part should explain why each potential issue in L2 does or does not qualify as a critical failure. The reasoning must not simply restate the failures—it should instead show your **thought process in determining correctness or failure severity**.
Guidelines for marking critical failures:
- Identify **all** critical failures in the given L2, and return them as a list called "critical_failures".
- The "failure" field should provide a concise and clear explanation of what the failure is.
- Each failure must be linked to **one or two** most relevant steps from L1 and/or L2. Record these in the "L1_steps" and "L2_steps" fields as lists of step numbers. Only link to more than two steps if there is a good reason to do so.
- If no failures are found, return "critical_failures": [].
Return your response in the following JSON format:
{
"reasoning": "<string>",
"critical_failures": [
{
"failure": "<string>",
"L1_steps": [<int>],
"L2_steps": [<int>]
},
...
]
}
Applying StepEval to T5
T5 asks for one step, not a procedure, so a plan-level judge does not apply to it directly. Rather than introduce a second judge with its own calibration, we reframe each T5 instance as a single-step splice: the reference procedure with only the step at the target index replaced by the model’s generated step, every other position identical. Because reference and candidate differ at exactly one position, any failure the judge reports must trace to that step—including failures that depend on context, such as redundancy with an earlier step or inconsistency with a later one, which a bare step-against-step comparison could not detect. It also avoids the spurious “missing action” flags that comparing a one-step output against a full reference would produce on every instance.
Validation via Human Annotations
To validate the definition of critical failures for StepEval, we ask three human annotators to label 200 examples using our evaluation protocol. We reuse the procedures sampled for step-curation evaluation, pairing each with a model-generated procedure. Annotators inspect the goal, reference procedure, and generated procedure and identify any critical failures. We derive a binary label, ‘has_failure‘ if at least one critical failure is identified, and ‘no_failure‘ otherwise.
Inter-annotator agreement. With binary score aggregation, we observe Krippendorff’s . Given the non-executable, open-world setting, we do not expect near-perfect agreement. To contextualise judge performance, we also measure leave-one-out agreement among annotators, which ranges from 87.2% to 90.5%, excluding examples where the other two annotators disagree.
Evaluating GPT-5 against human labels. We compare the judgements from GPT-5 with the human majority labels on the same 200 examples. GPT-5 achieves 85.5% agreement overall, with 87.1% agreement on human-majority has_failure cases and 83.8% on no_failure cases. We interpret these results alongside human leave-one-out agreement to assess how closely GPT-5 reproduces human judgements under the critical-failure protocol.
Additional Diagnostic Results
This appendix provides further breakdowns for the five diagnostic tasks and their relationship with full-plan generation. We use the same scores as § 7: accuracy for T1–T4, normalised edit distance for partial ordering agreement, and STEP-EVAL VPR for T5 and full-plan generation.
Performance by Procedure Length
Figure 5 shows incorrect-step detection and reordering scores by procedure length for eight configurations. Detection does not follow the same pattern across models. Gemini 3 Flash’s accuracy rises from 68.5% at four steps to 80.3% at nine or more, while Qwen3 4B falls from 51.1% at three steps to 42.5% at seven or eight. These comparisons show different responses to longer procedures; they do not establish whether additional context itself causes the change, since the procedures also differ.
Exact-order accuracy declines with length. GPT-5.6-so1 achieves 86.8% at two steps and 33.1% at nine or more. Its normalised edit distance score changes less, from 86.8 to 76.6. Reporting both metrics therefore distinguishes complete reference-order recovery from partial agreement. Neither score identifies the particular dependency that was violated, and a different order can still be valid when actions can be exchanged. The length breakdown describes difficulty in recovering the reference sequence rather than directly measuring execution success.
Position of the Missing or Incorrect Step
Figure 6 shows accuracy by the relative position of the missing or incorrect step. Neither sequence boundary offers a general advantage. On T3, Gemini 3 Flash scores 64.9% when the incorrect step is first and 77.0% when it is last. Qwen3 4B scores 27.2% and 59.7%, respectively. On T2, the pattern differs: Gemini 3 Flash scores 81.5% for a middle gap, compared with 77.0% for the first position and 74.8% for the last. Qwen3 4B instead performs best at the first position (39.2%).
Figure 6. Accuracy by relative step position for the same eight configurations as Figure 5. Left: the replaced step in T3. Right: the missing step in T2. Position effects vary across tasks and models.
These differences show why a single task score does not fully describe a model’s recovery or detection performance. Position groups also contain different procedures and length distributions, so the results should not be read as evidence of a particular attention mechanism.
Per-Category Results
Figure 7 shows mean scores across all 14 configurations for the twenty largest categories. It includes all five diagnostic tasks alongside full-plan generation. Categories are ordered by procedure count, and shading compares categories within each task. We do not combine accuracy and VPR into a single category-difficulty score.

Figure 7. Mean scores across 14 configurations for the twenty largest categories, ordered by procedure count. Counts in parentheses refer to the source procedure pool; task-specific counts can differ. T1–T4 use accuracy and T5 and full-plan generation use VPR. Shading is normalised within each column.
These breakdowns help identify domains in which a diagnostic task is particularly difficult. However, differences may reflect procedure length, content, or the available models’ familiarity with a domain. They do not by themselves establish that one domain has stricter dependencies or more interchangeable actions than another. The figure shows the largest categories for readability; results for all 81 categories are available in the released per-category analysis.
Sensitivity of Diagnostic-to-Generation Correlations
Table 6 shows how the correlations in § 7.2 change under two exclusions. First, we keep one configuration per underlying model by retaining Gemini 3 Flash at and Qwen3 32B without thinking. Second, we exclude GPT-5.6-sol from the full set to check whether the strongest full-plan result drives the association. All five correlations remain positive and above 0.80 in both checks. This supports the overall association, although the ordering of the diagnostic tasks by correlation changes slightly.
| Configurations | T1 | T2 | T3 | T4 | T5 | |
| All configurations | 14 | 0.95 | 0.85 | 0.93 | 0.96 | 0.89 |
| One per model | 12 | 0.96 | 0.82 | 0.93 | 0.94 | 0.86 |
| Without GPT-5.6-sol | 13 | 0.96 | 0.82 | 0.93 | 0.95 | 0.87 |
Table 6. Spearman correlations between diagnostic scores and full-plan VPR. Each column uses accuracy for T1–T4 or VPR for T5. The two restricted sets are sensitivity checks, not independent test sets.
These are sensitivity checks on the same model sample, not independent validation. Models within a family still share architecture and training, and the results do not establish prediction accuracy for individual plans or for models outside this sample.
Sampling Temperature
Table 2 shows results for Gemini 3 Flash at and . The two configurations differ by at most 1.5 percentage points across T1–T4 accuracy and VPR for T5 and full-plan generation. Full-plan VPR is 84.5% and 83.0%, respectively. Partial-order agreement changes more: normalised edit distance is 73.4 at and 76.6 at . The configuration also has fewer unparseable reordering outputs (57 versus 440), while exact-order accuracy changes from 58.7% to 59.4%. These results show that the two settings produce similar headline scores, but differ in output formatting and partial ordering agreement.
Results and Controls on StepBench-Val
Table 7 shows the change for each configuration after restricting the main set to the 27 shared categories and reweighting its scores to the validation set’s procedure-length distribution. The length groups are two–three, four–five, six–eight, and nine or more steps. These adjustments reduce differences in category coverage and length; they do not make the two sets equivalent in every respect.
| Configuration | T1 | T2 | T3 | T4 | T5 (VPR) | VPR |
| GPT-5.6-sol (T=0, no think) | -0.7 | -4.0 | +0.6 | -11.7 | -8.3 | -15.7 |
| Gemini 3 Flash (think, T=0) | -0.2 | -0.6 | +1.9 | -20.7 | -8.0 | -13.8 |
| Gemini 3 Flash (think, T=1) | -2.9 | -1.2 | -0.3 | -20.1 | -10.7 | -14.2 |
| Command A Reasoning (think) | -1.7 | -2.3 | -3.3 | -10.4 | -12.1 | -16.0 |
| Command A | -1.3 | +1.6 | +0.5 | -7.5 | -7.7 | -13.9 |
| Qwen3 32B (think) | -2.3 | +4.8 | +0.4 | -11.1 | -9.9 | -22.0 |
| Qwen3 32B | -8.8 | +1.7 | +0.1 | -7.2 | -7.9 | -21.7 |
| Gemma 3 27B | +0.0 | -2.2 | -1.1 | -5.8 | -7.8 | -13.0 |
| Qwen3 14B | +0.7 | +1.2 | -0.7 | -3.8 | -6.7 | -22.2 |
| GLM-4 9B | -3.6 | +0.6 | -5.0 | -5.5 | -3.5 | -13.7 |
| Qwen3 8B | -1.1 | -1.6 | +2.3 | -5.8 | -4.0 | -19.0 |
| Llama 3.1 8B | +0.4 | -1.6 | -5.5 | -0.1 | -6.1 | -22.1 |
| Qwen3 4B Instruct (2507) | +0.1 | -1.7 | +3.6 | -4.8 | -6.6 | -17.3 |
| Qwen3 4B | +1.9 | -2.0 | -2.0 | -2.2 | -5.7 | -2.0 |
| Mean | -1.4 | -0.5 | -0.6 | -8.3 | -7.5 | -16.2 |
Table 7. Per-configuration change from STEPBENCH to the 2026 validation set, after matching category composition and reweighting to the 2026 procedure-length distribution. T1–T4 use accuracy; T5 and full-plan generation use StepEval VPR. Values are percentage-point changes. Every configuration has lower scores on reordering, next-step generation, and full-plan generation. Amber marks a decrease of more than three points.
Twelve of 14 configurations change by less than three points on T1. The largest decrease is Qwen3 32B without thinking (-8.8 points), compared with -2.3 for its thinking configuration. In contrast, every configuration has lower scores on T4, T5, and full-plan generation. Thus, the small average changes on selection, recovery, and detection should not be generalised to all five diagnostic tasks.
Remaining differences between the datasets. The validation set contains only 214 procedures and uses a variant of the construction pipeline described in § 3. Its goal statements are also shorter on average (7.1 words versus 10.6). A separate sensitivity check matches shared categories and goal length instead of procedure length. The mean full-plan VPR change is then -19.7 points, compared with -16.2 under the main adjustment. This check is limited by sparse support: the main set has only 73 goals of six words or fewer, compared with 99 in the validation set.
Limits of the temporal comparison. A lower score on newer procedures is not sufficient evidence of memorisation. The model sample, dataset construction, and task difficulty can also affect the differences. GPT-5.6-sol’s training chronology is unverified, and documented cutoffs for other models do not measure their exposure to individual articles. We therefore interpret these results as a check of how diagnostic profiles change across the two datasets, rather than as a test that rules contamination in or out.
Experimental Setup
Table 8 lists the evaluated models, their exact identifiers, and the dates used for temporal comparisons. Our evaluation covers 14 configurations of 12 models, including open-weight and proprietary models. The Qwen3 family spans four parameter sizes, supporting comparisons within a model family, while additional configurations examine reasoning mode and sampling temperature. Where providers disclose a knowledge cutoff, we report it directly; otherwise, we use the model’s release date as a conservative temporal bound, rather than as its training cutoff. GPT-5.6-sol is included in the performance evaluation but excluded from temporal-exclusion claims because its relevant dates are unverified.
| Model | Model ID | Knowledge Cut-off | Date basis |
| Qwen3-4B | Qwen/Qwen3-4B | 2025-04-29 | Release date |
| Qwen3-8B | Qwen/Qwen3-8B | 2025-04-29 | Release date |
| Qwen3-14B | Qwen/Qwen3-14B | 2025-04-29 | Release date |
| Qwen3-32B | Qwen/Qwen3-32B | 2025-04-29 | Release date |
| Qwen3-4B-Instruct-2507 | Qwen/Qwen3-4B-Instruct-2507 | 2025-08-06 | Release date |
| Llama-3.1-8B-Instruct | meta-llama/Llama-3.1-8B-Instruct | 2023-12 | Knowledge cutoff |
| Gemma-3-27B-IT | google/gemma-3-27b-it | 2025-03-12 | Release date |
| GLM-4-9B | zai-org/GLM-4-9B-0414 | 2025-04-15 | Release date |
| GPT-5.6-sol | gpt-5.6-sol | Unverified | Not used for temporal claims |
| Gemini 3 Flash | gemini-3-flash-preview | 2025-01 | Knowledge cutoff |
| Command A | command-a-03-2025 | 2024-06-01 | Knowledge cutoff |
| Command A Reasoning | command-a-reasoning-08-2025 | 2024-06-01 | Knowledge cutoff |
Table 8. Evaluated models, exact identifiers, and knowledge cutoffs. Where the knowledge cutoff is undisclosed and a release date is available, we use that date as a proxy, as indicated in the final column. GPT-5.6-sol is excluded from temporal-exclusion claims because neither date is verified. Month-only dates preserve the precision of the reported cutoff.
References
- [1]Adrita Anika and Md Messal Monem Miah. 2025. Evaluating llms’ reasoning over ordered procedural steps. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, IJCNLP-AACL 2025, Mumbai, India, December 20-24, 2025, pages 2259–2267. The Asian Federation of Natural Language Processing and The Association for Computational Linguistics.arxiv.org/abs/2511.04688
- [2]Faeze Brahman, Chandra Bhagavatula, Valentina Pyatkin, Jena D. Hwang, Xiang Lorraine Li, Hirona J. Arai, Soumya Sanyal, Keisuke Sakaguchi, Xiang Ren, and Yejin Choi. 2024. PlaSma: Making small language models better procedural knowledge models for (counterfactual) planning. arXiv preprint arXiv:2305.19472.arxiv.org/abs/2305.19472
- [3]Yapei Chang, Kyle Lo, Mohit Iyyer, and Luca Soldaini. 2026. How2Everything: Mining the web for how-to procedures to evaluate and improve LLMs. arXiv preprint arXiv:2602.08808.arxiv.org/abs/2602.08808
- [4]Cohere. 2025. Command a: An enterprise-ready large language model. Model card. Accessed 2026-08-17.arxiv.org/abs/2504.00698
- [5]Florian E. Dorner, Vivian Yvonne Nastl, and Moritz Hardt. 2025. Limits to scalable evaluation at the frontier: LLMs as judge won’t beat twice the data. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.arxiv.org/abs/2410.13341
- [7]GLM Team. 2024. ChatGLM: A family of large language models from glm-130b to glm-4—all tools. arXiv preprint arXiv:2406.12793.arxiv.org/abs/2406.12793
- [8]Google DeepMind. 2025. Gemini 3 model card. Model card. Accessed 2026-08-17.
- [9]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.arxiv.org/abs/2407.21783
- [10]Yancheng He, Shilong Li, Jiaheng Liu, Weixin Wang, Xingyuan Bu, Ge Zhang, Zhongyuan Peng, Zhaoxiang Zhang, Zhicheng Zheng, Wenbo Su, and Bo Zheng. 2025. Can large language models detect errors in long chain-of-thought reasoning? In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 18468–18489. Association for Computational Linguistics.arxiv.org/abs/2502.19361
- [11]Shijue Huang, Wanjun Zhong, Jianqiao Lu, Qi Zhu, Jiahui Gao, Weiwen Liu, Yutai Hou, Xingshan Zeng, Yasheng Wang, Lifeng Shang, Xin Jiang, Ruifeng Xu, and Qun Liu. 2024. Planning, creation, usage: Benchmarking LLMs for comprehensive tool utilization in real-world complex scenarios. In Findings of the Association for Computational Linguistics: ACL 2024, pages 4363–4400. Association for Computational Linguistics.arxiv.org/abs/2401.17167
- [12]Harsha Kokel, Michael Katz, Kavitha Srinivas, and Shirin Sohrabi. 2025. Acpbench: Reasoning about action, change, and planning. In Thirty-Ninth AAAI Conference on Artificial Intelligence, Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence, Fifteenth Symposium on Educational Advances in Artificial Intelligence, AAAI 2025, Philadelphia, PA, USA, February 25 - March 4, 2025, pages 26559–26568. AAAI Press.
- [13]Harsha Kokel, Michael Katz, Kavitha Srinivas, and Shirin Sohrabi. 2026. Acpbench hard: Unrestrained reasoning about action, change, and planning. In The Fourteenth International Conference on Learning Representations, ICLR 2026, Rio de Janeiro, Brazil, April 23-27, 2026. proceedings.iclr.cc.arxiv.org/abs/2503.24378
- [14]Yash Kumar Lal, Vanya Cohen, Nathanael Chambers, Niranjan Balasubramanian, and Raymond Mooney. 2024. CaT-bench: Benchmarking language model understanding of causal and temporal dependencies in plans. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 19336–19354. Association for Computational Linguistics.arxiv.org/abs/2406.15823
- [15]Haoming Li, Zhaoliang Chen, Jonathan Zhang, and Fei Liu. 2025. PLANET: A collection of benchmarks for evaluating LLMs’ planning capabilities. arXiv preprint arXiv:2504.14773.arxiv.org/abs/2504.14773
- [16]Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang, Li Fei-Fei, Jiayuan Mao, and Jiajun Wu. 2024. Embodied agent interface: Benchmarking llms for embodied decision making. In Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.arxiv.org/abs/2410.07166
- [17]Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81. Association for Computational Linguistics.
- [18]Qing Lyu, Li Zhang, and Chris Callison-Burch. 2021. Goal-oriented script construction. In Proceedings of the 14th International Conference on Natural Language Generation, pages 184–200. Association for Computational Linguistics.
- [19]Drew V. McDermott. 2000. The 1998 AI planning systems competition. AI Mag., 21(2):35–55.DOI
- [20]OpenAI. 2025. Introducing GPT-4.1 in the API. https://openai.com/index/gpt-4-1/. Accessed 2026-08-17.
- [21]OpenAI. 2026. GPT-5 system card. arXiv preprint arXiv:2601.03267.
- [22]Shuofei Qiao, Runnan Fang, Zhisong Qiu, Xiaobin Wang, Ningyu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. 2025. Benchmarking agentic workflow generation. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.arxiv.org/abs/2410.07869
- [24]Yangjun Ruan, Chris J. Maddison, and Tatsunori B. Hashimoto. 2024. Observational scaling laws and the predictability of language model performance. In Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.
- [25]Keisuke Sakaguchi, Chandra Bhagavatula, Ronan Le Bras, Niket Tandon, Peter Clark, and Yejin Choi. 2021. proscript: Partially ordered scripts generation via pre-trained language models. arXiv preprint arXiv:2104.08251.arxiv.org/abs/2104.08251
- [26]Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Yueting Zhuang. 2024. TaskBench: Benchmarking large language models for task automation. In Advances in Neural Information Processing Systems 37 (NeurIPS Datasets and Benchmarks Track).arxiv.org/abs/2311.18760
- [27]Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. 2025. On the self-verification limitations of large language models on reasoning and planning tasks. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.arxiv.org/abs/2402.08115
- [28]Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Yuan Tang, Alejandro Cuadron, Chenguang Wang, Raluca A. Popa, and Ion Stoica. 2025. Judgebench: A benchmark for evaluating llm-based judges. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.arxiv.org/abs/2410.12784
- [29]Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. 2019. COIN: A large-scale dataset for comprehensive instructional video analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1207–1216.arxiv.org/abs/1903.02874
- [30]Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Diewuke Hupkes. 2024. Judging the judges: Evaluating alignment and vulnerabilities in llms-as-judges. CoRR, abs/2406.12624.arxiv.org/abs/2406.12624
- [31]Arda Uzunoglu, Abdalfatah Rashid Safa, and Gözde Gül Şahin. 2024. PARADISE: Evaluating implicit planning skills of language models with procedural warnings and tips dataset. arXiv preprint arXiv:2403.03167.arxiv.org/abs/2403.03167
- [32]Karthik Valmeekam, Matthew Marquez, and Subbarao Kambhampati. 2023a. Can large language models really improve by self-critiquing their own plans? CoRR, abs/2310.08118.arxiv.org/abs/2310.08118
- [33]Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2023b. PlanBench: An extensible benchmark for evaluating large language models on planning and reasoning about change. In Advances in Neural Information Processing Systems 36 (NeurIPS Datasets and Benchmarks Track).arxiv.org/abs/2206.10498
- [34]Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2022. Large language models still can’t plan (a benchmark for llms on planning and reasoning about change). In NeurIPS 2022 Foundation Models for Decision Making Workshop.




