betaFeedback

WorksComputer Science

WordEval: Evaluating Word-Native Operation Fidelity in Document Editing

Abstract

Microsoft Word is a central environment for office automation, and the faithful execution of its editing operations is an important capability for document agents. Unlike plain-text editing, editing in Word modifies a structured object model, where the requested effect is realized in object properties. Existing benchmarks, however, evaluate office workflows or desktop automation, treating word-processor documents as one task domain among many and verifying outcomes with keyword matching, per-task property checks, or screenshot comparison. We introduce WordEval, an operation-fidelity benchmark that instead centers evaluation on Word editing operations, testing whether each requested edit is realized as intended changes to Word objects and properties. It contains 642 cases across 15 Word feature areas, ranging from text and lists to charts, shape geometry, and SmartArt. 546 atomic cases cover 231 commands, 96 compositional cases combine multiple edits in a single document, and a Word-native oracle checks task-specific object properties and preservation conditions. Cases are constructed from editing-unit specifications, yielding controlled input documents, reference documents, and aligned oracle checks. Each compositional case is linked to corresponding atomic cases, supporting fine-grained diagnosis of editing failures. We evaluate eight agent systems and find that many struggle on the Atomic track, with performance varying across feature areas. Moreover, every system scores lower on Compositional than on Atomic, with gaps that vary substantially by system. Exact atomic-intent links and recorded failure diagnoses provide complementary views of this variation.

https://aka.ms/word-eval

Introduction

AI assistants have begun to operate inside the software where users work, including word processors such as Microsoft Word [1, 7, 17]. In this context, their output is no longer merely a natural-language reply, but a document that the user or agent itself can create, edit, and reuse. A Word document, however, is not merely a text canvas: it contains underlying information that the word processor reads and applies across subsequent document operations. A heading can be used for navigation or table-of-contents generation, a list can maintain continuous numbering as items are added or removed, and a field can be dynamically updated. Therefore, the central question is whether the requested operation takes effect in the document in a native form that Word can recognize and apply.

Preprint.

Overview of WordEval

Figure 1. Overview of WordEval. Atomic tasks test individual Word-editing operations; Compositional tasks combine linked operations across 15 feature areas. The evaluator checks the resulting document’s Word-native objects and properties against the reference document to determine whether the requested edits were realized.

Current evaluations of document agents, however, do not isolate this ability well. Existing benchmarks largely evaluate document generation, file I/O, information extraction, or GUI task completion [3, 22, 24, 25, 27]. They show that an agent can access a document and produce an output file, but not whether the effect of each Word-editing operation was faithfully realized. To address this gap, we introduce WordEval, a benchmark that evaluates how faithfully a Word agent realizes a user’s editing intent as a change in document state. We call this ability operation fidelity: whether the intended effect of an editing operation is realized in the resulting document as a state change with the correct target, attribute, and parameters. WordEval evaluates the output document itself, independent of the action path (e.g., GUI clicks, ribbon commands, or API calls) used to produce it.

WordEval organizes evaluation into Atomic and Compositional tracks. Atomic cases isolate whether the intended effect of a single editing operation is realized. Compositional cases examine whether the agent can manage dependencies, ordering, interference, and preservation when multiple edits are issued together. This design lets us separately analyze the ability to perform individual Word-editing operations and to integrate multiple edits into a coherent document state.

Contributions. (i) A Word-native operation-fidelity benchmark. WordEval comprises 642 cases spanning 15 Word feature areas, each generated from an editing-unit specification with an aligned Word-native oracle, with an Atomic track of 546 cases over 231 Word commands and a Compositional track of 96 cases from 27 composition scenarios.

(ii) Evaluation revealing a compositional gap. Across eight agent systems, many still struggle on the Atomic track, with system-specific Atomic-to-Compositional gaps.

(iii) Fine-grained failure diagnosis. Atomic-intent links, which connect each editing unit of a Compositional case to the Atomic case with the same command, trace compositional failures to individual operations, and a failure-mode analysis characterizes how incorrect outputs differ from the reference.

Related Work

Document editing in office benchmarks. Recent benchmarks for office-workflow agents span desktop and office settings. OfficeBench [24] and OdysseyBench [22] evaluate multi-application office workflows through file-level I/O and conversion utilities, while GUI benchmarks such as OSWorld [25], WindowsAgentArena [3], and WorldGUI [27] evaluate desktop automation with word processors such as Microsoft Word or LibreOffice Writer appearing as one target application. These benchmarks show that agents can reach and manipulate document files, but they do not provide a controlled measurement of document-editing capability organized around the space of editing operations. WordEval targets this missing layer by evaluating structured edits in Microsoft Word under controlled specifications drawn from a command-level taxonomy.

Figure 2. An illustrative example of Word-native operation-fidelity evaluation. In this example, the document agent inserts bullet characters without creating a list in Word, yielding an output visually similar to the reference. WordEval detects this property mismatch.

Verification in office and GUI benchmarks. Existing protocols typically verify returned text, file contents, saved-file state, task completion, or visual appearance: OfficeBench [24] and OdysseyBench [22] use exact-match, containment, keyword, and file-existence checks; OSWorld [25] and WindowsAgentArena [3] apply plain-text matching and per-property checks to saved files; and WorldGUI [27] evaluates screenshot-level GUI task success. These evaluators are effective for checking task completion, but they do not consistently verify the target Word-native object state across commands. For example, a numbered list can be approximated by typing plain-text markers such as 1. and 2., matching the rendered appearance without behaving as a Word list under subsequent renumbering or level changes. WordEval instead evaluates the resolved document state, checking that the specified change holds on the target object and that surrounding content and structure are preserved. Table 1 summarizes this comparison.

BenchmarkInterfaceWord tasksEdit operationsVerification oracle
OfficeBench [24]python-docx API80∗80^{*}4 file actionskeyword, file existence
OdysseyBench [22]python-docx API235∗235^{*}4 file actions [24]keyword, file existence
OSWorld [25]LibreOffice GUI23≈\approx23 property editstext match, property checks
WindowsAgentArena [3]LibreOffice GUI1918 property edits [25]text match, property checks
WorldGUI [27]Word GUI63†63^{\dagger}11 meta-tasksscreenshot task success
WordEval (ours)Unrestricted642231 commandsWord object properties, preservation

Table 1. Comparison with prior benchmarks containing word-processing tasks. Interface is the route through which an agent edits the document. WordEval agents choose their own route, such as python-docx, VBA, or the Word GUI. *Estimated from Word-document subtasks. †Variants of 11 meta-tasks.

WordEval

Problem Formulation

The Word object model. Word represents a document as a structured object graph, which we call the Word object model. In this model, Word exposes runtime objects such as Shape, Style, Font, Paragraph, Field, and Comment, together with properties including Shape.Rotation, Style.NameLocal, and Font.Bold. These runtime objects and their property values are resolved when Word opens and interprets a document.

Operation fidelity. WordEval evaluates Word-native operation fidelity, an agent’s ability to realize a requested edit as the intended changes to objects and properties in the Word object model. We formalize each evaluation case as a tuple (d0,u,d∗)(d_0, u, d^*), where d0d_0 is the input document, uu is a natural-language instruction, and d∗d^* is a reference document representing the expected post-edit state. Given (d0,u)(d_0, u), the agent produces an edited document d^\hat{d}. We evaluate d^\hat{d} using the Word-native oracle described in § 3.4, which verifies the required target changes and preservation properties. This evaluation is independent of the action sequence and specific Word commands used to produce d^\hat{d}.

Benchmark Structure and Coverage

Editing unit as case specification. We represent each edit as an editing unit defined by four components: a precondition that makes the edit executable, a target span or runtime object, an operation, and the parameter values for that operation. The precondition specifies the document state required for the operation to have a determinate, verifiable effect, such as an existing target object, an applicable selection, or a structural context. The operation is represented by a normalized Word command, whereas the target identifies the span or runtime object to which that command applies.

Operation taxonomy. The operation taxonomy organizes the normalized Word commands populating the operation component of each editing unit, associating each command with a generation route and verification schema. Built by consolidating redundant Word ribbon metadata4 into distinct semantic commands (Appendix § 9.1), each entry follows a three-tier hierarchy: behavior family (generation logic), target Word component (affected state), and verification schema (object-property oracle comparisons). This structure enables systematic case generation with aligned oracle specifications (§ 3.4). We exclude commands that are purely UI-structural, service-dependent, lack deterministic and verifiable effects, or fall outside our scope. Among the remaining commands, we instantiate those whose preconditions can be constructed, operations automated, and postconditions verified. This yields 231 released commands; Appendix § 9.1.§ 1 details the coverage funnel and exclusion criteria.

Word feature areas. Whereas the operation taxonomy organizes what edit is performed, Word feature areas organize where that edit acts in the Word object graph. We assign each command to a feature area by its target runtime object. The released commands span 15 Word feature areas, from Text, Paragraphs, and Tables to Charts, SmartArt, and Page Structure (Table 2).

CategoryCasesCoverageDefinition
WordEval642231 commands; 27 scenariosBenchmark across 15 Word feature areas with task-specific object-state checks.
Atomic546231 commandsSingle editing unit over one Word command.
Feature areas54615 feature areas (Text, Paragraphs, Lists, Tables, Shape geometry, Shape appearance, Shape text, Pictures, Graphics, Charts, SmartArt, Equations, Page Structure, Annotations, References)Object-model coverage axis; feature areas group runtime objects, which expose checked object-property pairs.
Compositional9627 scenariosMultiple editing units in the same controlled document substrate.
Additive4611 scenarios: font, paragraph, list, document enrichment/cleanup, chart, layout, picture, shape, SmartArt, tableUnit preconditions are jointly satisfied and edits do not invalidate one another.
Interaction-sensitive5016 scenariosEarlier edits affect later preconditions, target scope, or layout state.
Prerequisite-chain257 scenarios: tables, charts, icons, shapes, SmartArt, equations, text boxesLater unit requires an object or state created by an earlier unit.
Scope-changing124 scenarios: tables, SmartArt, equations, listsEarlier unit changes the target set or structural scope used by later units.
Layout-coupled135 scenarios: grouped shapes, shape rotation/alignment, shape position/alignment, shape multi-step geometry, text wrapping/positioningLater geometry or layout checks depend on earlier wrapping, alignment, or positioning state.

Table 2. Structure and coverage of WordEval, including Atomic and Compositional case counts, Word feature areas, and composition classes.

Evaluation tracks. To distinguish fidelity on individual edits from the additional burden of composition, we organize cases into two tracks, Atomic and Compositional. The Atomic track contains cases with a single editing unit, whereas compositional cases combine multiple units within the same controlled document substrate. Both tracks use the same operation pool and the same Word-native evaluation criterion, allowing individual and combined edits to be compared under one standard. We group compositional cases into two categories by how their editing units interact, additive and interaction-sensitive. In additive cases, unit preconditions are jointly satisfied in d0d_0, no unit invalidates another, and the units are order-independent. In interaction-sensitive cases, earlier edits affect later ones, and three classes specify the affected state, namely preconditions in prerequisite-chain cases, target scope in scope-changing cases, and layout state in layout-coupled cases. The additive category and these three classes form the four composition classes used in our analyses. Appendix 16 and 17 list representative released instructions. For example, an atomic case asks the agent to make the word “Target” bold, while a prerequisite-chain case asks it to insert a clustered column chart below the “Quarterly Sales:” heading and then add outside-end data labels.

Table 2 summarizes the resulting benchmark structure, case counts, and coverage over Word feature areas and composition classes. Appendix § 9.2.§ 1 gives the object-property coverage, and § 4.3 reports capability stratified by feature area.

Case Construction Pipeline

Generation pipeline. Figure 3 shows the end-to-end pipeline. Given a planned editing unit and its schema, an Input Builder constructs a controlled input document d0d_0 in which the unit’s precondition holds, materializing the required document state explicitly rather than sampling it from natural documents so that the input contains only the structure required by the operation under test. A Reference Executor then runs the planned operation on d0d_0 under Word’s native automation, dispatching to the execution route appropriate for the command (one-shot ribbon execution, parameter selection, or dialog automation), producing the reference document d∗d^*, and exporting the oracle specification defined by the schema. An oracle specification lists the checks applied to one case, covering target postconditions for the requested edit and preservation checks for state the edit should leave unchanged. In parallel, an Instruction Renderer produces the natural-language instruction uu from the same unit’s targets, operation, and parameters under a fixed authoring protocol: each unit names a single operation, references a concrete target visible in d0d_0, and avoids transient or invisible state such as selection or cursor focus. Together these stages yield a complete case (d0,u,d∗)(d_0, u, d^*) with its oracle specification attached. The generator also enumerates parameter variants through Word COM automation, so scores are not tied to a single parameter setting; Appendix § 9.1.§ 3 details the variant-generation procedure. The pipeline drives both case construction and oracle evaluation through Word’s native automation, a symmetric COM-API design that makes the target postconditions executable. Properties manipulated during reference construction can be read back during verification, which avoids inferring these postconditions from text, screenshots, or LLM judgments.

Figure 3. Case generation pipeline. A shared case source specifies the editing unit or unit sequence, preconditions, target objects, parameters, and verification schema. The generator uses this source to materialize the input document, instruction, reference document, metadata, and oracle specification.

Compositional case construction. We derive compositional cases from the same document substrates and operations as atomic cases and add dependencies across edits, keeping the operations fixed while varying how they are combined. For atomic cases, the unit’s precondition is satisfied by d0d_0 alone. For compositional cases, the precondition extends across the sequence and is distributed according to the composition class. In additive compositions, d0d_0 is constructed so that all unit preconditions are jointly satisfied and no unit invalidates another’s precondition. In interaction-sensitive compositions, d0d_0 satisfies only the initial precondition, and later preconditions are satisfied only by states produced by earlier units, with the specific carry-over pattern determined by the interaction class (prerequisite-chain, scope-changing, or layout-coupled).

Atomic-intent links. An atomic intent is one editing unit of a compositional case, linked to the Atomic case that applies the same command. In the chart case of § 3.2, inserting the chart and adding outside-end data labels are two atomic intents, linked to the ChartInsert and ChartDataLabel Atomic cases. § 4.4 uses these links to examine compositional cases for which the same system passes every linked Atomic case.

Quality control. Before release, the oracle applies each case’s specification to the reference document d∗d^* itself, and every check must pass. All 642 reference documents pass these checks. Cases that fail this self-check, produce inconsistent metadata, or violate the authoring protocol for uu are regenerated or discarded. This step checks consistency across d0d_0, uu, d∗d^*, and the oracle; released cases are also reviewed by human annotators for instruction clarity, input validity, and reference correctness (Appendix § 9.1.§ 4).

Word-Native Oracle

Each case defines a set of property comparisons, implemented as our oracle. A comparison reads a value from the agent output d^\hat{d} and checks it against a reference value obtained from d∗d^*, d0d_0, or case metadata, and a case passes only if all required comparisons pass. The oracle reads these values through Word’s object model; for some commands, it also checks the saved Office Open XML package to confirm that the requested state persists in the saved file. The oracle verifies task-specific object properties and preservation conditions across 15 Word feature areas. Appendix § 9.2 details coverage, schemas, and human validation. Appendix Table 5 summarizes the oracle’s object–property coverage across Word feature areas; each case’s oracle specification defines the applicable checks. These comparisons cover object attributes, structure counts, inserted content, reference positions, and preservation properties. Each schema pairs its target postcondition with invariants for non-target state that the operation should leave unchanged, such as surrounding text, paragraph or object counts, section structure, table dimensions, or field counts when those properties are relevant. This prevents narrow target-only passes while keeping preservation schema-scoped rather than a global whole-document equivalence rule. For compositional cases, the set combines unit-level checks for individual units with case-level checks for cross-unit dependencies. Additive compositions require no case-level checks, whereas interaction-sensitive compositions include checks specific to the interaction class. Comparisons use behavior-specific extractors and tolerances, including numerical tolerances for position-related properties. When a document opens in Word, each failed check reports the compared property with its reference and output values, such as a footnote number format of lowercase letters in the reference and Arabic numerals in the output. These reports ground the failure-mode analysis in § 4.5.

Experiments and Analysis

Experimental Setup

Evaluated systems. We evaluate eight agent systems: Claude Code [1] with Opus 5 [2]; Codex [17] with GPT-5.6 SO​L [18] (high); Kimi Code [16] with K3 [10] (high); Mini Agent [14] with Mini-Max M3 [15]; Qwen Code [21] with Qwen3-8-27B [20]; OpenCode [19] with Gemini 3.8 Flash [9] (high); ZCode [26] with GLM-5.2 [8] (high); and OpenCode [19] with DeepSeek V4 Pro [4]. Each system is evaluated on 642 cases: 546 Atomic and 96 Compositional. Three human participants perform the same task set.

Metrics. We report two Word Object pass-rate metrics. The case-level pass rate weights every case equally. The macro pass rate is the unweighted mean over atomic commands or composition scenarios, each scored by its variant mean. Each command or composition scenario thus contributes equally; feature-area breakdowns report macro rates unless otherwise stated. Missing outputs, including agent runs that end without a document, and invalid outputs count as non-passing cases (§ 4.5).

Execution environment. Benchmark generation, reference construction, and Word Object verification run on Windows with Microsoft Word through COM automation; these stages require no GPU. The reported run used Windows 11 virtual machines with Word 16.0 and 8GB RAM. Agent-side inference is provided by each agent system and may use external model-serving infrastructure.

Artifact and reproducibility. WordEval is a self-contained release whose evaluator needs no agent system; Appendix § 10.§ 4 details its contents, licenses, data card, and environment requirements.

Main Results

Table 3 reports the performance of eight agent systems on Atomic and Compositional tasks. Claude Code with Opus 5 achieves the highest Overall Case pass rate at 87.1%, while the remaining systems score between 44.5% and 66.7%. The three human participants achieve a mean Overall Case pass rate of 82.8% on the same tasks. Individual participant scores and the human execution protocol are reported in Table 9 and Appendix § 10.

ParticipantAtomicAtomicCompositionalCompositionalOverallOverall
CaseMacroCaseMacroCaseMacro
Human A0183.085.471.969.681.383.7
Human A0290.190.978.176.888.389.4
Human A0382.483.958.358.878.881.3

Table 9. Human execution on the same 642 tasks per participant. Case denominators are 546 Atomic, 96 Compositional and 642 Overall; Macro uses 231, 27 and 258 fixed-variant groups. The main-table Human row averages the three participant-specific rates.

Across the two tracks, all eight systems have lower Compositional than Atomic Case pass rates, but the gap varies from 3.2 to 19.1 percentage points. Opus 5 maintains high performance on both tracks, with pass rates of 87.5% and 84.4%, respectively. However, systems with similar performance on individual edits do not necessarily perform similarly on combined requests. Gemini 3.8 Flash and GLM-5.2 achieve Atomic Case pass rates of 57.9% and 58.2%, respectively, but their Compositional Case pass rates are 49.0% and 39.6%.

These results show that systems with similar aggregate performance on individual edits can differ in their ability to complete combined requests. We next examine performance across Word feature areas (§ 4.3), then compare exactly linked Atomic and Compositional cases to assess whether success on individual operations is maintained when they are combined (§ 4.4).

Atomic Capability Across Word Feature Areas

We analyze Atomic capability across the 15 Word feature areas introduced in § 3.2. Figure 4 shows macro-averaged pass rates under the Word Object oracle for each feature area and system. Tables and Paragraphs achieve the highest mean pass rates, at 83.2% and 81.1%, respectively, with strong performance on cell merging, row-height distribution, paragraph alignment, and indentation.

Figure 4. Atomic macro-averaged pass rates (%) across fifteen Word feature areas for eight agent systems. Each score averages command-level pass rates within the feature area, with missing predictions counted as failures. Columns are ordered by the mean across systems.

In contrast, Pictures has the lowest mean pass rate at 44.9%, followed by SmartArt (50.7%) and Page Structure (53.8%). These areas include image cropping, picture effects such as glow, bevel, and shadow, SmartArt node and formatting edits, and header/footer configuration. These results indicate that image and diagram editing and page-level configuration pose greater challenges than table and paragraph formatting in the evaluated Atomic tasks. We next examine whether success on individual operations carries over to compositional tasks that combine them.

Atomic-to-Compositional Case Analysis

We examine how success on individual Word operations relates to success when those operations are combined in one document. All 96 Compositional cases carry the atomic-intent links defined in § 3.3, covering 319 constituent intents per system. Let CC denote a pass verdict on a Compositional case, and let AA denote the event that the same system passes all linked Atomic variants in isolation. Figure 5 summarises the case pass rate P(C)P(C) with the conditional rate P(C∣A)P(C \mid A). The condition selects a system-specific subset; Appendix Table 11 reports its size for each system and class.

Figure 5. Compositional case pass rates by class for five selected systems. (A) Case pass rates over all cases in each class. (B) Conditional case pass rates restricted to cases for which the same system passes all linked Atomic variants. Rates with fewer than five such cases are omitted. Appendix Table 11 reports the denominators for all eight systems.

Pass rates by composition class. Figure 5A shows case pass rates across the four composition classes for five systems. Scope-changing tasks, in which earlier edits alter the targets of subsequent operations, have the highest pass rates for each system shown (58.3–100%). Layout-coupled tasks, in which edits interact through document layout, show the largest variation across systems (23.1– 84.6%). We examine these class-level differences further by conditioning on successful execution of the constituent Atomic tasks.

Conditioning on atomic-intent success. Figure 5B reports P(C∣A)P(C \mid A), restricting the analysis to cases whose linked Atomic variants all pass in isolation. For additive tasks, conditional pass rates are higher than raw rates, but the difference varies across systems: +41.6 percentage points for K3 and +2.3 points for GPT-5.6. The relationship is less uniform in other classes. On layout-coupled tasks, GPT-5.6 has a conditional pass rate of 66.7%, compared with a raw rate of 76.9%, while GLM-5.2 passes only two of its eight eligible cases (25.0%). Thus, successful execution of all constituent operations in isolation does not ensure completion of the combined request. As in the main results, missing outputs count as failures.

Appendix § 10.3.§ 1 complements this case-level view with intent-level outcomes. For each intent whose Atomic case passes, we check whether the same intent also passes within the combined request, using intents whose embedded verdict is observed. This transfer is lowest in layout-coupled tasks. These results suggest that operations that succeed in isolation are hardest to sustain when edits interact through document layout.

Failure Mode Analysis

Output status. Each non-passing case is assigned to one of three outcomes (Figure 6A). Missing output means that no document was submitted, invalid output means that Word cannot open the document or it fails OOXML validation, and incorrect output means that a valid document fails a requested postcondition or preservation check. For Opus 5, incorrect outputs account for 59 of its 83 non-passing cases, and no output is invalid. For GPT-5.6, invalid and missing outputs together account for 18.4% of all cases, while missing outputs alone account for 41.7% for Gemini 3.8 Flash. These systems often fail before the requested edit can be evaluated.

Figure 6. Failure analysis under Word Object. (A) Output status of non-passing cases over 642 cases per system. (B) Mismatch types of the 59 incorrect outputs of Opus 5 (bold in A), one type per case.

Mismatch types of incorrect outputs. We inspected the evaluator diagnostics and saved OOXML of all 59 incorrect outputs of Opus 5 and assigned each case to one of six types, namely property-value mismatch, structure mismatch, missing element, off-target change, content mismatch, or unrequested change (Figure 6B). Property-value mismatches are the largest type, accounting for 39.0% of the cases. Of the rest, 33.9% are structure mismatches, 11.9% missing elements, 6.8% off-target changes, 6.8% content mismatches, and 1.7% unrequested changes. In property-value mismatches, the requested object is present with a different setting, such as a shadow preset or a footnote numbering format. In picture-effect cases among structure mismatches, Word’s object model reports the requested effect, but the saved OOXML lacks the image layer that Word writes for it. Most missing elements are equation delimiters or matrices inserted with empty placeholders. In the SmartArt text-fill task, Opus 5 fills the shape instead of its text, and in sin⁡x\sin x and ln⁡x\ln x it inserts an invisible function-application character (U+2061) that leaves the rendering unchanged. These outcomes show that most incorrect outputs differ from the reference in settings or Word structure rather than omitting the requested element.

Conclusion

We introduced WordEval, a benchmark that evaluates document-editing agents by the runtime object properties and preserved document state that their edits produce in Word. Its Atomic and Compositional tracks span the execution of individual operations and the coordination of multiple edits within a shared document state. Across eight agent systems, many still struggle on Atomic tasks, performance varies across Word feature areas, and even the strongest system scores lower on Compositional tasks. The atomic-intent and failure-mode analyses further expose operation-level limitations, composition-specific failures, and artifact-level issues. Together, these results suggest that operation fidelity and compositional reliability are distinct requirements for document-editing agents.

Limitations

Synthetic controlled documents. WordEval is a controlled operation-fidelity benchmark, not a natural-document distribution or end-to-end office-workflow benchmark. Its documents isolate Word-native edits under known preconditions, so results should not be read as measuring open-ended writing quality, user productivity, or arbitrary real-world document performance.

Coverage and portability. The current benchmark uses English, protocol-guided instructions; multilingual instructions and user-specific document conventions are outside the current evaluation scope. The evaluator requires Windows and Microsoft Word because WordEval checks Word-resolved runtime state, including lists, fields, layout-dependent properties, and object effects, rather than only the saved .docx file structure.

Coverage-oriented, not frequency-weighted. The released set is a coverage-oriented instantiation of the in-scope verifiable operation space rather than a frequency-weighted sample of real user workloads. Scores should therefore be read as capability-space coverage rather than expected accuracy on natural document-editing workloads.

Broader Impacts

WordEval enables more rigorous evaluation of document-editing agents by exposing failures that surface-text, screenshot, or coarse task-completion metrics miss, helping developers diagnose agents before deployment. It is a capability benchmark rather than a deployment-safety guarantee, and high scores on controlled tasks do not imply safe behavior on out-of-distribution documents, long-horizon workflows, or adversarial inputs.

Acknowledgments

This research was supported by the MSIT (Ministry of Science, ICT), Korea, under the Global Research Support Program in the Digital Field program (RS-2024-0436680) supervised by the IITP (Institute for Information & Communications Technology Planning & Evaluation). This project is supported by Microsoft Research Asia.

Supplementary Material

The supplement follows the benchmark from construction and verification to execution and additional analysis. Task examples and the complete command catalogue are collected at the end for reference.

Benchmark Construction and Coverage

Command Selection and Scope

Table 4 summarizes the command-selection stages used to construct the released Atomic set from Word’s ribbon controls. We start from Word ribbon controls [12] and consolidate command-bearing entries into normalized semantic commands. We then identify commands whose effects are state-changing and verifiable in Word-native state, yielding 1,150 verifiable commands across the three behavior families. Within this verifiable pool, we define the current benchmark scope around document-editing operations; cross-application objects, form design, mail merge, macro/VBA authoring, multi-document workflows, automatic rewriting, and proofing are catalogued separately, yielding 992 in-scope, benchmark-eligible commands. We select the 231 released commands from this pool to cover all three behavior families and all 15 Word feature areas. The remaining in-scope commands define an extension space for future releases.

StageCountReduction
Ribbon control entries5,638
Normalized commands2,569drop 202 entries without a command identifier; deduplicate repeated placements
Verifiable state-changing commands1,150exclude view, settings, clipboard, external, UI-structural, and selection commands (1,419; categories below)
In-scope commands992remove 158 excluded-by-design commands (OLE, forms, mail merge, macro/VBA, master documents, AutoCorrect/AutoFormat, proofing)
Released atomic commands231selected set spanning 15 feature areas; remaining in-scope commands define an extension space

Table 4. Command selection from Word ribbon controls to the released Atomic command set.

Outside the verifiable state-changing pool. These commands either affect state not stored in .docx or lack deterministic Word-native postconditions.

  • View-state commands. Zoom, ruler, navigation pane, gridlines, page color preview.

  • Document-settings commands. Protect Document, Undo/Redo.

  • Clipboard operations. Copy, Cut, Paste, Format Painter.

  • External-service commands. Dictate, Translate, Copilot, Insert from Online, Smart Lookup.

  • UI-structural controls. Menu openers, group labels, tab headers, dialog launchers without a state-changing action.

Outside the in-scope command set.

  • Embedded OLE / ActiveX objects (cross-application state).

  • Form controls (Word as form-design tool, not document editor).

  • Mail merge (multi-document workflow).

  • Macro / VBA authoring (program editing, not document editing).

  • Multi-document workflows (split, master/sub-documents).

  • AutoCorrect / AutoFormat (automatic rewriting outside an explicit command).

  • Proofing (spelling and grammar checking).

Released-Command Selection

Starting from the 992 in-scope commands, we include a command in the released Atomic set when it satisfies three construction requirements: the generator can construct an input document satisfying its precondition, Word automation can execute the command, and the resulting state is verifiable through Word-native postconditions. Execution uses direct invocations for button commands, explicit parameter values for gallery commands, and documented object-model calls for dialog commands. This yields the 231 released commands, spanning all 15 feature areas. Table 18 lists the released commands by feature area, using Office Fluent UI command identifiers (idMso).

Word feature areaReleased commands
TextBold, ChangeCaseGallery, CharacterBorder, CharacterShading, ClearFormatting, Font, FontColor Picker, FontSize, FontSizeDecreaseWord, FontSizeIncreaseWord, Italic, LigatureGalleryWord, ShadingColorPicker, Strikethrough, Subscript, Superscript, SymbolInsertGallery, TextFillGradient Gallery, TextHighlightColorPicker, TextOutlineColorPickerAlternate, TextOutlineDashesGallery, TextOutlineWeightGallery, TextReflectionGallery, UnderlineColorPicker, UnderlineGallery
ParagraphsAlignCenter, AlignJustify, AlignLeft, AlignRight, BordersSelectionGallery, DropCapInsertGallery, IndentDecreaseWord, IndentIncreaseWord, LineSpacingGallery, ParagraphSpacingAfter, ParagraphSpacingBefore, QuickStylesGallery, TableOfContentsAddTextGallery
ListsBulletsGalleryWord, ListLevelGallery, MultilevelListGallery, NumberingGalleryWord
TablesBorderStylesGallery, MergeCells, QuickTablesInsertGallery, SortDialogClassic, SplitCells, Table AutoFitContents, TableAutoFitWindow, TableCellAlignBottomCenter, TableCellAlignBottomLeft, TableCellAlignBottomRight, TableCellAlignMiddleCenter, TableCellAlignMiddleLeft, TableCell AlignMiddleRight, TableCellAlignTopCenter, TableCellAlignTopLeft, TableCellAlignTopRight, TableColumnsDelete, TableColumnsDistribute, TableColumnsInsertLeft, TableColumnsInsert Right, TableDelete, TableInsertGallery, TableRowsDelete, TableRowsDistribute, TableRowsInsert AboveWord, TableRowsInsertBelowWord, TableSplitTable, TableStyleClear, TableStylesGallery Word

Table 18. Released atomic commands by Word feature area.

Parameter Variants

A single case per command would conflate command-level ability with familiarity with one particular parameter value. To ensure that scores reflect general command execution rather than memorization of specific parameter settings, the generator creates variants over the parameter space of each command. For parameterized commands such as galleries, dropdowns, and dialogs, the generator enumerates the available options through Word’s object model and creates cases that span up to eight options per command. Toggle commands are tested in both their on and off states, and other commands typically have one case. The same Reference Executor produces d∗d^* and exports the corresponding oracle specification for each variant, so adding variants requires no manual document authoring or labeling.

Human Review of Benchmark Cases

Three annotators experienced with Microsoft Word independently reviewed every released benchmark case for instruction clarity, input-document validity, and reference-document correctness. Cases flagged during review were manually adjudicated, and affected instructions and reference documents were corrected and revalidated before release.

Word-Native Verification and Validation

The oracle checks the requested edit and the preservation conditions specified for each case. This section details the object–property coverage and verification schemas, then compares evaluator precision, false-positive rates, and false-negative rates.

Object–Property Coverage

Word exposes document objects and their properties through its object model [13]; Office Open XML specifies the saved document format [5]. The following table summarizes the Word objects and properties covered by the oracle, organized by Word feature area. Each case’s oracle specification defines the target postconditions and preservation checks applied to that case.

Feature areaRuntime objectObject properties
TextBorderColor, LineStyle, LineWidth
TextColorFormatRGB
TextFillFormatGradientAngle, Type
TextFontBold, Color, Italic, Ligatures, Name, Size, StrikeThrough, Subscript, Superscript, Underline, UnderlineColor
TextLineFormatDashStyle, Weight
TextRangeHighlightColorIndex, Text
TextReflectionFormatType
TextShadingBackgroundPatternColor, Texture
ParagraphsBorderLineStyle
ParagraphsDropCapLinesToDrop, Position
ParagraphsParagraphOutlineLevel
ParagraphsParagraphFormatAlignment, LeftIndent, LineSpacing, LineSpacingRule, SpaceAfter, SpaceBefore
ParagraphsStyleNameLocal
ListsListFormatListLevelNumber, ListString
TablesBorderLineStyle
TablesCellVerticalAlignment
TablesColumnWidth
TablesColumnsCount

Table 5. Object–property coverage of the Word-native oracle, organized by Word feature area.

Feature areaRuntime objectObject properties
ParagraphAlignment
RangeText
RowHeight
RowsCount
StyleNameLocal
TableAutoFitBehavior
TablesCount
Shape geometryGroupShapesCount
ShapeAutoShapeType, Height, HorizontalFlip, Left, RelativeHorizontalPosition, RelativeVerticalPosition, Rotation, Top, Type, VerticalFlip, Width, ZOrderPosition
ShapesCount
WrapFormatType
Shape appearanceColorFormatRGB
FillFormatGradientStops, Type
GlowFormatRadius
ReflectionFormatType
ShadowFormatType
ShapeShapeStyle
SoftEdgeFormatType
ThreeDFormatBevelTopDepth, BevelTopType, PresetCamera, RotationX, RotationY
Shape textParagraphAlignment
RangeOrientation, Text
TextFrameOrientation
TextFrame2Orientation
PicturesColorFormatRGB
GlowFormatRadius
InlineShapeHeight, Type, Width
InlineShapesCount
PictureFormatBrightness, ColorType, Contrast, CropBottom, CropLeft, CropRight, CropTop
ReflectionFormatType
ShadowFormatType
SoftEdgeFormatType
ThreeDFormatBevelTopDepth, BevelTopType, PresetCamera, RotationX, RotationY
GraphicsColorFormatRGB
InlineShapeHeight, Width
InlineShapesCount
ChartsAxisMajorGridlines, MinorGridlines
ChartChartColor, ChartGroups, ChartStyle, ChartType, HasDataTable, HasLegend, HasTitle
ChartTitleText
ColorFormatRGB
DataLabelsPosition
DataTableShowLegendKey
LegendPosition
SeriesFormat, HasDataLabels, Name, Values
SeriesCollectionCount
TrendlineType
SmartArtColorFormatRGB
LineFormatBeginArrowheadStyle, DashStyle, EndArrowheadStyle, Weight
ShapeHeight, Width
SmartArtReverse
SmartArtColorName
SmartArtLayoutName
SmartArtNodeLevel
SmartArtNodesCount
SmartArtQuickStyleName
TextFrame2WarpFormat
TextRange2Text
ThreeDFormatBevelTopType, PresetCamera, RotationX
EquationsOMathsCount

Table 1 (PDF p. 16).

Feature areaRuntime objectObject properties
RangeText, WordOpenXML
Page StructureContentControlTag, Title
ContentControlsCount
DocumentComputeStatistics, Content
ParagraphFormatAlignment
RangeText
ShapesCount
StyleNameLocal
AnnotationsCommentAuthor, Done, Index
CommentsCount
DocumentTrackRevisions
EndnotesCount, NumberStyle
FootnotesCount, NumberingRule, NumberStyle
RangeStart, Text
RevisionsCount
ReferencesBibliographyBibliographyStyle
ContentControlTitle
ContentControlsCount
HyperlinkAddress, TextToDisplay
HyperlinksCount
IndexesCount
RangeText
SourcesCount
TableOfContentsIncludePageNumbers
TablesOfContentsCount

Table 5 (PDF p. 17).

Target Postconditions and Preservation

Verification schemaTarget postconditionPreservation invariants
Run / character formattingRange.Font.* (weight, style, size, color, highlight, text effects) on the target rangeNon-target (“control”) text content and formatting; paragraph count
Paragraph formattingParagraphFormat.* (alignment, indents, line spacing, space before/after, borders)Paragraph text; non-target paragraph properties; paragraph count
ListsListFormat (list string, list level) on the target paragraphsParagraph text; non-target list state; paragraph count
Shape / picture geometry & effectsShape.{Left, Top, Width, Height, Rotation, ZOrderPosition}, wrap type, fill / line / shadow / glow / reflectionDocument structure counts (shape, inline-shape, footnote, section); non-target shapes
TablesRow and column counts, cell content, borders, styleTable dimensions; non-target cells
Fields & references (footnote, endnote, TOC, citation)Field count, resolved content, reference positionBody text when the field is the sole mutation
Headers / footersHeader/footer story content, field countDocument body text; section count

Table 6. Representative verification schemas, target postconditions, and preservation categories from the benchmark design. Each case applies the checks defined by its oracle specification; not every listed invariant applies to every case.

Evaluator Validation

Positive control. We submit the reference document of each of the 303 cases in place of the agent output. Word Object accepts all 303.

Human reference. We compare evaluator judgments with the majority of three human evaluators on 303 agent-generated Word documents. Each human evaluator judged the agent output against the instruction by opening it alongside the input document in Word, without seeing any evaluator verdict, and could open the reference document on request. We exclude 41 documents that Word cannot open or that fail XML or package-structure checks, and four documents with no binary majority. The comparison set contains 258 documents, 167 labeled Pass and 91 labeled Fail. The box below reproduces the instructions given to the human evaluators.

Human evaluator instructions

# WordEval - Human Judging Package

You will judge whether an AI model correctly edited a Word document.
Your labels help validate WordEval’s automated checking against human
judgment.

For each case you see an instruction and download two documents - the
Original (before the edit) and a Candidate (the model’s attempt). You
open both in Word and decide, independently: did the Candidate
correctly carry out the instruction? You are never shown any automated
verdict - your own judgment is the point.

Some instructions bundle several sub-edits. Those show a checklist: you
judge each part separately, then give an overall verdict.

[... setup steps omitted ...]

## What You Need
- Windows with Microsoft Word - you judge the real .docx in Word, not a
  picture. A flat image hides differences that matter (a real footnote
  vs a typed superscript, a real field vs typed text, styles, page
  breaks vs blank lines).

## How to Judge a Case
1. Read the instruction shown on screen.
2. Download both documents and open them in Microsoft Word.
3. Compare before -> after. Helpful Word tools: Home -> Show/Hide
   (show marks), Reveal Formatting (Shift+F1).
4. Answer:
   - Does the Candidate correctly perform the instruction? - Pass /
     Fail / Unsure
   - How confident are you? - High / Medium / Low
   - Notes (optional, but please add them when unsure or when
     something is subtle).
5. Click Submit & next. [...]

How to decide:
- Judge against the instruction, not against one specific output - a
  different but valid way still passes. There is no single required
  result.
- Missing the required change, wrong target, wrong element type, or
  destroying other content -> fail.
- Pick unsure only when you genuinely cannot tell even after opening
  both documents in Word.

### Multi-part cases
When the instruction bundles several sub-edits, judge each checklist
row on its own merits: did that sub-edit happen correctly? Then give
the overall verdict separately - it is not automatically "all parts
pass": parts can interfere with each other, and unrelated content may
have been damaged.

### Stuck?
If you genuinely cannot tell what a correct result should look like
(e.g. an obscure Word feature), expand "Stuck? Reveal one correct
example" to download a Reference.docx - one example of a correct
output. Use it sparingly, and keep judging by the instruction: a
result that differs from the example can still pass. You will not
need it for most cases.

[... file-return steps omitted ...]

## Time
There is no time limit - take your time on each case. (A timer runs
silently for analysis; it is never shown and has no cap.) If a case
feels broken or impossible after a reasonable effort, mark it unsure
with a note and move on.

[... troubleshooting omitted ...]

Compared evaluators. Besides Word Object, we evaluate Word Revision [11], SSIM [23] between RGB renderings of the reference and submitted documents at thresholds of 0.99 and 0.999, and an agent judge. The agent judge runs GPT-6 Astra through Codex CLI with high reasoning effort and document-analysis tools. It sees the instruction, the input document, and the submitted document, but not the reference document, the human judgments, or Word Object results.

Metrics. Pass is the positive class. Precision is the fraction of accepted outputs that the human majority labels Pass. The false-positive rate (FPR) is the fraction of majority-Fail outputs accepted, and the false-negative rate (FNR) is the fraction of majority-Pass outputs rejected.

Results. Word Object accepts 5 of the 91 majority-Fail outputs and rejects 28 of the 167 majority-Pass outputs, for 96.5% precision, 5.5% FPR, and 16.8% FNR (Table 7). Word Revision reaches a similar precision of 97.1% but rejects 100 majority-Pass outputs, an FNR of 59.9%. In the comparison set, it reports every SmartArt output as one shape deletion and one shape insertion, whether or not the edit is correct. It therefore rejects all 13 majority-Pass SmartArt outputs. At SSIM ≥ 0.99, 41 of the 91 majority-Fail outputs pass, an FPR of 45.1%. Raising the threshold to 0.999 lowers the FPR to 7.7% but raises the FNR from 16.2% to 27.5%. Word Object is lower on both rates. In one header page-numbering task, the submitted document passes even the 0.999 threshold, yet both the human majority and Word Object reject it. Word Object reports a mismatch in the header style. The agent judge rejects fewer majority-Pass outputs than Word Object (FNR 12.0% versus 16.8%) but accepts 11 majority-Fail outputs, more than twice the FPR of Word Object (12.1% versus 5.5%).

EvaluatorPrecision (%) ↑FPR (%) ↓FNR (%) ↓
Word Object96.55.516.8
Word Revision97.12.259.9
SSIM ≥ 0.9977.345.116.2
SSIM ≥ 0.99994.57.727.5
agent judge93.012.112.0

Table 7. Evaluator judgments against the original human majority on 258 valid documents (167 Pass, 91 Fail). Pass is the positive class; all values are percentages.

Evaluation Setup and Prompts

We evaluate the eight agent systems listed in Table 3.

Agent systemAtomicAtomicCompositionalCompositionalOverallOverall
CaseMacroCaseMacroCaseMacro
Claude Code (Opus 5)87.589.284.485.587.188.8
Codex (GPT-5.6, high)67.668.261.560.366.767.3
Kimi Code (K3, high)63.468.153.155.861.866.8
OpenCode (Gemini 3.8 Flash, high)57.963.249.051.356.561.9
ZCode (GLM-5.2, high)58.263.739.640.455.561.2
OpenCode (DeepSeek V4 Pro)54.961.740.640.352.859.5
Qwen Code (Qwen3.8-27B)50.456.431.232.447.553.9
Mini Agent (MiniMax M3)47.354.929.230.644.552.4
Human (mean)85.286.769.468.482.884.8

Table 3. Pass rates (%) on WordEval under the Word Object oracle. Case weights cases equally, whereas Macro averages within each Atomic command or Compositional scenario and then across commands or scenarios (§ 4.1). Bold marks the best agent result in each column. Human reports the mean across three participants.

Agent Inputs and Task Prompt

Agents receive the task instruction and input document and are asked to return pred.docx with trajectory documentation. Reference documents and oracle files are available only to the evaluator. Figure 7 illustrates the task-file protocol. CLI wrappers and model-specific effort settings vary across systems.

Task-file protocol with an abridged task description

Figure 7. Task-file protocol with an abridged task description.

Per-case prompt

Please follow @TASK_DESCRIPTION_per_task.md to complete the task.  Only access files
in the current directory.

Task description (TASK_DESCRIPTION_per_task.md), abridged

# Word Command Benchmark - Task Description

## Overview
This benchmark evaluates your ability to execute Microsoft Word
formatting and editing commands on documents. Each test case provides
an input Word document and a natural language instruction describing
the modification to perform.

## Task Structure
The current directory contains a single test case:
  doc_in.docx       # Input document
  instruction.txt   # Task instruction
  pred.docx         # YOUR OUTPUT (to be created)
  method.md         # YOUR METHOD (full reproducible trajectory)

## Your Task
1. Read the instruction.txt file to understand what modification
   needs to be made
2. Open the doc_in.docx file (input document)
3. Execute the instruction on the document
4. Save the result as pred.docx in the current directory
5. Document your complete trajectory in method.md

## Method Documentation Requirements
# Status
STATUS: [SUCCESS|PARTIAL|FAILED]
REASON: [Optional: explain if partial or failed]
# Approach / # Step-by-Step Trajectory / # Complete Implementation

### Status Definitions
- SUCCESS: Task completed as instructed
- PARTIAL: Task attempted but may not be fully correct
- FAILED: Could not complete the task

You can use any method you prefer.
[... four worked examples (python-docx, VBA, Word GUI, other tools
such as Office.js, OpenXML SDK, Apache POI, pandoc) omitted ...]

Agent Systems and Run Settings

Qwen Code uses a 50-turn budget and OpenCode with Gemini a 50-step budget, each with a 600-second per-case limit. Failed runs and missing outputs remain in the 642-case denominator. All outputs were evaluated with Microsoft Word 16.0.17932.21000 on Windows 11 virtual machines (OS build 10.0.26200) with Python 3.12. Opus 5 uses a Copilot compatibility route for the Stack Object case. Table 8 reports packages, available versions, and model identifiers.

Agent systemPackageVersionModel identifier
Claude Code@anthropic-ai/claude-code–Opus 5
Codex@openai/codex0.128.0gpt-5.6-sol
Kimi Codekimi-cli1.41.0Kimi-K3
OpenCodeopencode1.18.3gemini-3.8-flash
ZCodezcode-agent-cli0.15.2GLM-5.2
OpenCodeopencode–DeepSeek-V4-Pro
Qwen Code@qwen-code/qwen-code0.22.3Qwen/Qwen3.8-27B
Mini AgentMini-Agent–MiniMax M3

Table 8. Agent packages and model identifiers. – denotes an unavailable version.

Human Execution Protocol

Three participants (A01–A03) each perform the same 642 editing tasks in Microsoft Word, through a local browser page that presents one task at a time with its instruction and input document. Tasks are grouped into bundles, and the Atomic tasks in a bundle precede the Compositional task that combines them. A practice task precedes the benchmark tasks and is excluded from analysis. For each task, the participant downloads the input document, carries out the instruction in Word, and uploads the saved document. The instructions given to participants, reproduced in the box below, specify which tools may be used. The final output submitted for each task is scored with the same evaluator as agent outputs. These execution results are distinct from the human judgments of agent outputs in Appendix § 9.2.§ 3. Table 9 reports individual Case and Macro pass rates, and the Human row in the main table averages the three participant-specific rates.

Human execution instructions, abridged

# WordEval Human Annotation

This task evaluates whether a human annotator can execute a Microsoft
Word editing instruction on a document, and validates the WordEval
benchmark cases (instruction clarity + Word Object oracle correctness).
Each task contains one input Word document and one natural-language
instruction.

Cases are organized into task bundles. Within each bundle, complete the
atomic tasks first, then complete the corresponding compositional task.

Each task has two stages:
1. Stage 1: Word-only baseline - complete the task using only memory,
   ribbon browsing, and in-document text search (no oracle).
2. Stage 2: Assisted attempt - optional, after Stage 1. Use external
   help to attempt the task again, starting from a fresh copy of the
   input document.
[... setup, onboarding, and step-by-step interface instructions omitted ...]

Stage 1 - Allowed
- Memory and ribbon/menu browsing
- Right-click menus and keyboard shortcuts
- Ctrl+F for in-document text search

Stage 1 - Forbidden
- Search box at the top of the Word window (the bar above the ribbon labeled "Search" or "Tell me what you want to do", with a magnifying glass icon). Typing into it suggests commands or invokes Copilot.
- Copilot panel (the AI sidebar accessible from the ribbon)
- F1 Help when it opens docs.microsoft.com or any web page
- Web search, ChatGPT, Claude, or any other AI assistant
- Scripts, macros, VBA, WordAgent, or programmatic editing

Stage 1 measures what a Word user can do from memory and ribbon browsing alone, matching the agent’s no-oracle condition.

Stage 2 - Allowed
- Everything Stage 1 allows, plus:
- Search box at the top of the Word window ("Search" / "Tell me")
- Word’s Copilot panel (for explanations only)
- F1 Help and docs.microsoft.com
- Web search, YouTube tutorials, Stack Overflow
- ChatGPT / Claude / external Copilot as a tutor (ask, read, then perform the clicks yourself)

Stage 2 - Forbidden
- AI directly editing the docx (no AI-generated macros, VBA, scripts, or XML run on the document)
- Pasting AI-generated final content into the docx as a shortcut
- Any agent or script that performs the edit on your behalf

AI may explain, not execute. You must perform the final clicks in Word yourself.

Compare to Gold (Submit step)
Before final submission, open gold.docx (the reference output) in Word alongside your final uploaded result (pred_stage2.docx if you continued, otherwise pred_stage1.docx), and rate human_matches_gold.
[...]

Instruction Specification Severity (0-3)
Rate from the perspective of another reasonable Word user, not your own struggle. Anchors:
- 0 - clear, all reasonable annotators converge on the same edit
- 1 - minor interpretation needed, a reasonable default exists
- 2 - important detail missing or multiple valid interpretations
- 3 - too ambiguous to execute, or contradicts the docx

human_matches_gold (Binary)
- match - the targeted edits from the instruction are realized in your output as in gold
- not_match - at least one targeted edit is missing or applied differently

[... status and confidence definitions omitted ...]

Time Policy
There is no time limit. Take your time on each case. If a case feels broken or impossible after a reasonable effort (~10 min), mark it as failed and move on.

Code and Data Availability

The released WordEval artifact is organized around a canonical benchmark directory with separate Atomic and Compositional tracks. Each track contains agent-visible task records, input .docx files, evaluator-only reference documents, oracle metadata, and task-level taxonomy metadata. The agent-visible records specify only the task identifier, natural-language instruction, and input document path; the evaluator-only records specify the corresponding reference document and oracle specification used to score predictions. This separation mirrors the evaluation protocol in § 3.1: agents receive the input document and instruction, while reference documents and oracle metadata are used only by the evaluator.

The artifact also includes the benchmark generator, Word Object evaluator, scripts for producing agent-visible task folders and scoring prediction files, a README, a data card, license files, and Croissant metadata. The primary evaluator requires Windows, Microsoft Word, and Python COM automation, matching the environment assumptions in § 4.1. Code is released under the MIT License, and the benchmark data are released under CDLA-Permissive-2.0. Code and data are available at https://aka.ms/word-eval. The data card and Croissant metadata document the dataset schema, intended use, limitations, privacy properties, licensing, and responsible-release metadata.

Additional Compositional Analyses

The main-text analysis (§ 4.4) treats each compositional case as a single binary outcome, so the whole case either passes or fails. Every released compositional case (96 cases / 319 atomic intents across the four classes) declares the atomic intents it contains via spec.composition.atomic_intents, and per-intent verdicts are extracted from the recorded verifier diagnostics when available. This appendix uses those per-intent verdicts to decompose outcome patterns that case-level outcomes collapse. These are operation-limited observations where the embedded intent fails along with its referenced Atomic case, transfer failures where a referenced Atomic pass verdict does not carry over to the corresponding embedded atomic intent, and end-state coincidences where a compositional final state satisfies an intent despite the corresponding Atomic failed verdict. Operation, feature area, and depth breakdowns are reported as secondary diagnostics after this intent-level view.

Intent-Level Outcome Decomposition

Definitions. For each declared intent ii, AiA_i is the verdict of its referenced Atomic case in isolation, and CiC_i is the verdict of the same intent inside the compositional output. The transfer rate Pr⁡(Ci∣Ai)\Pr(C_i \mid A_i) is the share of intents passing in isolation that also pass inside the composition. Each pair of AiA_i and CiC_i verdicts falls into one of four cells. ACAC marks an intent that passes in both settings. AC‾A\overline{C} marks a transfer failure, where the Atomic case passes but the embedded intent fails. A‾C\overline{A}C marks an operation-limited observation, where both fail. A‾C‾\overline{A}\overline{C} marks an end-state coincidence, where the compositional output satisfies the intent’s final-state check although the Atomic case fails. The cells describe outcomes, not their causes.

Coverage. The eight systems yield 8×319=25528 \times319 = 2552 declared intent observations. Embedded verdicts come from the verifier’s per-intent diagnostics whenever a prediction document exists and its diagnostics cover the intent. Two cases that record their checks in other diagnostic structures are mapped explicitly. Each intent verdict thus reflects only its own check. Together with the Atomic verdicts, these yield 1701 observed pairs (66.7%) from 36.1% to 92.5% across systems, with the remainder lacking an embedded or a binary Atomic verdict. All intent-level rates are computed over these observed pairs.

Composition classes. Table 10 pools the four cells across the eight systems by composition class. Its transfer-failure column is AC‾A\overline{C} over all Atomic-pass pairs, its operation-limited column is the share of A‾C‾\overline{A}\overline{C} among all pairs, and its coincidence column is the share of A‾C\overline{A}C among all pairs. Layout-coupled cases have the highest transfer-failure rate, 28.1%. Additive and prerequisite cases have the largest operation-limited shares, 9.7% and 8.2%. In layout-coupled cases, 45 of the 48 failed embedded intents passed in isolation. In additive and prerequisite cases, 113 of 195 and 18 of 28 failed embedded intents also failed in isolation. Embedded failures in layout-coupled cases are therefore mostly transfer failures, whereas more than half of those in additive and prerequisite cases are operation-limited.

ClassPr⁡(Ai)\operatorname{Pr}(A_i)TransferTransfer failOperation-limitedCoincidencenintentn_{\mathrm{intent}}
Additive79.791.28.89.710.71170
Prerequisite70.893.56.58.221.0219
Scope-changing91.995.24.81.56.6136
Layout-coupled90.971.928.11.77.4176

Table 10. Pooled intent outcomes by composition class. Rates use observed pairs across the eight systems (nintentn_{\mathrm{intent}}). Operation-limited and coincidence are the shares of AˉCˉ\bar{A}\bar{C} and AˉC\bar{A}C pairs, and Transfer fail is the share of ACˉA\bar{C} among Atomic-pass pairs.

Intent Transfer by System

Agent systems. Figure 8 reports Pr⁡(Ai)\operatorname{Pr}(A_i), Pr⁡(Ci)\operatorname{Pr}(C_i), and the transfer rate Pr⁡(Ci∣Ai)\operatorname{Pr}(C_i \mid A_i) across the four classes for all eight agent systems. This is the intent-level counterpart to the case-level analysis in Figure 5. Panel A gives the Atomic pass rate, panel B asks whether embedded intents pass in compositional outputs, and panel C asks the same question only for intents whose referenced Atomic case passes in isolation. Class differences vary across systems. Opus 5, GPT-5.6, Gemini 3.8 Flash, and K3 transfer at least 89.9% of their Atomic-pass intents in every class, while the other four systems have their lowest transfer on layout-coupled cases. Because intents without an observed embedded verdict are excluded, some system–class cells have small denominators (Table 11).

For the case-level conditioning used in Figure 5, Table 11 reports the case-level rates and the all-components-pass denominators for all eight systems, together with their intent-level rates. Its intent-level columns place Pr⁡(Ai)\operatorname{Pr}(A_i) and Pr⁡(Ci)\operatorname{Pr}(C_i) side by side, and their difference is the pass gap. On layout-coupled cases, the pass gap is largest for the same four systems, suggesting composite-intent loss that is not fully explained by atomic-component verdicts on the same operations. Section D.4 examines cells with negative gaps.

SystemClassP(A)P(A)P(C)P(C)P(C∣A)P(C \mid A)nAn_APr⁡(Ai)\Pr(A_i)Pr⁡(Ci)\Pr(C_i)Pass gapPr⁡(Ci∣Ai)\Pr(C_i \mid A_i)nC∣An_{C\mid A}
Opus 5Additive80.482.689.23797.493.83.693.7190
Prerequisite80.080.085.02095.595.50.097.642
Scope-changing58.3100.0100.0795.8100.0-4.2100.023
Layout-coupled100.084.684.613100.093.86.293.832
GPT-5.6Additive37.056.558.81773.290.9-17.791.7120
Prerequisite44.052.054.51172.790.9-18.295.824
Scope-changing58.383.3100.0790.0100.0-10.0100.018
Layout-coupled69.276.966.7985.792.9-7.191.724
K3Additive41.347.889.51985.682.72.989.9119
Prerequisite28.048.042.9760.088.6-28.695.221
Scope-changing75.075.066.7995.295.20.095.020
Layout-coupled38.561.580.0594.494.40.094.117
Gemini 3.8 FlashAdditive28.345.776.91396.2100.0-3.8100.076
Prerequisite40.048.060.010100.0100.00.0100.018
Scope-changing25.058.3–3100.0100.00.0100.06
Layout-coupled15.453.8–2100.091.78.391.712
GLM-5.2Additive41.343.573.71982.885.9-3.193.4106
Prerequisite16.032.0–445.586.4-40.990.010
Scope-changing41.758.3100.0588.994.4-5.6100.016
Layout-coupled61.523.125.0890.547.642.942.119
DeepSeek V4 ProAdditive39.139.161.11869.179.2-10.187.0123
Prerequisite28.036.057.1764.382.1-17.988.918
Scope-changing83.366.770.01090.585.74.889.519
Layout-coupled53.830.828.6784.060.024.052.421
Qwen3.8-27BAdditive37.030.464.71778.570.28.383.295
Prerequisite40.024.030.01063.677.3-13.685.714
Scope-changing33.366.7–486.793.3-6.792.313
Layout-coupled46.215.416.7691.737.554.236.422
MiniMax M3Additive26.134.883.31262.068.1-6.091.3103
Prerequisite8.016.0–247.164.7-17.675.08
Scope-changing58.341.742.9790.981.89.180.010
Layout-coupled38.523.140.0581.262.518.869.213

Table 11. Case-level and intent-level rates by system and composition class. Case-level rates use all assigned cases (46, 25, 12, and 13 per class) and count missing outputs as failures. P(A)P(A) is the share of cases whose linked Atomic variants all pass, and nAn_A counts them. P(C∣A)P(C \mid A) is omitted (–) when fewer than 5 such cases exist. Intent-level rates use observed pairs, the pass gap is Pr⁡(Ai)−Pr⁡(Ci)\Pr(A_i) - \Pr(C_i), and nC∣An_{C\mid A} counts Atomic-pass intents.

Closed- and open-weight systems. Among the evaluated agent systems, those using open-weight models (K3, MiniMax M3, Qwen3.8-27B, GLM-5.2, DeepSeek V4 Pro) have lower values than those using closed-weight models (Opus 5, GPT-5.6, Gemini 3.8 Flash) on both the embedded pass rate and the transfer rate (Figure 8, Table 12). A paired case-cluster bootstrap [6] with 2,000 resamples of shared cases estimates positive closed-weight versus open-weight transfer differences in all four classes, +5.3 pp for additive (95% CI [1.2, 10.2]), +8.2 pp for prerequisite ([0.0, 18.1]), +7.7 pp for scope-changing ([0.0, 18.4]), and +35.7 pp for layout-coupled cases ([19.5, 53.7]).

Figure 8. Intent-level rates by system and composition class. (A) Atomic pass rate Pr⁡(Ai)\operatorname{Pr}(A_i). (B) Embedded pass rate Pr⁡(Ci)\operatorname{Pr}(C_i), measured inside the compositional output. (C) Transfer rate Pr⁡(Ci∣Ai)\operatorname{Pr}(C_i \mid A_i), among intents whose Atomic case passes in isolation. Rates use observed pairs, and Table 11 lists the values.

Classclosed (%)open (%)Δ\Delta (pp)95% CInclosedn_{\mathrm{closed}}nopenn_{\mathrm{open}}clusters
Additive94.388.95.3[1.2, 10.2]38454244
Prerequisite97.088.78.2[0.0, 18.1]667119
Scope-changing100.092.37.7[0.0, 18.4]477812
Layout-coupled92.256.535.7[19.5, 53.7]649212

Table 12. Closed- versus open-weight transfer by composition class. Δ\Delta is closed minus open in percentage points with its 95% paired bootstrap CI, nn counts Atomic-pass intents, and clusters counts the cases observed in both groups.

Breakdown by Operation and Word Feature Area

Table 13 ranks atomic operations by intent-level transfer rate, pooled across the eight agent systems and restricted to operations with at least 15 observed pairs. Apart from shadow and reflection effects (29.4–44.4%), the lowest-transfer operations are geometric/spatial primitives (rotate_angle 47.2%, flip_direction 58.8%, align_command 72.6%), followed by list and icon-insertion primitives. This is consistent with the pattern by composition class. Layout-coupled and certain additive bundles share some of the same problematic primitives, suggesting that the underlying spatial coordination that those classes over-represent contributes to low transfer.

Operationnintentn_{\mathrm{intent}}Pr⁡(Ai)\Pr(A_i)Pr⁡(Ci)\Pr(C_i)Pr⁡(Ci∣Ai)\Pr(C_i \mid A_i)nC∣An_{C\mid A}
shadow1989.526.329.417
pic_shadow1984.226.331.216
reflection1752.935.344.49
rotate_angle4678.352.247.236
flip_direction17100.058.858.817
align_command73100.072.672.673
list_type2080.065.075.016
insert_icon1776.588.292.313
distribute_command47100.093.693.647
position_preset2295.590.995.221
legend3090.096.796.327
alignment46100.0100.0100.046

Table 13. Twelve lowest-transfer operations with at least 15 observed pairs, pooled across the eight systems. nintentn_{\mathrm{intent}} counts observed pairs.

Word feature areanintentn_{\mathrm{intent}}Pr⁡(Ai)\Pr(A_i)Pr⁡(Ci)\Pr(C_i)Pr⁡(Ci∣Ai)\Pr(C_i \mid A_i)nC∣An_{C\mid A}
Annotations5184.398.097.743
Charts17188.995.397.4152
Equations7561.392.093.546
Graphics1776.588.292.313
Lists4887.577.183.342
Page Structure4973.569.491.736
Paragraphs22589.396.999.5201
Pictures12859.446.961.876
References1471.478.690.010
Shape appearance13080.872.385.7105
Shape geometry28689.279.079.6255
Shape text1384.684.681.811
SmartArt12874.271.988.495
Tables13385.793.294.7114
Text23374.293.696.0173

Table 14. Intent transfer by Word feature area, pooled across the eight systems. nintentn_{\mathrm{intent}} counts observed pairs.

Table 14 groups transfer by Word feature area, the axis of Figure 4. Pictures (+12.5 pp), Lists (+10.4 pp), and Shape geometry (+10.1 pp) have the largest pass gaps. Text and Equations have negative gaps (−19.3-19.3 pp and −30.7-30.7 pp) because end-state coincidences outnumber transfer failures in these areas. Section D.4 examines these coincidences.

End-State Coincidence Audit

Intents can also pass more often inside compositions than in isolation. The pass gap Pr⁡(Ai)−Pr⁡(Ci)\operatorname{Pr}(A_i)-\operatorname{Pr}(C_i) is negative in prerequisite cases for six of the eight systems, and GPT-5.6 has negative gaps in every class, from −7.1-7.1 to −18.2-18.2 pp. In these cells, end-state coincidences outnumber transfer failures.

The AˉC\bar{A}C cell – Atomic-failed but composite-intent-passed – accounts for 193 of 1,701 observed pairs (11.3%), of which 121 (62.7% of the end-state coincidence set, 7.1% of all observed pairs) appear in cases that also pass at case level. In a composite, another unit or a no-op can sometimes satisfy the same end-state predicate (e.g., ungroup_command passes when no group exists). The transfer rate Pr⁡(Ci∣Ai)\operatorname{Pr}(C_i \mid A_i) is computed over Atomic-pass pairs, so the AˉC\bar{A}C cell does not enter it, and the case-level metric P(C)P(C) counts these cases as passes.

The most frequent operations among these coincidences are insert_equation (17), drop_cap (12), outline_weight (10), outline_color (8), and rotate_angle and table_style (7 each). Table 15 shows one example from a passing case for each of these operations and for ungroup_command.

OperationSystemClassCompositional caseAtomic case
insert_equationGPT-5.6PrerequisitePrerequisiteEquation/variant_fractionEquationInsert New/apply
drop_capGLM-5.2AdditiveAdditiveParagraph Composition/variant_center_dropcapDropCapInsert Gallery/dropped
outline_weightMiniMax M3AdditiveAdditiveFontComposition/variant_mixed_effectsTextOutline WeightGallery/apply_1pt
outline_colorK3AdditiveAdditiveFontComposition/variant_off_heavyTextOutlineColor PickerAlternate/apply_Blue
rotate_angleGPT-5.6AdditiveAdditiveLayout Composition/variant_rotate_cw_distribute_hObjectRotate Gallery/apply_rotate_right_90
table_styleQwen3.8-27BAdditiveAdditiveTable Composition/variant_style_textdir_borderTableStyles GalleryWord/Grid_Table_4___Accent_1
ungroup_commandDeepSeek V4Layout-coupledLayoutCoupledGroup Lifecycle/variant_flip_horizontalObjectsUngroup/apply

Table 15. End-state coincidence examples by operation. Each row is a case with an Atomic Fail, embedded Pass, and whole-case Pass.

Composition Depth

Composition depth kk is the number of atomic intents in a compositional case. The case-level pass rate of Opus 5 is 88.7% at k=2k=2 and 92.3% at k=3k=3, then falls to 81.8% at k=4k=4, 69.2% at k=5k=5–6, and 66.7% at k≥7k \ge7 (Figure 9). Its per-intent transfer stays between 90.9% and 96.8% in every bucket, including k≥7k \ge7 (Figure 10).

Figure 9. Case-level pass rates by composition depth for the eight systems. (A) P(C)P(C) over all assigned cases, counting missing outputs as failures. (B) P(C∣A)P(C \mid A) over cases whose linked Atomic variants all pass, omitted where no case is eligible. Depth buckets hold 53, 13, 11, 13, and 6 cases at k=2k = 2, 3, 4, 5–6, and k≥7k \geq7, and the k≥7k \geq7 bucket has a mean depth of 10.2.

Figure 10. Intent-level rates by composition depth for the eight systems. (A) Atomic pass rate Pr⁡(Ai)\Pr(A_i). (B) Embedded pass rate Pr⁡(Ci)\Pr(C_i). (C) Transfer rate Pr⁡(Ci∣Ai)\Pr(C_i \mid A_i). Rates use observed pairs.

Task Examples and Complete Command Catalogue

This section collects the task instructions, input-document examples, and command enumeration supporting Appendix A.

Representative Case Examples

Table 16 and Table 17 give representative instructions shown to agents for single-operation Atomic cases and multi-operation Compositional cases, respectively. Figure 11 and Figure 12 show, for one released case per track, the exact instruction together with a rendered preview of the input document.

Word feature areaSource commandInstruction example
TextBoldMake the word ’Target’ bold.
ParagraphsAlignCenterCenter the paragraph containing ’Target’.
ListsBulletsGalleryWordApply bullet list formatting to all paragraphs in the document using checkmark bullets.
TablesTableInsertGalleryInsert a 2x2 table.
Shape geometryAlignDistributeHorizontallyDistribute all three rectangles evenly horizontally with equal spacing between them.
Shape appearanceShapeFillColorPickerSet the shape’s fill color to Blue (RGB: 0, 112, 192).
Shape textTextAlignGalleryAlign the text to the bottom inside the text box.
PicturesPictureCropAspectRatio1To1Crop the image to a 1:1 (square) aspect ratio, centered on the image.
GraphicsIconInsertFromFileInsert a 150 x 150 pt star-shaped SVG icon filled with blue color (RGB: 68, 114, 196) into the document.
ChartsChartInsertInsert a clustered column chart into the document with categories ’Q1’ and ’Q2’, and one series ’Revenue’ with values 100 and 150.
SmartArtSmartArtInsertInsert a SmartArt graphic using the ’Basic Block List’ layout with nodes labeled ’Alpha’, ’Beta’, ’Charlie’, ’Delta’, and ’Echo’.
EquationsEquationFractionGalleryAdd the fraction ’dy/dx’ to the equation ’y =’.
Page StructurePageBreakInsertWordInsert a page break after the word ’TARGET’ to start a new page.
AnnotationsInsertNewCommentAdd a comment with text ’Test comment.’ on the word ’TARGET’.
ReferencesCitationInsertAt the beginning of the document body, insert an in-text citation using the existing source tagged ’Smith2020’.

Table 16. Representative atomic WordEval case examples by Word feature area.

Instruction and input document shown to agents for a representative atomic case

Figure 11. Instruction and input document shown to agents for a representative atomic case (AlignDistributeHorizontally, Shape geometry feature area). The input page render is cropped to the content region and presented in a Windows Word application frame.

Composition classReferenced atomic commandsInstruction example
AdditiveAlignCenter; ParagraphSpacingBefore; ParagraphSpacingAfter; DropCapInsertGalleryCenter the paragraph containing the word ’Target’, set paragraph spacing to 0 points before and 0 points after for all paragraphs, and apply a dropped drop cap to the first paragraph.
PrerequisiteChartInsert; ChartDataLabelBelow the ’Quarterly Sales:’ heading, insert a clustered column chart with categories ’Q1’ and ’Q2’, and one series ’Revenue’ with values 100 and 150, then add data labels at the ’Outside End’ position.
Scope-changingTableRowsInsertBelowWord; TableColumnsInsertRight; TableSplitTable; TableColumnsDistribute; TableRowsDistributeInsert a new row below the first row and a new column to the right of the first column, split the table at the inserted row, then distribute the columns in the upper table and distribute the rows in the lower table.
Layout-coupledTextWrapGallery; PicturePositionGalleryApply square wrapping to the cross shape (text wraps around the cross’s rectangular bounding box), then position it at the top-center of the page, using the Position preset (within default page margins).

Table 17. Representative compositional WordEval case examples by composition class. Referenced atomic commands are derived from the case’s linked atomic_intents.

Instruction and input document shown to agents for a representative layout-coupled compositional case

Figure 12. Instruction and input document shown to agents for a representative layout-coupled compositional case (TextWrapGallery followed by PicturePositionGallery). The input page render is cropped to the content region and presented in a Windows Word application frame.

Instruction

Apply square wrapping to the cross shape (text wraps around the cross’s rectangular bounding box), then position it at the top-center of the page, using the Position preset (within default page margins).

Input document (doc_in.docx)

Complete Released-Command Catalogue

Table 18 enumerates the 231 released Atomic commands by Word feature area.

Word feature areaReleased commands
Shape geometryAlignDistributeHorizontally, AlignDistributeVertically, GalleryAllShapesAndCanvas, ObjectBring Forward, ObjectBringToFront, ObjectRotateGallery, ObjectSendBackward, ObjectSendToBack, ObjectsAlignBottomSmart, ObjectsAlignCenterHorizontalSmart, ObjectsAlignLeftSmart, Objects AlignMiddleVerticalSmart, ObjectsAlignRightSmart, ObjectsAlignTopSmart, ObjectsGroup, ObjectsUngroup, PicturePositionGallery, ShapeChangeShapeGallery, TextBoxInsert, TextBox InsertGallery, TextBoxInsertVerticalWord, TextWrapGallery
Shape appearanceBevelShapeGallery, GlowColorPicker, GradientGallery, ObjectEffectGlowGallery, ObjectEffect ShadowGallery, ObjectEffectSoftEdgesGallery, ReflectionGallery, ShapeFillColorPicker, Shape OutlineColorPicker, ShapeStylesGallery, _3DRotationGallery
Shape textTextAlignGallery, TextDirection, TextDirectionGallery
PicturesHorizontalLineInsert, OutlineColorPicker, PictureArtisticEffectsGallery, PictureColorTemperature Gallery, PictureCropAspectRatio16To10, PictureCropAspectRatio16To9, PictureCropAspectRatio1 To1, PictureCropAspectRatio2To3, PictureCropAspectRatio3To2, PictureCropAspectRatio3To4, Picture CropAspectRatio3To5, PictureCropAspectRatio4To3, PictureCropAspectRatio4To5, Picture CropAspectRatio5To3, PictureCropAspectRatio5To4, PictureEffectsGlowGallery, PictureEffects ShadowGallery, PictureEffectsSoftEdgesGallery, PictureFillCrop, PictureFitCrop, PictureRecolor Gallery, PictureReflectionGallery, PictureReset, PictureResetAndSize, PictureRotationGallery, PictureSaturationGallery, PictureSharpenSoftenGallery, _3DBevelPictureTopGallery
GraphicsGraphicsColorPicker, GraphicsOutlineColorPicker, IconInsertFromFile
ChartsChartColorsGallery, ChartDataLabel, ChartDataTable, ChartGridlinesGallery, ChartInsert, Chart LayoutGallery, ChartLegend, ChartLines, ChartResetToMatchStyle, ChartSwitchRowColumn, ChartTitle, ChartTrendline, ChartUpDownBars, ColorPickerChartLineStyles
SmartArtArrowStyleGallery, BevelTextGallery, OutlineDashesGallery, OutlineWeightGallery, SmartArtAdd Assistant, SmartArtAddShapeAbove, SmartArtAddShapeAfter, SmartArtAddShapeBefore, Smart ArtAddShapeBelow, SmartArtChangeColorsGallery, SmartArtDemote, SmartArtInsert, SmartArt LargerShape, SmartArtLayoutGallery, SmartArtPromote, SmartArtReorderDown, SmartArt ReorderUp, SmartArtResetGraphic, SmartArtRightToLeft, SmartArtSmallerShape, SmartArtStyles Gallery, TextEffectTransformGallery, TextEffects3DRotationGallery, TextFillColorPicker
EquationsEquationAccentGallery, EquationDelimiterGallery, EquationFractionGallery, EquationFunction Gallery, EquationInsertNew, EquationIntegralGallery, EquationLargeOperatorGallery, Equation LimitGallery, EquationLinearAll, EquationLinearOne, EquationMatrixGallery, EquationNormal Text, EquationOperatorGallery, EquationProfessionalAll, EquationProfessionalOne, Equation RadicalGallery, EquationScriptGallery, EquationSymbolsInsertGallery, InsertBuildingBlocks EquationsGallery
Page StructureBlankPageInsert, CoverPageInsertGallery, DocInfoAuthor, DocInfoDocTitle, DocInfoFileName, FooterInsertGallery, HeaderFooterRemoveFooterWord, HeaderFooterRemoveHeaderWord, Header InsertGallery, PageBreakInsertWord, PageNumbersInfoInsertGallery, PageNumbersInMargins InsertGallery, PageNumbersInHeaderInsertGallery, WatermarkRemove
AnnotationsAcceptAllChangesInDocAndStopTracking, DeleteAllResolvedComments, EndnoteInsertWord, FootnoteEndnoteDialog, FootnoteInsert, InsertNewComment, RejectAllChangesInDocAndStop Tracking, ReviewDeleteAllCommentsInDocument, ReviewDeleteComment, ReviewTrackChanges
ReferencesBibliographyGallery, BibliographyInsert, BibliographyStyle, CaptionInsert, CitationInsert, HyperlinkEdit, HyperlinkInsert, HyperlinkRemove, IndexUpdate, PageNumberFieldInsertGallery, TableOfContentsGallery, TableOfContentsRemove

Table 19.

References

  1. [1]Anthropic. Claude code. https://www.anthropic.com/claude-code.
  2. [2]Anthropic. System card: Claude Opus 5, 2026. URL https://www-cdn.anthropic.com/b514064af1408018e64b1ad24e7d5e75850b4ffd/Claude%20opus%205%20System%20Card.pdf.
  3. [3]Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows agent arena: Evaluating multi-modal os agents at scale. In ICML, 2025.arxiv.org/abs/2409.08264
  4. [4]DeepSeek-AI et al. DeepSeek-V4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026. URL https://arxiv.org/abs/2606.19348.
  5. [5]ECMA International. Standard ECMA-376: Office Open XML file formats. Technical report, ECMA International, 2016. URL https://ecma-international.org/publications-and-standards/standards/ecma-376/.
  6. [6]Bradley Efron. Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7 (1):1–26, 1979.
  7. [7]GitHub. GitHub Copilot: Your ai pair programmer. URL https://github.com/features/copilot.
  8. [8]GLM-5 Team et al. GLM-5: From vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026. URL https://arxiv.org/abs/2602.15763.
  9. [9]Google DeepMind. Gemini 3.8 Flash model card. https://deepmind.google/models/model-cards/gemini-3-8-flash/.
  10. [10]Kimi Team et al. Kimi K3: Open frontier intelligence. arXiv preprint arXiv:2607.24653, 2026. URL https://arxiv.org/abs/2607.24653.
  11. [11]Microsoft. Application CompareDocuments method (Word). https://learn.microsoft.com/en-us/office/vba/api/word.application.comparedocuments.
  12. [12]Microsoft. Office fluent UI command identifiers. https://github.com/OfficeDev/office-fluent-ui-command-identifiers.
  13. [13]Microsoft. Object model (Word). https://learn.microsoft.com/en-us/office/vba/api/overview/word/object-model.
  14. [14]MiniMax. Mini-Agent. https://github.com/MiniMax-AI/Mini-Agent.
  15. [15]MiniMax. MiniMax M3: Frontier coding, 1m context, native multimodality—all in one model. https://www.minimax.io/blog/minimax-m3, June 2026.
  16. [16]Moonshot AI. Kimi Code CLI. https://moonshotai.github.io/kimi-code.
  17. [17]OpenAI. Introducing Codex. https://openai.com/index/introducing-codex/, 2025.
  18. [18]OpenAI. GPT-5.6 system card. https://deploymentsafety.openai.com/gpt-5-6, 2026.
  19. [19]OpenCode. OpenCode. https://opencode.ai.
  20. [20]Qwen Team. Qwen3.8-Max: A new bar for coding and cowork. https://qwen.ai/blog?id=qwen3.8,.
  21. [21]Qwen Team. Qwen Code. https://qwenlm.github.io/qwen-code-docs,.
  22. [22]Weixuan Wang, Dongge Han, Daniel Madrigal Diaz, Jin Xu, Victor Rühle, and Saravan Rajmohan. Odysseybench: Evaluating llm agents on long-horizon complex office application workflows. arXiv preprint arXiv:2508.09124, 2025.arxiv.org/abs/2508.09124
  23. [23]Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 13(4):600–612, 2004.pubmed.ncbi.nlm.nih.gov/15376593
  24. [24]Zilong Wang, Yuedong Cui, Li Zhong, Zimin Zhang, Da Yin, Bill Yuchen Lin, and Jingbo Shang. Officebench: Benchmarking language agents across multiple applications for office automation. arXiv preprint arXiv:2407.19056, 2024.arxiv.org/abs/2407.19056
  25. [25]Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In NeurIPS, 2024.arxiv.org/abs/2404.07972
  26. [26]Z.ai. ZCode. https://github.com/zai-org/ZCode.
  27. [27]Henry Hengyuan Zhao, Kaiming Yang, Wendi Yu, Difei Gao, and Mike Zheng Shou. Worldgui: An interactive benchmark for desktop gui automation from any starting point. arXiv preprint arXiv:2502.08047, 2025.arxiv.org/abs/2502.08047

Paper details

vlnc:W_t2n7edgz
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents