WorksComputer Science
WordEval: Evaluating Word-Native Operation Fidelity in Document Editing
Abstract
Microsoft Word is a central environment for office automation, and the faithful execution of its editing operations is an important capability for document agents. Unlike plain-text editing, editing in Word modifies a structured object model, where the requested effect is realized in object properties. Existing benchmarks, however, evaluate office workflows or desktop automation, treating word-processor documents as one task domain among many and verifying outcomes with keyword matching, per-task property checks, or screenshot comparison. We introduce WordEval, an operation-fidelity benchmark that instead centers evaluation on Word editing operations, testing whether each requested edit is realized as intended changes to Word objects and properties. It contains 642 cases across 15 Word feature areas, ranging from text and lists to charts, shape geometry, and SmartArt. 546 atomic cases cover 231 commands, 96 compositional cases combine multiple edits in a single document, and a Word-native oracle checks task-specific object properties and preservation conditions. Cases are constructed from editing-unit specifications, yielding controlled input documents, reference documents, and aligned oracle checks. Each compositional case is linked to corresponding atomic cases, supporting fine-grained diagnosis of editing failures. We evaluate eight agent systems and find that many struggle on the Atomic track, with performance varying across feature areas. Moreover, every system scores lower on Compositional than on Atomic, with gaps that vary substantially by system. Exact atomic-intent links and recorded failure diagnoses provide complementary views of this variation.
https://aka.ms/word-eval
Introduction
AI assistants have begun to operate inside the software where users work, including word processors such as Microsoft Word [1, 7, 17]. In this context, their output is no longer merely a natural-language reply, but a document that the user or agent itself can create, edit, and reuse. A Word document, however, is not merely a text canvas: it contains underlying information that the word processor reads and applies across subsequent document operations. A heading can be used for navigation or table-of-contents generation, a list can maintain continuous numbering as items are added or removed, and a field can be dynamically updated. Therefore, the central question is whether the requested operation takes effect in the document in a native form that Word can recognize and apply.
Preprint.

Figure 1. Overview of WordEval. Atomic tasks test individual Word-editing operations; Compositional tasks combine linked operations across 15 feature areas. The evaluator checks the resulting document’s Word-native objects and properties against the reference document to determine whether the requested edits were realized.
Current evaluations of document agents, however, do not isolate this ability well. Existing benchmarks largely evaluate document generation, file I/O, information extraction, or GUI task completion [3, 22, 24, 25, 27]. They show that an agent can access a document and produce an output file, but not whether the effect of each Word-editing operation was faithfully realized. To address this gap, we introduce WordEval, a benchmark that evaluates how faithfully a Word agent realizes a user’s editing intent as a change in document state. We call this ability operation fidelity: whether the intended effect of an editing operation is realized in the resulting document as a state change with the correct target, attribute, and parameters. WordEval evaluates the output document itself, independent of the action path (e.g., GUI clicks, ribbon commands, or API calls) used to produce it.
WordEval organizes evaluation into Atomic and Compositional tracks. Atomic cases isolate whether the intended effect of a single editing operation is realized. Compositional cases examine whether the agent can manage dependencies, ordering, interference, and preservation when multiple edits are issued together. This design lets us separately analyze the ability to perform individual Word-editing operations and to integrate multiple edits into a coherent document state.
Contributions. (i) A Word-native operation-fidelity benchmark. WordEval comprises 642 cases spanning 15 Word feature areas, each generated from an editing-unit specification with an aligned Word-native oracle, with an Atomic track of 546 cases over 231 Word commands and a Compositional track of 96 cases from 27 composition scenarios.
(ii) Evaluation revealing a compositional gap. Across eight agent systems, many still struggle on the Atomic track, with system-specific Atomic-to-Compositional gaps.
(iii) Fine-grained failure diagnosis. Atomic-intent links, which connect each editing unit of a Compositional case to the Atomic case with the same command, trace compositional failures to individual operations, and a failure-mode analysis characterizes how incorrect outputs differ from the reference.
Related Work
Document editing in office benchmarks. Recent benchmarks for office-workflow agents span desktop and office settings. OfficeBench [24] and OdysseyBench [22] evaluate multi-application office workflows through file-level I/O and conversion utilities, while GUI benchmarks such as OSWorld [25], WindowsAgentArena [3], and WorldGUI [27] evaluate desktop automation with word processors such as Microsoft Word or LibreOffice Writer appearing as one target application. These benchmarks show that agents can reach and manipulate document files, but they do not provide a controlled measurement of document-editing capability organized around the space of editing operations. WordEval targets this missing layer by evaluating structured edits in Microsoft Word under controlled specifications drawn from a command-level taxonomy.
Figure 2. An illustrative example of Word-native operation-fidelity evaluation. In this example, the document agent inserts bullet characters without creating a list in Word, yielding an output visually similar to the reference. WordEval detects this property mismatch.
Verification in office and GUI benchmarks. Existing protocols typically verify returned text, file contents, saved-file state, task completion, or visual appearance: OfficeBench [24] and OdysseyBench [22] use exact-match, containment, keyword, and file-existence checks; OSWorld [25] and WindowsAgentArena [3] apply plain-text matching and per-property checks to saved files; and WorldGUI [27] evaluates screenshot-level GUI task success. These evaluators are effective for checking task completion, but they do not consistently verify the target Word-native object state across commands. For example, a numbered list can be approximated by typing plain-text markers such as 1. and 2., matching the rendered appearance without behaving as a Word list under subsequent renumbering or level changes. WordEval instead evaluates the resolved document state, checking that the specified change holds on the target object and that surrounding content and structure are preserved. Table 1 summarizes this comparison.
| Benchmark | Interface | Word tasks | Edit operations | Verification oracle |
| OfficeBench [24] | python-docx API | 4 file actions | keyword, file existence | |
| OdysseyBench [22] | python-docx API | 4 file actions [24] | keyword, file existence | |
| OSWorld [25] | LibreOffice GUI | 23 | 23 property edits | text match, property checks |
| WindowsAgentArena [3] | LibreOffice GUI | 19 | 18 property edits [25] | text match, property checks |
| WorldGUI [27] | Word GUI | 11 meta-tasks | screenshot task success | |
| WordEval (ours) | Unrestricted | 642 | 231 commands | Word object properties, preservation |
Table 1. Comparison with prior benchmarks containing word-processing tasks. Interface is the route through which an agent edits the document. WordEval agents choose their own route, such as python-docx, VBA, or the Word GUI. *Estimated from Word-document subtasks. †Variants of 11 meta-tasks.
WordEval
Problem Formulation
The Word object model. Word represents a document as a structured object graph, which we call the Word object model. In this model, Word exposes runtime objects such as Shape, Style, Font, Paragraph, Field, and Comment, together with properties including Shape.Rotation, Style.NameLocal, and Font.Bold. These runtime objects and their property values are resolved when Word opens and interprets a document.
Operation fidelity. WordEval evaluates Word-native operation fidelity, an agent’s ability to realize a requested edit as the intended changes to objects and properties in the Word object model. We formalize each evaluation case as a tuple , where is the input document, is a natural-language instruction, and is a reference document representing the expected post-edit state. Given , the agent produces an edited document . We evaluate using the Word-native oracle described in § 3.4, which verifies the required target changes and preservation properties. This evaluation is independent of the action sequence and specific Word commands used to produce .
Benchmark Structure and Coverage
Editing unit as case specification. We represent each edit as an editing unit defined by four components: a precondition that makes the edit executable, a target span or runtime object, an operation, and the parameter values for that operation. The precondition specifies the document state required for the operation to have a determinate, verifiable effect, such as an existing target object, an applicable selection, or a structural context. The operation is represented by a normalized Word command, whereas the target identifies the span or runtime object to which that command applies.
Operation taxonomy. The operation taxonomy organizes the normalized Word commands populating the operation component of each editing unit, associating each command with a generation route and verification schema. Built by consolidating redundant Word ribbon metadata4 into distinct semantic commands (Appendix § 9.1), each entry follows a three-tier hierarchy: behavior family (generation logic), target Word component (affected state), and verification schema (object-property oracle comparisons). This structure enables systematic case generation with aligned oracle specifications (§ 3.4). We exclude commands that are purely UI-structural, service-dependent, lack deterministic and verifiable effects, or fall outside our scope. Among the remaining commands, we instantiate those whose preconditions can be constructed, operations automated, and postconditions verified. This yields 231 released commands; Appendix § 9.1.§ 1 details the coverage funnel and exclusion criteria.
Word feature areas. Whereas the operation taxonomy organizes what edit is performed, Word feature areas organize where that edit acts in the Word object graph. We assign each command to a feature area by its target runtime object. The released commands span 15 Word feature areas, from Text, Paragraphs, and Tables to Charts, SmartArt, and Page Structure (Table 2).
| Category | Cases | Coverage | Definition |
| WordEval | 642 | 231 commands; 27 scenarios | Benchmark across 15 Word feature areas with task-specific object-state checks. |
| Atomic | 546 | 231 commands | Single editing unit over one Word command. |
| Feature areas | 546 | 15 feature areas (Text, Paragraphs, Lists, Tables, Shape geometry, Shape appearance, Shape text, Pictures, Graphics, Charts, SmartArt, Equations, Page Structure, Annotations, References) | Object-model coverage axis; feature areas group runtime objects, which expose checked object-property pairs. |
| Compositional | 96 | 27 scenarios | Multiple editing units in the same controlled document substrate. |
| Additive | 46 | 11 scenarios: font, paragraph, list, document enrichment/cleanup, chart, layout, picture, shape, SmartArt, table | Unit preconditions are jointly satisfied and edits do not invalidate one another. |
| Interaction-sensitive | 50 | 16 scenarios | Earlier edits affect later preconditions, target scope, or layout state. |
| Prerequisite-chain | 25 | 7 scenarios: tables, charts, icons, shapes, SmartArt, equations, text boxes | Later unit requires an object or state created by an earlier unit. |
| Scope-changing | 12 | 4 scenarios: tables, SmartArt, equations, lists | Earlier unit changes the target set or structural scope used by later units. |
| Layout-coupled | 13 | 5 scenarios: grouped shapes, shape rotation/alignment, shape position/alignment, shape multi-step geometry, text wrapping/positioning | Later geometry or layout checks depend on earlier wrapping, alignment, or positioning state. |
Table 2. Structure and coverage of WordEval, including Atomic and Compositional case counts, Word feature areas, and composition classes.
Evaluation tracks. To distinguish fidelity on individual edits from the additional burden of composition, we organize cases into two tracks, Atomic and Compositional. The Atomic track contains cases with a single editing unit, whereas compositional cases combine multiple units within the same controlled document substrate. Both tracks use the same operation pool and the same Word-native evaluation criterion, allowing individual and combined edits to be compared under one standard. We group compositional cases into two categories by how their editing units interact, additive and interaction-sensitive. In additive cases, unit preconditions are jointly satisfied in , no unit invalidates another, and the units are order-independent. In interaction-sensitive cases, earlier edits affect later ones, and three classes specify the affected state, namely preconditions in prerequisite-chain cases, target scope in scope-changing cases, and layout state in layout-coupled cases. The additive category and these three classes form the four composition classes used in our analyses. Appendix 16 and 17 list representative released instructions. For example, an atomic case asks the agent to make the word “Target” bold, while a prerequisite-chain case asks it to insert a clustered column chart below the “Quarterly Sales:” heading and then add outside-end data labels.
Table 2 summarizes the resulting benchmark structure, case counts, and coverage over Word feature areas and composition classes. Appendix § 9.2.§ 1 gives the object-property coverage, and § 4.3 reports capability stratified by feature area.
Case Construction Pipeline
Generation pipeline. Figure 3 shows the end-to-end pipeline. Given a planned editing unit and its schema, an Input Builder constructs a controlled input document in which the unit’s precondition holds, materializing the required document state explicitly rather than sampling it from natural documents so that the input contains only the structure required by the operation under test. A Reference Executor then runs the planned operation on under Word’s native automation, dispatching to the execution route appropriate for the command (one-shot ribbon execution, parameter selection, or dialog automation), producing the reference document , and exporting the oracle specification defined by the schema. An oracle specification lists the checks applied to one case, covering target postconditions for the requested edit and preservation checks for state the edit should leave unchanged. In parallel, an Instruction Renderer produces the natural-language instruction from the same unit’s targets, operation, and parameters under a fixed authoring protocol: each unit names a single operation, references a concrete target visible in , and avoids transient or invisible state such as selection or cursor focus. Together these stages yield a complete case with its oracle specification attached. The generator also enumerates parameter variants through Word COM automation, so scores are not tied to a single parameter setting; Appendix § 9.1.§ 3 details the variant-generation procedure. The pipeline drives both case construction and oracle evaluation through Word’s native automation, a symmetric COM-API design that makes the target postconditions executable. Properties manipulated during reference construction can be read back during verification, which avoids inferring these postconditions from text, screenshots, or LLM judgments.
Figure 3. Case generation pipeline. A shared case source specifies the editing unit or unit sequence, preconditions, target objects, parameters, and verification schema. The generator uses this source to materialize the input document, instruction, reference document, metadata, and oracle specification.
Compositional case construction. We derive compositional cases from the same document substrates and operations as atomic cases and add dependencies across edits, keeping the operations fixed while varying how they are combined. For atomic cases, the unit’s precondition is satisfied by alone. For compositional cases, the precondition extends across the sequence and is distributed according to the composition class. In additive compositions, is constructed so that all unit preconditions are jointly satisfied and no unit invalidates another’s precondition. In interaction-sensitive compositions, satisfies only the initial precondition, and later preconditions are satisfied only by states produced by earlier units, with the specific carry-over pattern determined by the interaction class (prerequisite-chain, scope-changing, or layout-coupled).
Atomic-intent links. An atomic intent is one editing unit of a compositional case, linked to the Atomic case that applies the same command. In the chart case of § 3.2, inserting the chart and adding outside-end data labels are two atomic intents, linked to the ChartInsert and ChartDataLabel Atomic cases. § 4.4 uses these links to examine compositional cases for which the same system passes every linked Atomic case.
Quality control. Before release, the oracle applies each case’s specification to the reference document itself, and every check must pass. All 642 reference documents pass these checks. Cases that fail this self-check, produce inconsistent metadata, or violate the authoring protocol for are regenerated or discarded. This step checks consistency across , , , and the oracle; released cases are also reviewed by human annotators for instruction clarity, input validity, and reference correctness (Appendix § 9.1.§ 4).
Word-Native Oracle
Each case defines a set of property comparisons, implemented as our oracle. A comparison reads a value from the agent output and checks it against a reference value obtained from , , or case metadata, and a case passes only if all required comparisons pass. The oracle reads these values through Word’s object model; for some commands, it also checks the saved Office Open XML package to confirm that the requested state persists in the saved file. The oracle verifies task-specific object properties and preservation conditions across 15 Word feature areas. Appendix § 9.2 details coverage, schemas, and human validation. Appendix Table 5 summarizes the oracle’s object–property coverage across Word feature areas; each case’s oracle specification defines the applicable checks. These comparisons cover object attributes, structure counts, inserted content, reference positions, and preservation properties. Each schema pairs its target postcondition with invariants for non-target state that the operation should leave unchanged, such as surrounding text, paragraph or object counts, section structure, table dimensions, or field counts when those properties are relevant. This prevents narrow target-only passes while keeping preservation schema-scoped rather than a global whole-document equivalence rule. For compositional cases, the set combines unit-level checks for individual units with case-level checks for cross-unit dependencies. Additive compositions require no case-level checks, whereas interaction-sensitive compositions include checks specific to the interaction class. Comparisons use behavior-specific extractors and tolerances, including numerical tolerances for position-related properties. When a document opens in Word, each failed check reports the compared property with its reference and output values, such as a footnote number format of lowercase letters in the reference and Arabic numerals in the output. These reports ground the failure-mode analysis in § 4.5.
Experiments and Analysis
Experimental Setup
Evaluated systems. We evaluate eight agent systems: Claude Code [1] with Opus 5 [2]; Codex [17] with GPT-5.6 SOL [18] (high); Kimi Code [16] with K3 [10] (high); Mini Agent [14] with Mini-Max M3 [15]; Qwen Code [21] with Qwen3-8-27B [20]; OpenCode [19] with Gemini 3.8 Flash [9] (high); ZCode [26] with GLM-5.2 [8] (high); and OpenCode [19] with DeepSeek V4 Pro [4]. Each system is evaluated on 642 cases: 546 Atomic and 96 Compositional. Three human participants perform the same task set.
Metrics. We report two Word Object pass-rate metrics. The case-level pass rate weights every case equally. The macro pass rate is the unweighted mean over atomic commands or composition scenarios, each scored by its variant mean. Each command or composition scenario thus contributes equally; feature-area breakdowns report macro rates unless otherwise stated. Missing outputs, including agent runs that end without a document, and invalid outputs count as non-passing cases (§ 4.5).
Execution environment. Benchmark generation, reference construction, and Word Object verification run on Windows with Microsoft Word through COM automation; these stages require no GPU. The reported run used Windows 11 virtual machines with Word 16.0 and 8GB RAM. Agent-side inference is provided by each agent system and may use external model-serving infrastructure.
Artifact and reproducibility. WordEval is a self-contained release whose evaluator needs no agent system; Appendix § 10.§ 4 details its contents, licenses, data card, and environment requirements.
Main Results
Table 3 reports the performance of eight agent systems on Atomic and Compositional tasks. Claude Code with Opus 5 achieves the highest Overall Case pass rate at 87.1%, while the remaining systems score between 44.5% and 66.7%. The three human participants achieve a mean Overall Case pass rate of 82.8% on the same tasks. Individual participant scores and the human execution protocol are reported in Table 9 and Appendix § 10.
| Participant | Atomic | Atomic | Compositional | Compositional | Overall | Overall |
| Case | Macro | Case | Macro | Case | Macro | |
| Human A01 | 83.0 | 85.4 | 71.9 | 69.6 | 81.3 | 83.7 |
| Human A02 | 90.1 | 90.9 | 78.1 | 76.8 | 88.3 | 89.4 |
| Human A03 | 82.4 | 83.9 | 58.3 | 58.8 | 78.8 | 81.3 |
Table 9. Human execution on the same 642 tasks per participant. Case denominators are 546 Atomic, 96 Compositional and 642 Overall; Macro uses 231, 27 and 258 fixed-variant groups. The main-table Human row averages the three participant-specific rates.
Across the two tracks, all eight systems have lower Compositional than Atomic Case pass rates, but the gap varies from 3.2 to 19.1 percentage points. Opus 5 maintains high performance on both tracks, with pass rates of 87.5% and 84.4%, respectively. However, systems with similar performance on individual edits do not necessarily perform similarly on combined requests. Gemini 3.8 Flash and GLM-5.2 achieve Atomic Case pass rates of 57.9% and 58.2%, respectively, but their Compositional Case pass rates are 49.0% and 39.6%.
These results show that systems with similar aggregate performance on individual edits can differ in their ability to complete combined requests. We next examine performance across Word feature areas (§ 4.3), then compare exactly linked Atomic and Compositional cases to assess whether success on individual operations is maintained when they are combined (§ 4.4).
Atomic Capability Across Word Feature Areas
We analyze Atomic capability across the 15 Word feature areas introduced in § 3.2. Figure 4 shows macro-averaged pass rates under the Word Object oracle for each feature area and system. Tables and Paragraphs achieve the highest mean pass rates, at 83.2% and 81.1%, respectively, with strong performance on cell merging, row-height distribution, paragraph alignment, and indentation.
Figure 4. Atomic macro-averaged pass rates (%) across fifteen Word feature areas for eight agent systems. Each score averages command-level pass rates within the feature area, with missing predictions counted as failures. Columns are ordered by the mean across systems.
In contrast, Pictures has the lowest mean pass rate at 44.9%, followed by SmartArt (50.7%) and Page Structure (53.8%). These areas include image cropping, picture effects such as glow, bevel, and shadow, SmartArt node and formatting edits, and header/footer configuration. These results indicate that image and diagram editing and page-level configuration pose greater challenges than table and paragraph formatting in the evaluated Atomic tasks. We next examine whether success on individual operations carries over to compositional tasks that combine them.
Atomic-to-Compositional Case Analysis
We examine how success on individual Word operations relates to success when those operations are combined in one document. All 96 Compositional cases carry the atomic-intent links defined in § 3.3, covering 319 constituent intents per system. Let denote a pass verdict on a Compositional case, and let denote the event that the same system passes all linked Atomic variants in isolation. Figure 5 summarises the case pass rate with the conditional rate . The condition selects a system-specific subset; Appendix Table 11 reports its size for each system and class.
Figure 5. Compositional case pass rates by class for five selected systems. (A) Case pass rates over all cases in each class. (B) Conditional case pass rates restricted to cases for which the same system passes all linked Atomic variants. Rates with fewer than five such cases are omitted. Appendix Table 11 reports the denominators for all eight systems.
Pass rates by composition class. Figure 5A shows case pass rates across the four composition classes for five systems. Scope-changing tasks, in which earlier edits alter the targets of subsequent operations, have the highest pass rates for each system shown (58.3–100%). Layout-coupled tasks, in which edits interact through document layout, show the largest variation across systems (23.1– 84.6%). We examine these class-level differences further by conditioning on successful execution of the constituent Atomic tasks.
Conditioning on atomic-intent success. Figure 5B reports , restricting the analysis to cases whose linked Atomic variants all pass in isolation. For additive tasks, conditional pass rates are higher than raw rates, but the difference varies across systems: +41.6 percentage points for K3 and +2.3 points for GPT-5.6. The relationship is less uniform in other classes. On layout-coupled tasks, GPT-5.6 has a conditional pass rate of 66.7%, compared with a raw rate of 76.9%, while GLM-5.2 passes only two of its eight eligible cases (25.0%). Thus, successful execution of all constituent operations in isolation does not ensure completion of the combined request. As in the main results, missing outputs count as failures.
Appendix § 10.3.§ 1 complements this case-level view with intent-level outcomes. For each intent whose Atomic case passes, we check whether the same intent also passes within the combined request, using intents whose embedded verdict is observed. This transfer is lowest in layout-coupled tasks. These results suggest that operations that succeed in isolation are hardest to sustain when edits interact through document layout.
Failure Mode Analysis
Output status. Each non-passing case is assigned to one of three outcomes (Figure 6A). Missing output means that no document was submitted, invalid output means that Word cannot open the document or it fails OOXML validation, and incorrect output means that a valid document fails a requested postcondition or preservation check. For Opus 5, incorrect outputs account for 59 of its 83 non-passing cases, and no output is invalid. For GPT-5.6, invalid and missing outputs together account for 18.4% of all cases, while missing outputs alone account for 41.7% for Gemini 3.8 Flash. These systems often fail before the requested edit can be evaluated.
Figure 6. Failure analysis under Word Object. (A) Output status of non-passing cases over 642 cases per system. (B) Mismatch types of the 59 incorrect outputs of Opus 5 (bold in A), one type per case.
Mismatch types of incorrect outputs. We inspected the evaluator diagnostics and saved OOXML of all 59 incorrect outputs of Opus 5 and assigned each case to one of six types, namely property-value mismatch, structure mismatch, missing element, off-target change, content mismatch, or unrequested change (Figure 6B). Property-value mismatches are the largest type, accounting for 39.0% of the cases. Of the rest, 33.9% are structure mismatches, 11.9% missing elements, 6.8% off-target changes, 6.8% content mismatches, and 1.7% unrequested changes. In property-value mismatches, the requested object is present with a different setting, such as a shadow preset or a footnote numbering format. In picture-effect cases among structure mismatches, Word’s object model reports the requested effect, but the saved OOXML lacks the image layer that Word writes for it. Most missing elements are equation delimiters or matrices inserted with empty placeholders. In the SmartArt text-fill task, Opus 5 fills the shape instead of its text, and in and it inserts an invisible function-application character (U+2061) that leaves the rendering unchanged. These outcomes show that most incorrect outputs differ from the reference in settings or Word structure rather than omitting the requested element.
Conclusion
We introduced WordEval, a benchmark that evaluates document-editing agents by the runtime object properties and preserved document state that their edits produce in Word. Its Atomic and Compositional tracks span the execution of individual operations and the coordination of multiple edits within a shared document state. Across eight agent systems, many still struggle on Atomic tasks, performance varies across Word feature areas, and even the strongest system scores lower on Compositional tasks. The atomic-intent and failure-mode analyses further expose operation-level limitations, composition-specific failures, and artifact-level issues. Together, these results suggest that operation fidelity and compositional reliability are distinct requirements for document-editing agents.
Limitations
Synthetic controlled documents. WordEval is a controlled operation-fidelity benchmark, not a natural-document distribution or end-to-end office-workflow benchmark. Its documents isolate Word-native edits under known preconditions, so results should not be read as measuring open-ended writing quality, user productivity, or arbitrary real-world document performance.
Coverage and portability. The current benchmark uses English, protocol-guided instructions; multilingual instructions and user-specific document conventions are outside the current evaluation scope. The evaluator requires Windows and Microsoft Word because WordEval checks Word-resolved runtime state, including lists, fields, layout-dependent properties, and object effects, rather than only the saved .docx file structure.
Coverage-oriented, not frequency-weighted. The released set is a coverage-oriented instantiation of the in-scope verifiable operation space rather than a frequency-weighted sample of real user workloads. Scores should therefore be read as capability-space coverage rather than expected accuracy on natural document-editing workloads.
Broader Impacts
WordEval enables more rigorous evaluation of document-editing agents by exposing failures that surface-text, screenshot, or coarse task-completion metrics miss, helping developers diagnose agents before deployment. It is a capability benchmark rather than a deployment-safety guarantee, and high scores on controlled tasks do not imply safe behavior on out-of-distribution documents, long-horizon workflows, or adversarial inputs.
Acknowledgments
This research was supported by the MSIT (Ministry of Science, ICT), Korea, under the Global Research Support Program in the Digital Field program (RS-2024-0436680) supervised by the IITP (Institute for Information & Communications Technology Planning & Evaluation). This project is supported by Microsoft Research Asia.
Supplementary Material
The supplement follows the benchmark from construction and verification to execution and additional analysis. Task examples and the complete command catalogue are collected at the end for reference.
Benchmark Construction and Coverage
Command Selection and Scope
Table 4 summarizes the command-selection stages used to construct the released Atomic set from Word’s ribbon controls. We start from Word ribbon controls [12] and consolidate command-bearing entries into normalized semantic commands. We then identify commands whose effects are state-changing and verifiable in Word-native state, yielding 1,150 verifiable commands across the three behavior families. Within this verifiable pool, we define the current benchmark scope around document-editing operations; cross-application objects, form design, mail merge, macro/VBA authoring, multi-document workflows, automatic rewriting, and proofing are catalogued separately, yielding 992 in-scope, benchmark-eligible commands. We select the 231 released commands from this pool to cover all three behavior families and all 15 Word feature areas. The remaining in-scope commands define an extension space for future releases.
| Stage | Count | Reduction |
| Ribbon control entries | 5,638 | |
| Normalized commands | 2,569 | drop 202 entries without a command identifier; deduplicate repeated placements |
| Verifiable state-changing commands | 1,150 | exclude view, settings, clipboard, external, UI-structural, and selection commands (1,419; categories below) |
| In-scope commands | 992 | remove 158 excluded-by-design commands (OLE, forms, mail merge, macro/VBA, master documents, AutoCorrect/AutoFormat, proofing) |
| Released atomic commands | 231 | selected set spanning 15 feature areas; remaining in-scope commands define an extension space |
Table 4. Command selection from Word ribbon controls to the released Atomic command set.
Outside the verifiable state-changing pool. These commands either affect state not stored in .docx or lack deterministic Word-native postconditions.
View-state commands. Zoom, ruler, navigation pane, gridlines, page color preview.
Document-settings commands. Protect Document, Undo/Redo.
Clipboard operations. Copy, Cut, Paste, Format Painter.
External-service commands. Dictate, Translate, Copilot, Insert from Online, Smart Lookup.
UI-structural controls. Menu openers, group labels, tab headers, dialog launchers without a state-changing action.
Outside the in-scope command set.
Embedded OLE / ActiveX objects (cross-application state).
Form controls (Word as form-design tool, not document editor).
Mail merge (multi-document workflow).
Macro / VBA authoring (program editing, not document editing).
Multi-document workflows (split, master/sub-documents).
AutoCorrect / AutoFormat (automatic rewriting outside an explicit command).
Proofing (spelling and grammar checking).
Released-Command Selection
Starting from the 992 in-scope commands, we include a command in the released Atomic set when it satisfies three construction requirements: the generator can construct an input document satisfying its precondition, Word automation can execute the command, and the resulting state is verifiable through Word-native postconditions. Execution uses direct invocations for button commands, explicit parameter values for gallery commands, and documented object-model calls for dialog commands. This yields the 231 released commands, spanning all 15 feature areas. Table 18 lists the released commands by feature area, using Office Fluent UI command identifiers (idMso).
| Word feature area | Released commands |
| Text | Bold, ChangeCaseGallery, CharacterBorder, CharacterShading, ClearFormatting, Font, FontColor Picker, FontSize, FontSizeDecreaseWord, FontSizeIncreaseWord, Italic, LigatureGalleryWord, ShadingColorPicker, Strikethrough, Subscript, Superscript, SymbolInsertGallery, TextFillGradient Gallery, TextHighlightColorPicker, TextOutlineColorPickerAlternate, TextOutlineDashesGallery, TextOutlineWeightGallery, TextReflectionGallery, UnderlineColorPicker, UnderlineGallery |
| Paragraphs | AlignCenter, AlignJustify, AlignLeft, AlignRight, BordersSelectionGallery, DropCapInsertGallery, IndentDecreaseWord, IndentIncreaseWord, LineSpacingGallery, ParagraphSpacingAfter, ParagraphSpacingBefore, QuickStylesGallery, TableOfContentsAddTextGallery |
| Lists | BulletsGalleryWord, ListLevelGallery, MultilevelListGallery, NumberingGalleryWord |
| Tables | BorderStylesGallery, MergeCells, QuickTablesInsertGallery, SortDialogClassic, SplitCells, Table AutoFitContents, TableAutoFitWindow, TableCellAlignBottomCenter, TableCellAlignBottomLeft, TableCellAlignBottomRight, TableCellAlignMiddleCenter, TableCellAlignMiddleLeft, TableCell AlignMiddleRight, TableCellAlignTopCenter, TableCellAlignTopLeft, TableCellAlignTopRight, TableColumnsDelete, TableColumnsDistribute, TableColumnsInsertLeft, TableColumnsInsert Right, TableDelete, TableInsertGallery, TableRowsDelete, TableRowsDistribute, TableRowsInsert AboveWord, TableRowsInsertBelowWord, TableSplitTable, TableStyleClear, TableStylesGallery Word |
Table 18. Released atomic commands by Word feature area.
Parameter Variants
A single case per command would conflate command-level ability with familiarity with one particular parameter value. To ensure that scores reflect general command execution rather than memorization of specific parameter settings, the generator creates variants over the parameter space of each command. For parameterized commands such as galleries, dropdowns, and dialogs, the generator enumerates the available options through Word’s object model and creates cases that span up to eight options per command. Toggle commands are tested in both their on and off states, and other commands typically have one case. The same Reference Executor produces and exports the corresponding oracle specification for each variant, so adding variants requires no manual document authoring or labeling.
Human Review of Benchmark Cases
Three annotators experienced with Microsoft Word independently reviewed every released benchmark case for instruction clarity, input-document validity, and reference-document correctness. Cases flagged during review were manually adjudicated, and affected instructions and reference documents were corrected and revalidated before release.
Word-Native Verification and Validation
The oracle checks the requested edit and the preservation conditions specified for each case. This section details the object–property coverage and verification schemas, then compares evaluator precision, false-positive rates, and false-negative rates.
Object–Property Coverage
Word exposes document objects and their properties through its object model [13]; Office Open XML specifies the saved document format [5]. The following table summarizes the Word objects and properties covered by the oracle, organized by Word feature area. Each case’s oracle specification defines the target postconditions and preservation checks applied to that case.
| Feature area | Runtime object | Object properties |
| Text | Border | Color, LineStyle, LineWidth |
| Text | ColorFormat | RGB |
| Text | FillFormat | GradientAngle, Type |
| Text | Font | Bold, Color, Italic, Ligatures, Name, Size, StrikeThrough, Subscript, Superscript, Underline, UnderlineColor |
| Text | LineFormat | DashStyle, Weight |
| Text | Range | HighlightColorIndex, Text |
| Text | ReflectionFormat | Type |
| Text | Shading | BackgroundPatternColor, Texture |
| Paragraphs | Border | LineStyle |
| Paragraphs | DropCap | LinesToDrop, Position |
| Paragraphs | Paragraph | OutlineLevel |
| Paragraphs | ParagraphFormat | Alignment, LeftIndent, LineSpacing, LineSpacingRule, SpaceAfter, SpaceBefore |
| Paragraphs | Style | NameLocal |
| Lists | ListFormat | ListLevelNumber, ListString |
| Tables | Border | LineStyle |
| Tables | Cell | VerticalAlignment |
| Tables | Column | Width |
| Tables | Columns | Count |
Table 5. Object–property coverage of the Word-native oracle, organized by Word feature area.
| Feature area | Runtime object | Object properties |
| Paragraph | Alignment | |
| Range | Text | |
| Row | Height | |
| Rows | Count | |
| Style | NameLocal | |
| Table | AutoFitBehavior | |
| Tables | Count | |
| Shape geometry | GroupShapes | Count |
| Shape | AutoShapeType, Height, HorizontalFlip, Left, RelativeHorizontalPosition, RelativeVerticalPosition, Rotation, Top, Type, VerticalFlip, Width, ZOrderPosition | |
| Shapes | Count | |
| WrapFormat | Type | |
| Shape appearance | ColorFormat | RGB |
| FillFormat | GradientStops, Type | |
| GlowFormat | Radius | |
| ReflectionFormat | Type | |
| ShadowFormat | Type | |
| Shape | ShapeStyle | |
| SoftEdgeFormat | Type | |
| ThreeDFormat | BevelTopDepth, BevelTopType, PresetCamera, RotationX, RotationY | |
| Shape text | Paragraph | Alignment |
| Range | Orientation, Text | |
| TextFrame | Orientation | |
| TextFrame2 | Orientation | |
| Pictures | ColorFormat | RGB |
| GlowFormat | Radius | |
| InlineShape | Height, Type, Width | |
| InlineShapes | Count | |
| PictureFormat | Brightness, ColorType, Contrast, CropBottom, CropLeft, CropRight, CropTop | |
| ReflectionFormat | Type | |
| ShadowFormat | Type | |
| SoftEdgeFormat | Type | |
| ThreeDFormat | BevelTopDepth, BevelTopType, PresetCamera, RotationX, RotationY | |
| Graphics | ColorFormat | RGB |
| InlineShape | Height, Width | |
| InlineShapes | Count | |
| Charts | Axis | MajorGridlines, MinorGridlines |
| Chart | ChartColor, ChartGroups, ChartStyle, ChartType, HasDataTable, HasLegend, HasTitle | |
| ChartTitle | Text | |
| ColorFormat | RGB | |
| DataLabels | Position | |
| DataTable | ShowLegendKey | |
| Legend | Position | |
| Series | Format, HasDataLabels, Name, Values | |
| SeriesCollection | Count | |
| Trendline | Type | |
| SmartArt | ColorFormat | RGB |
| LineFormat | BeginArrowheadStyle, DashStyle, EndArrowheadStyle, Weight | |
| Shape | Height, Width | |
| SmartArt | Reverse | |
| SmartArtColor | Name | |
| SmartArtLayout | Name | |
| SmartArtNode | Level | |
| SmartArtNodes | Count | |
| SmartArtQuickStyle | Name | |
| TextFrame2 | WarpFormat | |
| TextRange2 | Text | |
| ThreeDFormat | BevelTopType, PresetCamera, RotationX | |
| Equations | OMaths | Count |
Table 1 (PDF p. 16).
| Feature area | Runtime object | Object properties |
| Range | Text, WordOpenXML | |
| Page Structure | ContentControl | Tag, Title |
| ContentControls | Count | |
| Document | ComputeStatistics, Content | |
| ParagraphFormat | Alignment | |
| Range | Text | |
| Shapes | Count | |
| Style | NameLocal | |
| Annotations | Comment | Author, Done, Index |
| Comments | Count | |
| Document | TrackRevisions | |
| Endnotes | Count, NumberStyle | |
| Footnotes | Count, NumberingRule, NumberStyle | |
| Range | Start, Text | |
| Revisions | Count | |
| References | Bibliography | BibliographyStyle |
| ContentControl | Title | |
| ContentControls | Count | |
| Hyperlink | Address, TextToDisplay | |
| Hyperlinks | Count | |
| Indexes | Count | |
| Range | Text | |
| Sources | Count | |
| TableOfContents | IncludePageNumbers | |
| TablesOfContents | Count |
Table 5 (PDF p. 17).
Target Postconditions and Preservation
| Verification schema | Target postcondition | Preservation invariants |
| Run / character formatting | Range.Font.* (weight, style, size, color, highlight, text effects) on the target range | Non-target (“control”) text content and formatting; paragraph count |
| Paragraph formatting | ParagraphFormat.* (alignment, indents, line spacing, space before/after, borders) | Paragraph text; non-target paragraph properties; paragraph count |
| Lists | ListFormat (list string, list level) on the target paragraphs | Paragraph text; non-target list state; paragraph count |
| Shape / picture geometry & effects | Shape.{Left, Top, Width, Height, Rotation, ZOrderPosition}, wrap type, fill / line / shadow / glow / reflection | Document structure counts (shape, inline-shape, footnote, section); non-target shapes |
| Tables | Row and column counts, cell content, borders, style | Table dimensions; non-target cells |
| Fields & references (footnote, endnote, TOC, citation) | Field count, resolved content, reference position | Body text when the field is the sole mutation |
| Headers / footers | Header/footer story content, field count | Document body text; section count |
Table 6. Representative verification schemas, target postconditions, and preservation categories from the benchmark design. Each case applies the checks defined by its oracle specification; not every listed invariant applies to every case.
Evaluator Validation
Positive control. We submit the reference document of each of the 303 cases in place of the agent output. Word Object accepts all 303.
Human reference. We compare evaluator judgments with the majority of three human evaluators on 303 agent-generated Word documents. Each human evaluator judged the agent output against the instruction by opening it alongside the input document in Word, without seeing any evaluator verdict, and could open the reference document on request. We exclude 41 documents that Word cannot open or that fail XML or package-structure checks, and four documents with no binary majority. The comparison set contains 258 documents, 167 labeled Pass and 91 labeled Fail. The box below reproduces the instructions given to the human evaluators.
Human evaluator instructions
# WordEval - Human Judging Package
You will judge whether an AI model correctly edited a Word document.
Your labels help validate WordEval’s automated checking against human
judgment.
For each case you see an instruction and download two documents - the
Original (before the edit) and a Candidate (the model’s attempt). You
open both in Word and decide, independently: did the Candidate
correctly carry out the instruction? You are never shown any automated
verdict - your own judgment is the point.
Some instructions bundle several sub-edits. Those show a checklist: you
judge each part separately, then give an overall verdict.
[... setup steps omitted ...]
## What You Need
- Windows with Microsoft Word - you judge the real .docx in Word, not a
picture. A flat image hides differences that matter (a real footnote
vs a typed superscript, a real field vs typed text, styles, page
breaks vs blank lines).
## How to Judge a Case
1. Read the instruction shown on screen.
2. Download both documents and open them in Microsoft Word.
3. Compare before -> after. Helpful Word tools: Home -> Show/Hide
(show marks), Reveal Formatting (Shift+F1).
4. Answer:
- Does the Candidate correctly perform the instruction? - Pass /
Fail / Unsure
- How confident are you? - High / Medium / Low
- Notes (optional, but please add them when unsure or when
something is subtle).
5. Click Submit & next. [...]
How to decide:
- Judge against the instruction, not against one specific output - a
different but valid way still passes. There is no single required
result.
- Missing the required change, wrong target, wrong element type, or
destroying other content -> fail.
- Pick unsure only when you genuinely cannot tell even after opening
both documents in Word.
### Multi-part cases
When the instruction bundles several sub-edits, judge each checklist
row on its own merits: did that sub-edit happen correctly? Then give
the overall verdict separately - it is not automatically "all parts
pass": parts can interfere with each other, and unrelated content may
have been damaged.
### Stuck?
If you genuinely cannot tell what a correct result should look like
(e.g. an obscure Word feature), expand "Stuck? Reveal one correct
example" to download a Reference.docx - one example of a correct
output. Use it sparingly, and keep judging by the instruction: a
result that differs from the example can still pass. You will not
need it for most cases.
[... file-return steps omitted ...]
## Time
There is no time limit - take your time on each case. (A timer runs
silently for analysis; it is never shown and has no cap.) If a case
feels broken or impossible after a reasonable effort, mark it unsure
with a note and move on.
[... troubleshooting omitted ...]
Compared evaluators. Besides Word Object, we evaluate Word Revision [11], SSIM [23] between RGB renderings of the reference and submitted documents at thresholds of 0.99 and 0.999, and an agent judge. The agent judge runs GPT-6 Astra through Codex CLI with high reasoning effort and document-analysis tools. It sees the instruction, the input document, and the submitted document, but not the reference document, the human judgments, or Word Object results.
Metrics. Pass is the positive class. Precision is the fraction of accepted outputs that the human majority labels Pass. The false-positive rate (FPR) is the fraction of majority-Fail outputs accepted, and the false-negative rate (FNR) is the fraction of majority-Pass outputs rejected.
Results. Word Object accepts 5 of the 91 majority-Fail outputs and rejects 28 of the 167 majority-Pass outputs, for 96.5% precision, 5.5% FPR, and 16.8% FNR (Table 7). Word Revision reaches a similar precision of 97.1% but rejects 100 majority-Pass outputs, an FNR of 59.9%. In the comparison set, it reports every SmartArt output as one shape deletion and one shape insertion, whether or not the edit is correct. It therefore rejects all 13 majority-Pass SmartArt outputs. At SSIM ≥ 0.99, 41 of the 91 majority-Fail outputs pass, an FPR of 45.1%. Raising the threshold to 0.999 lowers the FPR to 7.7% but raises the FNR from 16.2% to 27.5%. Word Object is lower on both rates. In one header page-numbering task, the submitted document passes even the 0.999 threshold, yet both the human majority and Word Object reject it. Word Object reports a mismatch in the header style. The agent judge rejects fewer majority-Pass outputs than Word Object (FNR 12.0% versus 16.8%) but accepts 11 majority-Fail outputs, more than twice the FPR of Word Object (12.1% versus 5.5%).
| Evaluator | Precision (%) ↑ | FPR (%) ↓ | FNR (%) ↓ |
| Word Object | 96.5 | 5.5 | 16.8 |
| Word Revision | 97.1 | 2.2 | 59.9 |
| SSIM ≥ 0.99 | 77.3 | 45.1 | 16.2 |
| SSIM ≥ 0.999 | 94.5 | 7.7 | 27.5 |
| agent judge | 93.0 | 12.1 | 12.0 |
Table 7. Evaluator judgments against the original human majority on 258 valid documents (167 Pass, 91 Fail). Pass is the positive class; all values are percentages.
Evaluation Setup and Prompts
We evaluate the eight agent systems listed in Table 3.
| Agent system | Atomic | Atomic | Compositional | Compositional | Overall | Overall |
| Case | Macro | Case | Macro | Case | Macro | |
| Claude Code (Opus 5) | 87.5 | 89.2 | 84.4 | 85.5 | 87.1 | 88.8 |
| Codex (GPT-5.6, high) | 67.6 | 68.2 | 61.5 | 60.3 | 66.7 | 67.3 |
| Kimi Code (K3, high) | 63.4 | 68.1 | 53.1 | 55.8 | 61.8 | 66.8 |
| OpenCode (Gemini 3.8 Flash, high) | 57.9 | 63.2 | 49.0 | 51.3 | 56.5 | 61.9 |
| ZCode (GLM-5.2, high) | 58.2 | 63.7 | 39.6 | 40.4 | 55.5 | 61.2 |
| OpenCode (DeepSeek V4 Pro) | 54.9 | 61.7 | 40.6 | 40.3 | 52.8 | 59.5 |
| Qwen Code (Qwen3.8-27B) | 50.4 | 56.4 | 31.2 | 32.4 | 47.5 | 53.9 |
| Mini Agent (MiniMax M3) | 47.3 | 54.9 | 29.2 | 30.6 | 44.5 | 52.4 |
| Human (mean) | 85.2 | 86.7 | 69.4 | 68.4 | 82.8 | 84.8 |
Table 3. Pass rates (%) on WordEval under the Word Object oracle. Case weights cases equally, whereas Macro averages within each Atomic command or Compositional scenario and then across commands or scenarios (§ 4.1). Bold marks the best agent result in each column. Human reports the mean across three participants.
Agent Inputs and Task Prompt
Agents receive the task instruction and input document and are asked to return pred.docx with trajectory documentation. Reference documents and oracle files are available only to the evaluator. Figure 7 illustrates the task-file protocol. CLI wrappers and model-specific effort settings vary across systems.

Figure 7. Task-file protocol with an abridged task description.
Per-case prompt
Please follow @TASK_DESCRIPTION_per_task.md to complete the task. Only access files
in the current directory.
Task description (TASK_DESCRIPTION_per_task.md), abridged
# Word Command Benchmark - Task Description
## Overview
This benchmark evaluates your ability to execute Microsoft Word
formatting and editing commands on documents. Each test case provides
an input Word document and a natural language instruction describing
the modification to perform.
## Task Structure
The current directory contains a single test case:
doc_in.docx # Input document
instruction.txt # Task instruction
pred.docx # YOUR OUTPUT (to be created)
method.md # YOUR METHOD (full reproducible trajectory)
## Your Task
1. Read the instruction.txt file to understand what modification
needs to be made
2. Open the doc_in.docx file (input document)
3. Execute the instruction on the document
4. Save the result as pred.docx in the current directory
5. Document your complete trajectory in method.md
## Method Documentation Requirements
# Status
STATUS: [SUCCESS|PARTIAL|FAILED]
REASON: [Optional: explain if partial or failed]
# Approach / # Step-by-Step Trajectory / # Complete Implementation
### Status Definitions
- SUCCESS: Task completed as instructed
- PARTIAL: Task attempted but may not be fully correct
- FAILED: Could not complete the task
You can use any method you prefer.
[... four worked examples (python-docx, VBA, Word GUI, other tools
such as Office.js, OpenXML SDK, Apache POI, pandoc) omitted ...]
Agent Systems and Run Settings
Qwen Code uses a 50-turn budget and OpenCode with Gemini a 50-step budget, each with a 600-second per-case limit. Failed runs and missing outputs remain in the 642-case denominator. All outputs were evaluated with Microsoft Word 16.0.17932.21000 on Windows 11 virtual machines (OS build 10.0.26200) with Python 3.12. Opus 5 uses a Copilot compatibility route for the Stack Object case. Table 8 reports packages, available versions, and model identifiers.
| Agent system | Package | Version | Model identifier |
| Claude Code | @anthropic-ai/claude-code | – | Opus 5 |
| Codex | @openai/codex | 0.128.0 | gpt-5.6-sol |
| Kimi Code | kimi-cli | 1.41.0 | Kimi-K3 |
| OpenCode | opencode | 1.18.3 | gemini-3.8-flash |
| ZCode | zcode-agent-cli | 0.15.2 | GLM-5.2 |
| OpenCode | opencode | – | DeepSeek-V4-Pro |
| Qwen Code | @qwen-code/qwen-code | 0.22.3 | Qwen/Qwen3.8-27B |
| Mini Agent | Mini-Agent | – | MiniMax M3 |
Table 8. Agent packages and model identifiers. – denotes an unavailable version.
Human Execution Protocol
Three participants (A01–A03) each perform the same 642 editing tasks in Microsoft Word, through a local browser page that presents one task at a time with its instruction and input document. Tasks are grouped into bundles, and the Atomic tasks in a bundle precede the Compositional task that combines them. A practice task precedes the benchmark tasks and is excluded from analysis. For each task, the participant downloads the input document, carries out the instruction in Word, and uploads the saved document. The instructions given to participants, reproduced in the box below, specify which tools may be used. The final output submitted for each task is scored with the same evaluator as agent outputs. These execution results are distinct from the human judgments of agent outputs in Appendix § 9.2.§ 3. Table 9 reports individual Case and Macro pass rates, and the Human row in the main table averages the three participant-specific rates.
Human execution instructions, abridged
# WordEval Human Annotation
This task evaluates whether a human annotator can execute a Microsoft
Word editing instruction on a document, and validates the WordEval
benchmark cases (instruction clarity + Word Object oracle correctness).
Each task contains one input Word document and one natural-language
instruction.
Cases are organized into task bundles. Within each bundle, complete the
atomic tasks first, then complete the corresponding compositional task.
Each task has two stages:
1. Stage 1: Word-only baseline - complete the task using only memory,
ribbon browsing, and in-document text search (no oracle).
2. Stage 2: Assisted attempt - optional, after Stage 1. Use external
help to attempt the task again, starting from a fresh copy of the
input document.
[... setup, onboarding, and step-by-step interface instructions omitted ...]
Stage 1 - Allowed
- Memory and ribbon/menu browsing
- Right-click menus and keyboard shortcuts
- Ctrl+F for in-document text search
Stage 1 - Forbidden
- Search box at the top of the Word window (the bar above the ribbon labeled "Search" or "Tell me what you want to do", with a magnifying glass icon). Typing into it suggests commands or invokes Copilot.
- Copilot panel (the AI sidebar accessible from the ribbon)
- F1 Help when it opens docs.microsoft.com or any web page
- Web search, ChatGPT, Claude, or any other AI assistant
- Scripts, macros, VBA, WordAgent, or programmatic editing
Stage 1 measures what a Word user can do from memory and ribbon browsing alone, matching the agent’s no-oracle condition.
Stage 2 - Allowed
- Everything Stage 1 allows, plus:
- Search box at the top of the Word window ("Search" / "Tell me")
- Word’s Copilot panel (for explanations only)
- F1 Help and docs.microsoft.com
- Web search, YouTube tutorials, Stack Overflow
- ChatGPT / Claude / external Copilot as a tutor (ask, read, then perform the clicks yourself)
Stage 2 - Forbidden
- AI directly editing the docx (no AI-generated macros, VBA, scripts, or XML run on the document)
- Pasting AI-generated final content into the docx as a shortcut
- Any agent or script that performs the edit on your behalf
AI may explain, not execute. You must perform the final clicks in Word yourself.
Compare to Gold (Submit step)
Before final submission, open gold.docx (the reference output) in Word alongside your final uploaded result (pred_stage2.docx if you continued, otherwise pred_stage1.docx), and rate human_matches_gold.
[...]
Instruction Specification Severity (0-3)
Rate from the perspective of another reasonable Word user, not your own struggle. Anchors:
- 0 - clear, all reasonable annotators converge on the same edit
- 1 - minor interpretation needed, a reasonable default exists
- 2 - important detail missing or multiple valid interpretations
- 3 - too ambiguous to execute, or contradicts the docx
human_matches_gold (Binary)
- match - the targeted edits from the instruction are realized in your output as in gold
- not_match - at least one targeted edit is missing or applied differently
[... status and confidence definitions omitted ...]
Time Policy
There is no time limit. Take your time on each case. If a case feels broken or impossible after a reasonable effort (~10 min), mark it as failed and move on.
Code and Data Availability
The released WordEval artifact is organized around a canonical benchmark directory with separate Atomic and Compositional tracks. Each track contains agent-visible task records, input .docx files, evaluator-only reference documents, oracle metadata, and task-level taxonomy metadata. The agent-visible records specify only the task identifier, natural-language instruction, and input document path; the evaluator-only records specify the corresponding reference document and oracle specification used to score predictions. This separation mirrors the evaluation protocol in § 3.1: agents receive the input document and instruction, while reference documents and oracle metadata are used only by the evaluator.
The artifact also includes the benchmark generator, Word Object evaluator, scripts for producing agent-visible task folders and scoring prediction files, a README, a data card, license files, and Croissant metadata. The primary evaluator requires Windows, Microsoft Word, and Python COM automation, matching the environment assumptions in § 4.1. Code is released under the MIT License, and the benchmark data are released under CDLA-Permissive-2.0. Code and data are available at https://aka.ms/word-eval. The data card and Croissant metadata document the dataset schema, intended use, limitations, privacy properties, licensing, and responsible-release metadata.
Additional Compositional Analyses
The main-text analysis (§ 4.4) treats each compositional case as a single binary outcome, so the whole case either passes or fails. Every released compositional case (96 cases / 319 atomic intents across the four classes) declares the atomic intents it contains via spec.composition.atomic_intents, and per-intent verdicts are extracted from the recorded verifier diagnostics when available. This appendix uses those per-intent verdicts to decompose outcome patterns that case-level outcomes collapse. These are operation-limited observations where the embedded intent fails along with its referenced Atomic case, transfer failures where a referenced Atomic pass verdict does not carry over to the corresponding embedded atomic intent, and end-state coincidences where a compositional final state satisfies an intent despite the corresponding Atomic failed verdict. Operation, feature area, and depth breakdowns are reported as secondary diagnostics after this intent-level view.
Intent-Level Outcome Decomposition
Definitions. For each declared intent , is the verdict of its referenced Atomic case in isolation, and is the verdict of the same intent inside the compositional output. The transfer rate is the share of intents passing in isolation that also pass inside the composition. Each pair of and verdicts falls into one of four cells. marks an intent that passes in both settings. marks a transfer failure, where the Atomic case passes but the embedded intent fails. marks an operation-limited observation, where both fail. marks an end-state coincidence, where the compositional output satisfies the intent’s final-state check although the Atomic case fails. The cells describe outcomes, not their causes.
Coverage. The eight systems yield declared intent observations. Embedded verdicts come from the verifier’s per-intent diagnostics whenever a prediction document exists and its diagnostics cover the intent. Two cases that record their checks in other diagnostic structures are mapped explicitly. Each intent verdict thus reflects only its own check. Together with the Atomic verdicts, these yield 1701 observed pairs (66.7%) from 36.1% to 92.5% across systems, with the remainder lacking an embedded or a binary Atomic verdict. All intent-level rates are computed over these observed pairs.
Composition classes. Table 10 pools the four cells across the eight systems by composition class. Its transfer-failure column is over all Atomic-pass pairs, its operation-limited column is the share of among all pairs, and its coincidence column is the share of among all pairs. Layout-coupled cases have the highest transfer-failure rate, 28.1%. Additive and prerequisite cases have the largest operation-limited shares, 9.7% and 8.2%. In layout-coupled cases, 45 of the 48 failed embedded intents passed in isolation. In additive and prerequisite cases, 113 of 195 and 18 of 28 failed embedded intents also failed in isolation. Embedded failures in layout-coupled cases are therefore mostly transfer failures, whereas more than half of those in additive and prerequisite cases are operation-limited.
| Class | Transfer | Transfer fail | Operation-limited | Coincidence | ||
| Additive | 79.7 | 91.2 | 8.8 | 9.7 | 10.7 | 1170 |
| Prerequisite | 70.8 | 93.5 | 6.5 | 8.2 | 21.0 | 219 |
| Scope-changing | 91.9 | 95.2 | 4.8 | 1.5 | 6.6 | 136 |
| Layout-coupled | 90.9 | 71.9 | 28.1 | 1.7 | 7.4 | 176 |
Table 10. Pooled intent outcomes by composition class. Rates use observed pairs across the eight systems (). Operation-limited and coincidence are the shares of and pairs, and Transfer fail is the share of among Atomic-pass pairs.
Intent Transfer by System
Agent systems. Figure 8 reports , , and the transfer rate across the four classes for all eight agent systems. This is the intent-level counterpart to the case-level analysis in Figure 5. Panel A gives the Atomic pass rate, panel B asks whether embedded intents pass in compositional outputs, and panel C asks the same question only for intents whose referenced Atomic case passes in isolation. Class differences vary across systems. Opus 5, GPT-5.6, Gemini 3.8 Flash, and K3 transfer at least 89.9% of their Atomic-pass intents in every class, while the other four systems have their lowest transfer on layout-coupled cases. Because intents without an observed embedded verdict are excluded, some system–class cells have small denominators (Table 11).
For the case-level conditioning used in Figure 5, Table 11 reports the case-level rates and the all-components-pass denominators for all eight systems, together with their intent-level rates. Its intent-level columns place and side by side, and their difference is the pass gap. On layout-coupled cases, the pass gap is largest for the same four systems, suggesting composite-intent loss that is not fully explained by atomic-component verdicts on the same operations. Section D.4 examines cells with negative gaps.
| System | Class | Pass gap | ||||||||
| Opus 5 | Additive | 80.4 | 82.6 | 89.2 | 37 | 97.4 | 93.8 | 3.6 | 93.7 | 190 |
| Prerequisite | 80.0 | 80.0 | 85.0 | 20 | 95.5 | 95.5 | 0.0 | 97.6 | 42 | |
| Scope-changing | 58.3 | 100.0 | 100.0 | 7 | 95.8 | 100.0 | -4.2 | 100.0 | 23 | |
| Layout-coupled | 100.0 | 84.6 | 84.6 | 13 | 100.0 | 93.8 | 6.2 | 93.8 | 32 | |
| GPT-5.6 | Additive | 37.0 | 56.5 | 58.8 | 17 | 73.2 | 90.9 | -17.7 | 91.7 | 120 |
| Prerequisite | 44.0 | 52.0 | 54.5 | 11 | 72.7 | 90.9 | -18.2 | 95.8 | 24 | |
| Scope-changing | 58.3 | 83.3 | 100.0 | 7 | 90.0 | 100.0 | -10.0 | 100.0 | 18 | |
| Layout-coupled | 69.2 | 76.9 | 66.7 | 9 | 85.7 | 92.9 | -7.1 | 91.7 | 24 | |
| K3 | Additive | 41.3 | 47.8 | 89.5 | 19 | 85.6 | 82.7 | 2.9 | 89.9 | 119 |
| Prerequisite | 28.0 | 48.0 | 42.9 | 7 | 60.0 | 88.6 | -28.6 | 95.2 | 21 | |
| Scope-changing | 75.0 | 75.0 | 66.7 | 9 | 95.2 | 95.2 | 0.0 | 95.0 | 20 | |
| Layout-coupled | 38.5 | 61.5 | 80.0 | 5 | 94.4 | 94.4 | 0.0 | 94.1 | 17 | |
| Gemini 3.8 Flash | Additive | 28.3 | 45.7 | 76.9 | 13 | 96.2 | 100.0 | -3.8 | 100.0 | 76 |
| Prerequisite | 40.0 | 48.0 | 60.0 | 10 | 100.0 | 100.0 | 0.0 | 100.0 | 18 | |
| Scope-changing | 25.0 | 58.3 | – | 3 | 100.0 | 100.0 | 0.0 | 100.0 | 6 | |
| Layout-coupled | 15.4 | 53.8 | – | 2 | 100.0 | 91.7 | 8.3 | 91.7 | 12 | |
| GLM-5.2 | Additive | 41.3 | 43.5 | 73.7 | 19 | 82.8 | 85.9 | -3.1 | 93.4 | 106 |
| Prerequisite | 16.0 | 32.0 | – | 4 | 45.5 | 86.4 | -40.9 | 90.0 | 10 | |
| Scope-changing | 41.7 | 58.3 | 100.0 | 5 | 88.9 | 94.4 | -5.6 | 100.0 | 16 | |
| Layout-coupled | 61.5 | 23.1 | 25.0 | 8 | 90.5 | 47.6 | 42.9 | 42.1 | 19 | |
| DeepSeek V4 Pro | Additive | 39.1 | 39.1 | 61.1 | 18 | 69.1 | 79.2 | -10.1 | 87.0 | 123 |
| Prerequisite | 28.0 | 36.0 | 57.1 | 7 | 64.3 | 82.1 | -17.9 | 88.9 | 18 | |
| Scope-changing | 83.3 | 66.7 | 70.0 | 10 | 90.5 | 85.7 | 4.8 | 89.5 | 19 | |
| Layout-coupled | 53.8 | 30.8 | 28.6 | 7 | 84.0 | 60.0 | 24.0 | 52.4 | 21 | |
| Qwen3.8-27B | Additive | 37.0 | 30.4 | 64.7 | 17 | 78.5 | 70.2 | 8.3 | 83.2 | 95 |
| Prerequisite | 40.0 | 24.0 | 30.0 | 10 | 63.6 | 77.3 | -13.6 | 85.7 | 14 | |
| Scope-changing | 33.3 | 66.7 | – | 4 | 86.7 | 93.3 | -6.7 | 92.3 | 13 | |
| Layout-coupled | 46.2 | 15.4 | 16.7 | 6 | 91.7 | 37.5 | 54.2 | 36.4 | 22 | |
| MiniMax M3 | Additive | 26.1 | 34.8 | 83.3 | 12 | 62.0 | 68.1 | -6.0 | 91.3 | 103 |
| Prerequisite | 8.0 | 16.0 | – | 2 | 47.1 | 64.7 | -17.6 | 75.0 | 8 | |
| Scope-changing | 58.3 | 41.7 | 42.9 | 7 | 90.9 | 81.8 | 9.1 | 80.0 | 10 | |
| Layout-coupled | 38.5 | 23.1 | 40.0 | 5 | 81.2 | 62.5 | 18.8 | 69.2 | 13 |
Table 11. Case-level and intent-level rates by system and composition class. Case-level rates use all assigned cases (46, 25, 12, and 13 per class) and count missing outputs as failures. is the share of cases whose linked Atomic variants all pass, and counts them. is omitted (–) when fewer than 5 such cases exist. Intent-level rates use observed pairs, the pass gap is , and counts Atomic-pass intents.
Closed- and open-weight systems. Among the evaluated agent systems, those using open-weight models (K3, MiniMax M3, Qwen3.8-27B, GLM-5.2, DeepSeek V4 Pro) have lower values than those using closed-weight models (Opus 5, GPT-5.6, Gemini 3.8 Flash) on both the embedded pass rate and the transfer rate (Figure 8, Table 12). A paired case-cluster bootstrap [6] with 2,000 resamples of shared cases estimates positive closed-weight versus open-weight transfer differences in all four classes, +5.3 pp for additive (95% CI [1.2, 10.2]), +8.2 pp for prerequisite ([0.0, 18.1]), +7.7 pp for scope-changing ([0.0, 18.4]), and +35.7 pp for layout-coupled cases ([19.5, 53.7]).
Figure 8. Intent-level rates by system and composition class. (A) Atomic pass rate . (B) Embedded pass rate , measured inside the compositional output. (C) Transfer rate , among intents whose Atomic case passes in isolation. Rates use observed pairs, and Table 11 lists the values.
| Class | closed (%) | open (%) | (pp) | 95% CI | clusters | ||
| Additive | 94.3 | 88.9 | 5.3 | [1.2, 10.2] | 384 | 542 | 44 |
| Prerequisite | 97.0 | 88.7 | 8.2 | [0.0, 18.1] | 66 | 71 | 19 |
| Scope-changing | 100.0 | 92.3 | 7.7 | [0.0, 18.4] | 47 | 78 | 12 |
| Layout-coupled | 92.2 | 56.5 | 35.7 | [19.5, 53.7] | 64 | 92 | 12 |
Table 12. Closed- versus open-weight transfer by composition class. is closed minus open in percentage points with its 95% paired bootstrap CI, counts Atomic-pass intents, and clusters counts the cases observed in both groups.
Breakdown by Operation and Word Feature Area
Table 13 ranks atomic operations by intent-level transfer rate, pooled across the eight agent systems and restricted to operations with at least 15 observed pairs. Apart from shadow and reflection effects (29.4–44.4%), the lowest-transfer operations are geometric/spatial primitives (rotate_angle 47.2%, flip_direction 58.8%, align_command 72.6%), followed by list and icon-insertion primitives. This is consistent with the pattern by composition class. Layout-coupled and certain additive bundles share some of the same problematic primitives, suggesting that the underlying spatial coordination that those classes over-represent contributes to low transfer.
| Operation | |||||
shadow | 19 | 89.5 | 26.3 | 29.4 | 17 |
pic_shadow | 19 | 84.2 | 26.3 | 31.2 | 16 |
reflection | 17 | 52.9 | 35.3 | 44.4 | 9 |
rotate_angle | 46 | 78.3 | 52.2 | 47.2 | 36 |
flip_direction | 17 | 100.0 | 58.8 | 58.8 | 17 |
align_command | 73 | 100.0 | 72.6 | 72.6 | 73 |
list_type | 20 | 80.0 | 65.0 | 75.0 | 16 |
insert_icon | 17 | 76.5 | 88.2 | 92.3 | 13 |
distribute_command | 47 | 100.0 | 93.6 | 93.6 | 47 |
position_preset | 22 | 95.5 | 90.9 | 95.2 | 21 |
legend | 30 | 90.0 | 96.7 | 96.3 | 27 |
alignment | 46 | 100.0 | 100.0 | 100.0 | 46 |
Table 13. Twelve lowest-transfer operations with at least 15 observed pairs, pooled across the eight systems. counts observed pairs.
| Word feature area | |||||
| Annotations | 51 | 84.3 | 98.0 | 97.7 | 43 |
| Charts | 171 | 88.9 | 95.3 | 97.4 | 152 |
| Equations | 75 | 61.3 | 92.0 | 93.5 | 46 |
| Graphics | 17 | 76.5 | 88.2 | 92.3 | 13 |
| Lists | 48 | 87.5 | 77.1 | 83.3 | 42 |
| Page Structure | 49 | 73.5 | 69.4 | 91.7 | 36 |
| Paragraphs | 225 | 89.3 | 96.9 | 99.5 | 201 |
| Pictures | 128 | 59.4 | 46.9 | 61.8 | 76 |
| References | 14 | 71.4 | 78.6 | 90.0 | 10 |
| Shape appearance | 130 | 80.8 | 72.3 | 85.7 | 105 |
| Shape geometry | 286 | 89.2 | 79.0 | 79.6 | 255 |
| Shape text | 13 | 84.6 | 84.6 | 81.8 | 11 |
| SmartArt | 128 | 74.2 | 71.9 | 88.4 | 95 |
| Tables | 133 | 85.7 | 93.2 | 94.7 | 114 |
| Text | 233 | 74.2 | 93.6 | 96.0 | 173 |
Table 14. Intent transfer by Word feature area, pooled across the eight systems. counts observed pairs.
Table 14 groups transfer by Word feature area, the axis of Figure 4. Pictures (+12.5 pp), Lists (+10.4 pp), and Shape geometry (+10.1 pp) have the largest pass gaps. Text and Equations have negative gaps ( pp and pp) because end-state coincidences outnumber transfer failures in these areas. Section D.4 examines these coincidences.
End-State Coincidence Audit
Intents can also pass more often inside compositions than in isolation. The pass gap is negative in prerequisite cases for six of the eight systems, and GPT-5.6 has negative gaps in every class, from to pp. In these cells, end-state coincidences outnumber transfer failures.
The cell – Atomic-failed but composite-intent-passed – accounts for 193 of 1,701 observed pairs (11.3%), of which 121 (62.7% of the end-state coincidence set, 7.1% of all observed pairs) appear in cases that also pass at case level. In a composite, another unit or a no-op can sometimes satisfy the same end-state predicate (e.g., ungroup_command passes when no group exists). The transfer rate is computed over Atomic-pass pairs, so the cell does not enter it, and the case-level metric counts these cases as passes.
The most frequent operations among these coincidences are insert_equation (17), drop_cap (12), outline_weight (10), outline_color (8), and rotate_angle and table_style (7 each). Table 15 shows one example from a passing case for each of these operations and for ungroup_command.
| Operation | System | Class | Compositional case | Atomic case |
insert_equation | GPT-5.6 | Prerequisite | PrerequisiteEquation/variant_fraction | EquationInsert New/apply |
drop_cap | GLM-5.2 | Additive | AdditiveParagraph Composition/variant_center_dropcap | DropCapInsert Gallery/dropped |
outline_weight | MiniMax M3 | Additive | AdditiveFontComposition/variant_mixed_effects | TextOutline WeightGallery/apply_1pt |
outline_color | K3 | Additive | AdditiveFontComposition/variant_off_heavy | TextOutlineColor PickerAlternate/apply_Blue |
rotate_angle | GPT-5.6 | Additive | AdditiveLayout Composition/variant_rotate_cw_distribute_h | ObjectRotate Gallery/apply_rotate_right_90 |
table_style | Qwen3.8-27B | Additive | AdditiveTable Composition/variant_style_textdir_border | TableStyles GalleryWord/Grid_Table_4___Accent_1 |
ungroup_command | DeepSeek V4 | Layout-coupled | LayoutCoupledGroup Lifecycle/variant_flip_horizontal | ObjectsUngroup/apply |
Table 15. End-state coincidence examples by operation. Each row is a case with an Atomic Fail, embedded Pass, and whole-case Pass.
Composition Depth
Composition depth is the number of atomic intents in a compositional case. The case-level pass rate of Opus 5 is 88.7% at and 92.3% at , then falls to 81.8% at , 69.2% at –6, and 66.7% at (Figure 9). Its per-intent transfer stays between 90.9% and 96.8% in every bucket, including (Figure 10).
Figure 9. Case-level pass rates by composition depth for the eight systems. (A) over all assigned cases, counting missing outputs as failures. (B) over cases whose linked Atomic variants all pass, omitted where no case is eligible. Depth buckets hold 53, 13, 11, 13, and 6 cases at , 3, 4, 5–6, and , and the bucket has a mean depth of 10.2.
Figure 10. Intent-level rates by composition depth for the eight systems. (A) Atomic pass rate . (B) Embedded pass rate . (C) Transfer rate . Rates use observed pairs.
Task Examples and Complete Command Catalogue
This section collects the task instructions, input-document examples, and command enumeration supporting Appendix A.
Representative Case Examples
Table 16 and Table 17 give representative instructions shown to agents for single-operation Atomic cases and multi-operation Compositional cases, respectively. Figure 11 and Figure 12 show, for one released case per track, the exact instruction together with a rendered preview of the input document.
| Word feature area | Source command | Instruction example |
| Text | Bold | Make the word ’Target’ bold. |
| Paragraphs | AlignCenter | Center the paragraph containing ’Target’. |
| Lists | BulletsGalleryWord | Apply bullet list formatting to all paragraphs in the document using checkmark bullets. |
| Tables | TableInsertGallery | Insert a 2x2 table. |
| Shape geometry | AlignDistributeHorizontally | Distribute all three rectangles evenly horizontally with equal spacing between them. |
| Shape appearance | ShapeFillColorPicker | Set the shape’s fill color to Blue (RGB: 0, 112, 192). |
| Shape text | TextAlignGallery | Align the text to the bottom inside the text box. |
| Pictures | PictureCropAspectRatio1To1 | Crop the image to a 1:1 (square) aspect ratio, centered on the image. |
| Graphics | IconInsertFromFile | Insert a 150 x 150 pt star-shaped SVG icon filled with blue color (RGB: 68, 114, 196) into the document. |
| Charts | ChartInsert | Insert a clustered column chart into the document with categories ’Q1’ and ’Q2’, and one series ’Revenue’ with values 100 and 150. |
| SmartArt | SmartArtInsert | Insert a SmartArt graphic using the ’Basic Block List’ layout with nodes labeled ’Alpha’, ’Beta’, ’Charlie’, ’Delta’, and ’Echo’. |
| Equations | EquationFractionGallery | Add the fraction ’dy/dx’ to the equation ’y =’. |
| Page Structure | PageBreakInsertWord | Insert a page break after the word ’TARGET’ to start a new page. |
| Annotations | InsertNewComment | Add a comment with text ’Test comment.’ on the word ’TARGET’. |
| References | CitationInsert | At the beginning of the document body, insert an in-text citation using the existing source tagged ’Smith2020’. |
Table 16. Representative atomic WordEval case examples by Word feature area.

Figure 11. Instruction and input document shown to agents for a representative atomic case (AlignDistributeHorizontally, Shape geometry feature area). The input page render is cropped to the content region and presented in a Windows Word application frame.
| Composition class | Referenced atomic commands | Instruction example |
| Additive | AlignCenter; ParagraphSpacingBefore; ParagraphSpacingAfter; DropCapInsertGallery | Center the paragraph containing the word ’Target’, set paragraph spacing to 0 points before and 0 points after for all paragraphs, and apply a dropped drop cap to the first paragraph. |
| Prerequisite | ChartInsert; ChartDataLabel | Below the ’Quarterly Sales:’ heading, insert a clustered column chart with categories ’Q1’ and ’Q2’, and one series ’Revenue’ with values 100 and 150, then add data labels at the ’Outside End’ position. |
| Scope-changing | TableRowsInsertBelowWord; TableColumnsInsertRight; TableSplitTable; TableColumnsDistribute; TableRowsDistribute | Insert a new row below the first row and a new column to the right of the first column, split the table at the inserted row, then distribute the columns in the upper table and distribute the rows in the lower table. |
| Layout-coupled | TextWrapGallery; PicturePositionGallery | Apply square wrapping to the cross shape (text wraps around the cross’s rectangular bounding box), then position it at the top-center of the page, using the Position preset (within default page margins). |
Table 17. Representative compositional WordEval case examples by composition class. Referenced atomic commands are derived from the case’s linked atomic_intents.

Figure 12. Instruction and input document shown to agents for a representative layout-coupled compositional case (TextWrapGallery followed by PicturePositionGallery). The input page render is cropped to the content region and presented in a Windows Word application frame.
Instruction
Apply square wrapping to the cross shape (text wraps around the cross’s rectangular bounding box), then position it at the top-center of the page, using the Position preset (within default page margins).
Input document (doc_in.docx)
Complete Released-Command Catalogue
Table 18 enumerates the 231 released Atomic commands by Word feature area.
| Word feature area | Released commands |
| Shape geometry | AlignDistributeHorizontally, AlignDistributeVertically, GalleryAllShapesAndCanvas, ObjectBring Forward, ObjectBringToFront, ObjectRotateGallery, ObjectSendBackward, ObjectSendToBack, ObjectsAlignBottomSmart, ObjectsAlignCenterHorizontalSmart, ObjectsAlignLeftSmart, Objects AlignMiddleVerticalSmart, ObjectsAlignRightSmart, ObjectsAlignTopSmart, ObjectsGroup, ObjectsUngroup, PicturePositionGallery, ShapeChangeShapeGallery, TextBoxInsert, TextBox InsertGallery, TextBoxInsertVerticalWord, TextWrapGallery |
| Shape appearance | BevelShapeGallery, GlowColorPicker, GradientGallery, ObjectEffectGlowGallery, ObjectEffect ShadowGallery, ObjectEffectSoftEdgesGallery, ReflectionGallery, ShapeFillColorPicker, Shape OutlineColorPicker, ShapeStylesGallery, _3DRotationGallery |
| Shape text | TextAlignGallery, TextDirection, TextDirectionGallery |
| Pictures | HorizontalLineInsert, OutlineColorPicker, PictureArtisticEffectsGallery, PictureColorTemperature Gallery, PictureCropAspectRatio16To10, PictureCropAspectRatio16To9, PictureCropAspectRatio1 To1, PictureCropAspectRatio2To3, PictureCropAspectRatio3To2, PictureCropAspectRatio3To4, Picture CropAspectRatio3To5, PictureCropAspectRatio4To3, PictureCropAspectRatio4To5, Picture CropAspectRatio5To3, PictureCropAspectRatio5To4, PictureEffectsGlowGallery, PictureEffects ShadowGallery, PictureEffectsSoftEdgesGallery, PictureFillCrop, PictureFitCrop, PictureRecolor Gallery, PictureReflectionGallery, PictureReset, PictureResetAndSize, PictureRotationGallery, PictureSaturationGallery, PictureSharpenSoftenGallery, _3DBevelPictureTopGallery |
| Graphics | GraphicsColorPicker, GraphicsOutlineColorPicker, IconInsertFromFile |
| Charts | ChartColorsGallery, ChartDataLabel, ChartDataTable, ChartGridlinesGallery, ChartInsert, Chart LayoutGallery, ChartLegend, ChartLines, ChartResetToMatchStyle, ChartSwitchRowColumn, ChartTitle, ChartTrendline, ChartUpDownBars, ColorPickerChartLineStyles |
| SmartArt | ArrowStyleGallery, BevelTextGallery, OutlineDashesGallery, OutlineWeightGallery, SmartArtAdd Assistant, SmartArtAddShapeAbove, SmartArtAddShapeAfter, SmartArtAddShapeBefore, Smart ArtAddShapeBelow, SmartArtChangeColorsGallery, SmartArtDemote, SmartArtInsert, SmartArt LargerShape, SmartArtLayoutGallery, SmartArtPromote, SmartArtReorderDown, SmartArt ReorderUp, SmartArtResetGraphic, SmartArtRightToLeft, SmartArtSmallerShape, SmartArtStyles Gallery, TextEffectTransformGallery, TextEffects3DRotationGallery, TextFillColorPicker |
| Equations | EquationAccentGallery, EquationDelimiterGallery, EquationFractionGallery, EquationFunction Gallery, EquationInsertNew, EquationIntegralGallery, EquationLargeOperatorGallery, Equation LimitGallery, EquationLinearAll, EquationLinearOne, EquationMatrixGallery, EquationNormal Text, EquationOperatorGallery, EquationProfessionalAll, EquationProfessionalOne, Equation RadicalGallery, EquationScriptGallery, EquationSymbolsInsertGallery, InsertBuildingBlocks EquationsGallery |
| Page Structure | BlankPageInsert, CoverPageInsertGallery, DocInfoAuthor, DocInfoDocTitle, DocInfoFileName, FooterInsertGallery, HeaderFooterRemoveFooterWord, HeaderFooterRemoveHeaderWord, Header InsertGallery, PageBreakInsertWord, PageNumbersInfoInsertGallery, PageNumbersInMargins InsertGallery, PageNumbersInHeaderInsertGallery, WatermarkRemove |
| Annotations | AcceptAllChangesInDocAndStopTracking, DeleteAllResolvedComments, EndnoteInsertWord, FootnoteEndnoteDialog, FootnoteInsert, InsertNewComment, RejectAllChangesInDocAndStop Tracking, ReviewDeleteAllCommentsInDocument, ReviewDeleteComment, ReviewTrackChanges |
| References | BibliographyGallery, BibliographyInsert, BibliographyStyle, CaptionInsert, CitationInsert, HyperlinkEdit, HyperlinkInsert, HyperlinkRemove, IndexUpdate, PageNumberFieldInsertGallery, TableOfContentsGallery, TableOfContentsRemove |
Table 19.
References
- [1]Anthropic. Claude code. https://www.anthropic.com/claude-code.
- [2]Anthropic. System card: Claude Opus 5, 2026. URL https://www-cdn.anthropic.com/b514064af1408018e64b1ad24e7d5e75850b4ffd/Claude%20opus%205%20System%20Card.pdf.
- [3]Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows agent arena: Evaluating multi-modal os agents at scale. In ICML, 2025.arxiv.org/abs/2409.08264
- [4]DeepSeek-AI et al. DeepSeek-V4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026. URL https://arxiv.org/abs/2606.19348.
- [5]ECMA International. Standard ECMA-376: Office Open XML file formats. Technical report, ECMA International, 2016. URL https://ecma-international.org/publications-and-standards/standards/ecma-376/.
- [6]Bradley Efron. Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7 (1):1–26, 1979.
- [7]GitHub. GitHub Copilot: Your ai pair programmer. URL https://github.com/features/copilot.
- [8]GLM-5 Team et al. GLM-5: From vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026. URL https://arxiv.org/abs/2602.15763.
- [9]Google DeepMind. Gemini 3.8 Flash model card. https://deepmind.google/models/model-cards/gemini-3-8-flash/.
- [10]Kimi Team et al. Kimi K3: Open frontier intelligence. arXiv preprint arXiv:2607.24653, 2026. URL https://arxiv.org/abs/2607.24653.
- [11]Microsoft. Application CompareDocuments method (Word). https://learn.microsoft.com/en-us/office/vba/api/word.application.comparedocuments.
- [12]Microsoft. Office fluent UI command identifiers. https://github.com/OfficeDev/office-fluent-ui-command-identifiers.
- [13]Microsoft. Object model (Word). https://learn.microsoft.com/en-us/office/vba/api/overview/word/object-model.
- [14]MiniMax. Mini-Agent. https://github.com/MiniMax-AI/Mini-Agent.
- [15]MiniMax. MiniMax M3: Frontier coding, 1m context, native multimodality—all in one model. https://www.minimax.io/blog/minimax-m3, June 2026.
- [16]Moonshot AI. Kimi Code CLI. https://moonshotai.github.io/kimi-code.
- [17]OpenAI. Introducing Codex. https://openai.com/index/introducing-codex/, 2025.
- [18]OpenAI. GPT-5.6 system card. https://deploymentsafety.openai.com/gpt-5-6, 2026.
- [19]OpenCode. OpenCode. https://opencode.ai.
- [20]Qwen Team. Qwen3.8-Max: A new bar for coding and cowork. https://qwen.ai/blog?id=qwen3.8,.
- [21]Qwen Team. Qwen Code. https://qwenlm.github.io/qwen-code-docs,.
- [22]Weixuan Wang, Dongge Han, Daniel Madrigal Diaz, Jin Xu, Victor Rühle, and Saravan Rajmohan. Odysseybench: Evaluating llm agents on long-horizon complex office application workflows. arXiv preprint arXiv:2508.09124, 2025.arxiv.org/abs/2508.09124
- [23]Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 13(4):600–612, 2004.pubmed.ncbi.nlm.nih.gov/15376593
- [24]Zilong Wang, Yuedong Cui, Li Zhong, Zimin Zhang, Da Yin, Bill Yuchen Lin, and Jingbo Shang. Officebench: Benchmarking language agents across multiple applications for office automation. arXiv preprint arXiv:2407.19056, 2024.arxiv.org/abs/2407.19056
- [25]Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In NeurIPS, 2024.arxiv.org/abs/2404.07972
- [26]Z.ai. ZCode. https://github.com/zai-org/ZCode.
- [27]Henry Hengyuan Zhao, Kaiming Yang, Wendi Yu, Difei Gao, and Mike Zheng Shou. Worldgui: An interactive benchmark for desktop gui automation from any starting point. arXiv preprint arXiv:2502.08047, 2025.arxiv.org/abs/2502.08047







