betaFeedback

WorksComputer Science

WordInferBench: Can Agents Infer How to Edit from Word Document Context?

Abstract

Document editing is among the most common knowledge-work tasks, yet existing benchmarks for evaluating autonomous agents on Word document editing provide fully specified attribute-modification instructions. They do not evaluate agents' ability to infer the appropriate attributes from document context, a capability essential in real-world editing where users expect consistency without spelling out every attribute. We introduce WordInferBench, a benchmark comprising 280 base document-editing instances, each paired with three instruction conditions: descriptive (the instruction explicitly lists every attribute change), referential (the instruction points to another object as an exemplar), and inferential (the agent must examine document context to infer what attributes to apply). To evaluate contextual attribute inference, each base instance is constructed so that the target formatting is recoverable from an in-document exemplar. The instances are derived from real Word documents collected from Common Crawl and cover 13 object types and 81 attributes. We evaluate six recent coding-agent systems and find that the best-performing evaluated system, Claude Code with Opus 4.6, achieves a 62.9% task-level pass rate on descriptive variants but only 40.7% on inferential variants, indicating that current systems complete tasks less reliably when the required formatting changes must be determined from document context.

Introduction

Document editing is one of the most common knowledge-work tasks. In many scenarios, users expect an object to match existing document formatting without specifying every attribute change—for example, after inserting a new shape or pasting content. As agents increasingly operate computer applications autonomously, evaluating their document editing capabilities becomes an important research question.

Recent benchmarks have begun to evaluate agents on document-processing tasks [1] (Xie et al. 2024; Wang et al. 2024; Wang et al. 2025). However, their coverage of Word editing tasks remains incomplete. Existing tasks provide fully specified formatting instructions (e.g., “change the font to Times New Roman”). Such tasks do not evaluate whether an agent can infer the required formatting changes from document context.

We introduce WordInferBench, a benchmark of 280 base editing instances and 840 instruction-conditioned variants designed to evaluate agents on Word document formatting. The benchmark covers 13 object types and 81 distinct attributes. Each base instance contains an input document in which a target object’s attributes have been reset to defaults and a held-out reference document representing the desired state. At evaluation time, the agent receives only the input document and an instruction and must restore the target object’s attributes; the reference document is used only for scoring. We define three instruction conditions reflecting different user needs and types of available information:

  • Descriptive: The user specifies every attribute change explicitly (e.g., “For the picture on page 1, set brightness to 0.85, contrast to 0.15, and set the fill type to solid”).

  • Referential: The user asks the agent to make one object consistent with another (e.g., “Modify the picture on page 6 to be consistent with the picture on page 3”) without listing attributes.

  • Inferential: The user expects the agent to determine what changes are needed from context (e.g., “Review the document and determine

Overview of WordInferBench

Figure 1. Overview of WordInferBench. Left: Tasks are constructed from real Word documents by identifying same-type objects with identical formatting, resetting the target’s attributes to defaults, and generating three instruction variants. Right: To solve tasks, agents explore context, locate the target, infer the required changes, and execute the edits.

what modification is needed for the header in section 3, then apply the appropriate change”).

Together, these conditions evaluate how agents perform formatting tasks under different information conditions. In descriptive tasks, the agent must parse the specifications and execute the attribute changes correctly. In referential tasks, the agent must determine which attributes are relevant for consistency and ensure complete attribute coverage. In inferential tasks, the agent must explore the document context and infer both which existing object serves as the formatting reference and which attributes need to be modified.

For construction, we use real Word documents from Common Crawl ([6]) as reference documents and reset the attributes of a selected object to their defaults to produce the input document. The object’s original attributes serve as the ground truth. We require each document to contain at least one other same-type object with an identical formatting signature, ensuring that the original formatting is exactly recoverable from an in-document exemplar. This design limits inferential tasks to same-type exemplar discovery, providing agents with a relatively simple inference setting. Human verification further confirms that the intended formatting is inferable from context and that the reference is unambiguous.

We evaluate six recent coding-agent systems, including Claude Code with Opus 4.6 and Sonnet 4.6, Codex with GPT-5.5 and GPT-5.4, Claude Code with GLM 5.2, and Kimi CLI with Kimi K2.6 (Anthropic, 2026a,b,c; OpenAI, 2025, 2026b,a; Z.ai, 2026; Moonshot AI, 2026a,b), in sandboxed Windows environments. Even on descriptive variants, where all intended attribute changes are explicitly specified, the best evaluated system achieves only a 62.9% pass rate. All evaluated systems have lower pass rates on inferential variants (best: 40.7%), indicating that current systems complete tasks less reliably when the required formatting changes must be determined from document context.

Our contributions are as follows:

  • We propose WordInferBench, a benchmark with three paired instruction conditions: explicit specification, reference-based consistency, and context-driven inference.

  • Our benchmark covers 13 object types and 81 attributes, spanning paragraphs, tables, headers/footers, and various shape types.

  • We evaluate six recent coding-agent systems, showing that format execution remains difficult for complex objects, and contextual exemplar identification is challenging.

Related Work

Word-Related Evaluation

Recent benchmarks evaluate autonomous agents in real computer environments, where Word-related

BenchmarkTask ScenarioWord TasksObject TypesAttributesContextual Inference
OSWorldExplicit formatting & content editing47412✗
OfficeBenchCross-app workflow8010✗
DocFormEvalExplicit formatting1,911119✗
WorldGUIExplicit formatting & content editing6377✗
OursContext-aware formatting modification2801381✓

caption: Comparison of Word-related task coverage across open-source computer-use and document-formatting benchmarks. Contextual Inference denotes tasks requiring agents to infer the required formatting from document context.

:::

tasks are common given the ubiquity of document processing in daily workflows.

OSWorld (Xie et al., 2024) constructs tasks spanning multiple desktop applications, testing agents’ ability to complete diverse computer tasks through GUI interactions. Windows Agent Arena (Bonatti et al., [1]) adapts the OSWorld framework to a scalable Windows environment. WorldGUI (Zhao et al., 2025) benchmarks desktop GUI automation under varied initial states across ten applications. OfficeBench (Wang et al., 2024) focuses on multi-application workflow orchestration across office tools. OdysseyBench (Wang et al., 2025) evaluates long-horizon workflows that require agents to recover information from interaction histories and act across office applications. DocFormEval (Rao et al., 2024) benchmarks code generation for Word formatting from natural-language instructions.

More recently, DocFormBench (Rao et al., 2026) proposes 500 Chinese and English instances for content-aware Word formatting, where agents must locate target elements via natural-language descriptions and apply explicitly specified format changes. OfficeEval (Lv et al., [9]) constructs 200 tasks drawn from standardized Office proficiency exams with 7,118 machine-scorable rubrics, covering fine-grained formatting attributes in Word and other applications.

Despite their contributions, these benchmarks remain limited in evaluating formatting inference from document context under different information conditions. WordInferBench addresses this limitation by constructing formatting inference tasks with descriptive, referential, and inferential instruction variants.

Document Editing

A related line of work studies automated document editing driven by natural-language instructions. DocEdit-v2 (Suri et al., 2024) grounds edit requests to regions in document images and generates editing commands via a multimodal LLM. Agent-DocEdit (Wu et al., 2024) localizes target elements in content-rich documents and generates edit programs through pre-defined APIs. LEDGER (Wang et al., 2026) uses dependency-aware graphs and graph-guided retrieval for semantically consistent edits across long-document paragraphs.

These works address semantic or layout editing, target grounding, and cross-section consistency under high-level instructions. They do not evaluate exact recovery of native Word formatting attributes from in-document exemplars, which is the focus of our benchmark.

Benchmarks for Other Office Applications

Benchmarks have also been developed for other Office applications. In the presentation domain, SlidesGen-Bench (Yang et al., 2026) evaluates automated slide generation across content, aesthetics, and editability, while PresentBench (Chen et al., [5]) provides instance-specific, fine-grained rubric evaluation. In the spreadsheet domain, FINCH (also introduced as FinWorkBench) (Dong et al., [7]) targets enterprise finance and accounting workflows. SpreadsheetBench (Ma et al., [10]) collects real user questions from online forums and tests spreadsheet manipulation.

Spreadsheet benchmarks often require agents to inspect file structure before acting. However, existing Word benchmarks do not specifically evaluate tasks requiring target formatting to be inferred from within the document. WordInferBench evaluates this capability through context-driven formatting tasks.

Benchmark Construction

Task Formulation

In WordInferBench, each base instance consists of an input document, a held-out reference document, and three editing-instruction variants. The agent receives only the input document and one instruction; the reference document is not exposed and is used solely for scoring. The instruction requires modifying a target object in the input document to match its state in the reference document.

Tasks are divided into three instruction specificity conditions based on the amount of information provided in the instruction:

  • Descriptive: The instruction describes the specific attribute changes in detail. The agent only needs to understand and execute each change.

  • Referential: The instruction specifies another object of the same type in the document as a reference, without describing the specific changes. The agent must read the reference object’s attributes and apply them to the target.

  • Inferential: The instruction only identifies the target object, providing no reference or specific description. The agent must browse the document context and infer what changes should be made.

Construction Method

We collect real Word documents from Common Crawl ([6]) as reference documents. These documents originate from heterogeneous web sources and contain varied typographic structures. We reset the attributes of a selected object in a copy of a real document to construct the input document, while the object’s original attributes define the evaluation ground truth. We select objects for which the context contains at least one exactly matching same-type exemplar. If an object’s formatting appears only once in the document, its original attributes are not recoverable from context after resetting, so it is ineligible.

To filter documents and objects that can produce valid tasks, we first parse the XML of Word documents to identify their object types, retain documents containing at least two objects of a given type, and sample approximately 1,000 documents per category as candidates. For each candidate document, we extract the original attributes of all objects via Word COM. We then reset each candidate object independently in a separate document copy and compute the difference between its reset and original states.

On this basis, we pair objects of the same type within the same document that share identical formatting but differ in content, forming (target, reference) pairs. This construction scopes our inferential tasks to same-type exemplar discovery and exact attribute restoration. Exact matching ensures that the original formatting is recoverable from at least one in-document exemplar and enables deterministic evaluation through exact attribute comparison. We further use human verification to confirm that the intended formatting is inferable from context and that the reference answer is unambiguous (§ 3.4). While real-world scenarios may also involve inferring from multiple objects or coordinating changes across multiple targets, these are beyond the current scope. To cover as many attributes as possible, we employ a greedy set-cover algorithm to sample from all candidate pairs, maximizing the variety of modified attributes covered: iteratively selecting the object whose attribute differences cover the most uncovered attributes, breaking ties by preferring objects that cover more low-frequency attributes, and limiting each document to contributing at most one object per type.

We use GPT-5.5 to generate three instruction variants for each base instance (see Appendix § 10 for full prompts). For descriptive instructions, we first compute the target object’s structured attribute diff, converting all raw COM values to human-readable representations, e.g., enum constants to names (fill type 3 becomes “Gradient”). This structured diff is fed to the LLM to rewrite all changes as natural-language instructions, followed by a second refinement pass for conciseness. For referential instructions, the LLM generates instructions requiring format consistency between target and exemplar objects based solely on their positional information (type, page number, content), without specifying attributes. Inferential instructions provide only the target object’s positional information, requiring the agent to infer the modification plan from document context.

PII Removal

To protect privacy, we perform PII detection at both the image and text levels. Documents are rendered as page screenshots and an LLM detects face photographs; documents containing faces are entirely removed. Document text is scanned by an LLM for 8 categories of PII (see Appendix § 11), and detected PII text is replaced with placeholders. After replacement, we verify that the target object’s attributes are unaffected; otherwise the entry is removed.

Comparison of input and reference documents showing formatting attributes

Figure 2. A complete test case from WordInFerBench. The inferential instruction specifies only the target object; the agent must infer what attribute modifications are needed from document context. Left: the input document whose target object has been reset to defaults. Right: the held-out reference document with original formatting. The evaluator compares the 9 listed attributes between the agent’s output and the reference document.

Human Verification

We recruit 9 annotators divided into 3 groups of 3 to verify all samples (see Appendix § 12 for the detailed annotation protocol). Each group annotates all samples, where each sample is assigned to exactly one instruction type within that group. The assignment is rotated across groups so that every sample’s three instruction types are each verified by one group. To verify inferability and answer uniqueness, annotators first complete the edit using only the input document and instruction, without access to the held-out reference. They then inspect the reference and rate instruction clarity, reference correctness, and agreement between their independently produced output and the reference. This blind-first procedure provides an independent check that the intended formatting is inferable from context and that the reference answer is unambiguous. Annotators also check for PII. A sample is removed if all 3 annotators in at least one group reach consensus that it is problematic on any dimension. Cases with non-unanimous judgments are manually reviewed based on the annotators’ feedback, and the instruction or reference is revised when necessary. After this process, 280 of the original 312 candidates are retained (89.7%).

Statistics

The benchmark contains 280 base editing instances, each paired with descriptive, referential, and inferential instructions, yielding 840 instruction-conditioned variants across 272 distinct documents. It covers 81 attributes across 13 object types: plain paragraph, listing, hyperlink, table of contents, table, header, footer, autoshape, textbox, picture, chart, SmartArt, and WordArt (see Appendix § 9 for details). These attributes include character and paragraph formatting, hyperlinks, table borders and cell formatting, shape fill, line, shadow, 3D, glow, reflection, and other object-specific properties.

The statistics of the 280 inferential variants highlight why this setting requires more than local attribute copying. In 42.9% of inferential variants, the selected matching exemplar is on a different page from the target, so the relevant evidence can extend beyond the target page. Documents also contain many same-type distractors: each instance contains a median of 11.5 same-type objects, with a median non-matching ratio of 80.9%. Thus, inferential variants require agents to decide which contextual objects are relevant before determining the attributes to edit.

Results and Analysis

Experimental Setup

We evaluate the benchmark on six coding-agent systems: Claude Code [2] with Opus 4.6 [3], Claude Code with Sonnet 4.6 [4], Codex [13] with GPT-5.5 [15], Codex with GPT-5.4 [14], Claude Code with GLM 5.2 (Z.ai, 2026), and Kimi CLI [11] with Kimi K2.6 [12] (see Appendix § 8 for exact versions and environment configuration). Each system runs in an isolated Windows sandbox. A fresh sandbox is provisioned for each variant run; no execution history, cache, or workspace state is shared between runs, including among variants of the same base instance. Each system autonomously chooses its editing approach and executes modifications; the output document is scored against the held-out reference through Word COM.

We use two metrics: Task-Level Pass Rate, where a task passes only when all evaluated changed attributes of the target object are correct; and Attribute-Level Accuracy, which measures the fraction of target attributes requiring modification that are correctly updated in each task, macro-averaged across tasks. The evaluator compares the target attributes that differ between the input and reference documents. Both the reference and prediction documents are processed by the same COM-based extraction pipeline, which normalizes attribute values into canonical string representations with standardized units, maps COM enum constants to human-readable names, and treats sentinel values such as wdUndefined as absent. Because extraction reads the effective property values resolved by Word’s rendering engine, attributes set via style inheritance and direct formatting are compared equivalently. Prediction files that cannot be opened are scored as zero.

Beyond target correctness, we separately evaluate whether model outputs preserve non-target document content. Target correctness and non-target preservation are scored independently. The preservation evaluator compares supported non-target objects and their contents and formatting between the reference and model-generated documents. An output passes the preservation check only when no differences are detected in any evaluated non-target object.

Evaluation Results

Table 2 presents the overall results. All evaluated agent systems obtain lower pass rates on inferential variants than on descriptive and referential variants. Even the best-performing system, Claude Code with Opus 4.6, achieves only a 40.7% pass rate on inferential variants.

AgentDescriptive Pass RateDescriptive Attr AccReferential Pass RateReferential Attr AccInferential Pass RateInferential Attr AccAverage Pass RateAverage Attr Acc
Human Baseline50.0%85.1%43.3%75.3%34.0%70.1%42.5%76.9%
Claude Code (Opus 4.6)62.9%82.3%77.1%87.5%40.7%59.3%60.2%76.4%
Claude Code (Sonnet 4.6)45.4%78.9%56.8%71.1%29.3%48.2%43.8%66.1%
Codex (GPT-5.5)55.4%70.0%43.6%58.7%25.7%43.7%41.5%57.5%
Claude Code (GLM 5.2)31.4%51.8%56.4%67.8%20.0%35.9%36.0%51.8%
Codex (GPT-5.4)41.4%60.4%32.1%45.0%23.2%38.0%32.3%47.8%
Kimi CLI (Kimi K2.6)37.5%68.2%41.8%63.4%15.4%28.8%31.5%53.5%

Table 2. Overall results across three instruction conditions. Pass Rate is the proportion of tasks in which all evaluated changed attributes are correct. Attr Acc is the fraction of target attributes requiring modification that are correctly updated in each task, macro-averaged across tasks. Both metrics evaluate only the target object; non-target preservation is reported separately.

Notably, several agents achieve higher pass rates on referential tasks than on descriptive tasks, though this pattern does not hold uniformly across all agents. One possible explanation is that programmatically copying attributes from a reference object can be simpler than parsing detailed natural-language attribute specifications, which requires correctly interpreting human-readable attribute names and values and mapping them to exact API parameters. The inconsistency across agents may reflect differences in their natural-language instruction following and code generation capabilities.

Figure 3 shows average task-level pass rates by object type. Tables have the lowest pass rates across all agents, as they require modifying attributes across multiple cells, where each cell contains border properties, shading, vertical alignment, and text formatting, creating a deep attribute hierarchy that agents struggle to handle completely. Per-attribute accuracy results are provided in Table 6 of Appendix § 9.

Figure 3. Average task-level pass rates (%) by object type across all three instruction conditions.

AttributeNNAcc%AttributeNNAcc%
font_formatting.bold10865.5contrast255.6
font_formatting.color9653.8data_table_show_legend_key741.3
font_formatting.font14859.6fill_back_color_rgb1063.9
font_formatting.font_size15564.1fill_fore_color_rgb4061.9
font_formatting.italic4964.4fill_gradient_angle771.4
font_formatting.strike150.0fill_transparency1558.5
font_formatting.subscript652.8fill_type1861.7
font_formatting.superscript1456.0glow_color_rgb161.1
font_formatting.underline6359.0glow_radius161.1
paragraph_formatting.alignment9164.8has_data_table741.3
paragraph_formatting.indent.hanging8261.9has_legend1053.3
paragraph_formatting.indent.left8646.0has_title1148.5
paragraph_formatting.indent.right3762.6layout_name2348.3
paragraph_formatting.keep_lines_together4768.9line_dash_style450.0
paragraph_formatting.keep_with_next5269.1line_fore_color_rgb3259.4
paragraph_formatting.line_spacing11063.0line_visible2063.1
paragraph_formatting.line_spacing_rule10759.9line_weight3562.9
paragraph_formatting.list.level6159.3quick_style_name1948.5
paragraph_formatting.list.type6257.3reflection_blur359.3
paragraph_formatting.page_break_before1071.7reflection_offset359.3
paragraph_formatting.spacing_after12764.5reflection_size351.9
paragraph_formatting.spacing_before8965.2reflection_transparency359.3
paragraph_formatting.style15863.5rotation651.9
alignment859.7series_info.data_label_position650.9
axes.has_major_gridlines1348.7series_info.has_data_labels653.7
axes.has_minor_gridlines244.4series_info.has_error_bars233.3
brightness255.6series_info.line_color_rgb1548.5
cell_formatting.border.color2747.7shadow_angle2960.2
cell_formatting.border.exists2651.9shadow_distance3061.3
cell_formatting.border.line_style1352.1shadow_visible1962.3
cell_formatting.border.line_width2250.8soft_edge_radius144.4
cell_formatting.shading2252.8style3353.4
hyperlink4857.3text_frame_orientation244.4
cell_formatting.vertical_alignment1751.3threed_bevel_top_depth166.7
cell_margin1942.7threed_bevel_top_type166.7
cell_spacing430.6threed_depth554.4
chart_type1546.3threed_rotation_x155.6
color_name1948.8threed_visible450.0
color_type359.3wordart_kerned_pairs355.6
column_widths.width1128.8wordart_tracking363.0
column_widths.width_rule953.1

Table 6. All 81 evaluated formatting attributes with per-attribute accuracy averaged across all six agents. NN = number of tasks (out of 280) that evaluate each attribute.

We also include a human baseline (Table 2), where 9 annotators complete tasks under the same conditions as agents: given an instruction and an input document, they manually edit the document in Microsoft Word without access to the reference document, and the output is scored using the identical automated evaluation pipeline (see Appendix § 12 for annotation protocol details). Analyzing human errors, we find that annotators generally execute inindividual attribute modifications correctly but tend to overlook some of the required attributes. As a result, their attribute-level accuracy is comparable to or slightly better than the best agent (76.9% vs. 76.4%), yet their task-level pass rate is lower (42.5% vs. 60.2%), since pass rate requires all attributes to be correct and a single omission fails the entire task. Appendix § 13 compares common attribute errors in human and agent outputs.

Preservation results show that collateral changes can occur even when editing is directed at a specific target. For example, Claude Code with Opus 4.6 achieves a preservation pass rate of 89.6% under the inferential condition. Common failure types include changes to textual content, formatting changes propagating to unrelated objects, and structural changes to non-target objects. Detailed results by model and instruction condition are provided in Appendix § 16.

Sources of Difficulty in Inferential Editing

To probe different sources of difficulty, we conduct two diagnostic experiments using Codex with GPT-5.5. The exemplar-selection diagnostic requires only an object selection and is evaluated with localization accuracy, whereas the property-path diagnostic performs document editing and is evaluated by exact target-attribute matching.

First, we isolate exemplar localization, one component of contextual inference. GPT-5.5 receives the input document and inferential instruction but performs no document modification and is not given a candidate list. It selects a same-type object as the formatting exemplar. If multiple same-type objects have the required target formatting, selecting any of them is counted as correct. GPT-5.5 achieves an exemplar-selection accuracy of 81.2%. This diagnostic measures only exemplar localization, not property-path identification or document editing. This result indicates that exemplar inference itself remains a source of error even when editing execution is excluded.

Second, starting from the referential setting, where the exemplar is explicitly provided, we additionally provide the property paths to be modified without revealing their target values. The agent must infer the values from the exemplar and produce the edited document. Providing the property paths increases the task-level pass rate from 43.6% to 45.4%. Thus, under this setting, explicit property paths yield only a small improvement in target-object modification. Separately, in the descriptive condition, both the properties and their target values are explicitly specified, yet the same agent achieves a pass rate of only 55.4%. This shows that failures remain even when contextual exemplar and property discovery are not required. Taken together, these results indicate that inferential-task failures reflect both errors in exemplar localization and failures to correctly apply the required formatting changes.

Failure Analysis

Qualitative inspection of failed agent trajectories revealed the following illustrative failure modes:

Improper editing methods causing file corruption or ineffective modifications. A Word document is essentially a ZIP archive containing XML files. In inspected failures, some agents unzip the document, modify the internal XML, and repackage it rather than editing through Word’s programming interface. This approach can produce a malformed package even when the local XML edit appears correct. For example, an XML library may rename namespace prefixes during re-serialization while other locations still reference the original names, causing broken references. Word may then reject the file or silently discard modifications.

Incorrect understanding of API parameter values or semantics. When executing attribute modifications through programming interfaces, agents sometimes misunderstand parameter values or meanings. For example, when setting chart type to “Scatter,” an agent used enum value 74, which actually corresponds to “Scatter with Straight Lines and Markers,” while the correct value for a plain scatter plot is −4169. Similarly, when setting both paragraph left indent and hanging indent simultaneously, agents confuse the meaning of the LeftIndent property, which actually represents the sum of left indent and hanging indent, not the left indent alone.

Incomplete attribute coverage. In referential and inferential tasks, agents must identify which attributes to modify on their own. Selected trajectories show cases where an agent locates the appropriate exemplar but omits some required attributes. The observed examples include complex nested objects for which the agent modifies outer attributes but neglects per-run formatting within paragraphs, per-cell styles in tables, or internal text formatting within shapes. Descriptive instructions specify these attributes, removing the discovery requirement, though execution may still fail.

Limited Visual Understanding of Document Formatting

In referential and inferential tasks, agents must autonomously determine which attributes of the target object need to be modified. A direct approach is to visually inspect the document context, comparing the target object with other same-type objects to infer the required attribute changes. We designed an experiment to test whether GPT-5.5 can identify formatting attribute changes through visual inspection.

For each of the 280 input–reference pairs, we render screenshots of up to three differing pages per document and ask GPT-5.5 to select the changed attributes from an 81-attribute checklist. We compare its selections with programmatically computed ground-truth attribute diffs. This setting is easier than the actual task, where agents have access to neither the reference document nor an attribute checklist.

GPT-5.5 achieves 0.740 precision, 0.351 recall, and 0.476 F1: its predictions are mostly correct but recover only about one-third of the changes, often missing visually subtle properties such as line spacing, paragraph spacing, and font. Figure 4 in Appendix § 15 shows an example, consistent with prior findings on fine-grained visual perception ([16, 8]). Because coding agents may also inspect document properties through COM or XML, this result does not explain all agent failures, but cautions against relying on screenshot-only attribute detection or vision-only LLM judges without validation.

Example from the visual attribute detection experiment showing an input document, reference document, and detected and missed visual attributes

Figure 4. Example from the visual attribute detection experiment. GPT-5.5 correctly identifies visually salient changes such as fill color and bold weight, but misses subtle attributes like line spacing, spacing after, and shadow parameters whose changes produce barely perceptible visual differences.

Conclusion

We present WordInferBench, a benchmark of 280 base editing instances and 840 instruction-conditioned variants that evaluates coding-agent systems on Word document formatting under descriptive, referential, and inferential instructions. Across six evaluated systems, pass rates are lowest on inferential variants. Qualitative trajectory inspection identifies illustrative bottlenecks including improper editing methods, misunderstanding of API parameter semantics, and incomplete attribute coverage, while a controlled GPT-5.5 experiment shows that screenshot-only inspection misses many fine-grained formatting differences. These findings show that context-aware attribute inference remains a core challenge for current coding-agent systems.

Limitations

Our benchmark currently targets only formatting attribute modifications in Word documents, without covering other Office applications such as PowerPoint and Excel. Our three-level instruction design and attribute-comparison-based evaluation framework are in principle transferable to other applications, and extending to a broader range of Office application scenarios is a direction for future work.

Additionally, for evaluation convenience, our task design is limited to modifying a single target object per task. In real-world document editing, user requests often involve coordinated modifications across multiple objects, such as uniformly adjusting the formatting of all headings in a document. However, an agent’s ability to modify formatting on a single object is the foundation for completing multi-object coordinated edits, and performance on single-object tasks can reflect its core capability in more complex scenarios.

Ethical Considerations

We use publicly accessible Word documents collected through Common Crawl. The original documents may contain personally identifiable information (PII). To reduce privacy risks, we replace detected PII with placeholders, exclude documents containing detected face photographs, and conduct additional privacy checks during human verification.

Agent Evaluation Configuration

Agent Systems Under Test

Table 3 lists the six agent systems evaluated in our experiments.

Agent SystemUnderlying ModelCLI Tool & Version
Claude CodeClaude Opus 4.6[email protected]
Claude CodeClaude Sonnet 4.6[email protected]
CodexGPT-5.5[email protected]
CodexGPT-5.4[email protected]
Claude CodeGLM 5.2[email protected]
Kimi CLIKimi K2.6[email protected]

Table 3. Agent systems and their configurations.

Execution Environment

Each task run uses a freshly provisioned sandbox. No execution history, cache, or workspace state is shared between runs. Agents have access to PowerShell, Python, and the Word COM automation interface. No network access is provided beyond the agent’s own API endpoint.

Task Prompt

All agents receive the identical task prompt shown below. No additional system prompts, skill libraries, or built-in Office knowledge are injected beyond what each CLI tool provides by default.

You are a Word document formatting
expert.

You need to modify a Word document
(.docx) according to the given
instruction.

### Instruction
{instruction}

### Document Path
{doc_path}

### Output Path
{output_path}

Please follow these steps:
1. First, make a copy of the document
at the document path to the output path,
so you work on the copy.
2. Complete the task described in the
instruction on the document at the
output path.

Input documents are placed in the agent’s sandbox directory with a fixed filename (input.docx). The output path is explicitly specified in the prompt (output.docx in the same directory). Agents do not need to discover or search for files.

Benchmark Details

Table 5 shows the number of tasks per object type in the final benchmark after filtering. Table 6 lists all 81 evaluated attributes with the number of tasks testing each attribute (NN) and the available per-attribute accuracy averaged across all six agents.

Object TypeTasks
Plain paragraph33
Listing31
Table28
Table of contents27
Hyperlink23
SmartArt23
Textbox22
WordArt20
Autoshape19
Chart17
Footer15
Header13
Picture9
Total280

Table 5. Number of tasks per object type.

ComponentSpecification
OSWindows 11 Enterprise 25H2
VM Resources8 GB RAM, 2 cores, 32 GB disk
Per-task Timeout600 seconds
Runs per Variant1; provider-default decoding settings
OfficeMicrosoft Office LTSC 2024
Language / LocaleEnglish (default Windows 11 install)

Table 4. Execution environment specification.

The benchmark documents are selected from Common Crawl content. Copyright in third-party documents and their contents remains with the respective rights holders; the benchmark release does not grant additional rights to these materials. Project-authored task instructions, annotations, and metadata are released under the MIT License.

Data Generation Prompts

All prompts below are sent to GPT-5.2 as user messages with no system prompt.

Instruction Generation (Two-pass)

Pass 1: Structured changes to natural language.

Rewrite the given document modification changes into a clear, natural language instruction:

Requirements:

  1. Describe every change listed. Do not skip any, even when a value is set to 0 or reset to default.

  2. When multiple components (e.g., different runs in a paragraph) change differently, describe each one individually.

  3. When multiple parts share the exact same change, merge them into one statement instead of repeating.

  4. Only return the instruction. Use plain text without markdown formatting.

  5. Always write the instruction in English, but keep document content in its original language.

Pass 2: Refinement.

Refine the following document modification instruction to be more natural and concise:

Rules:

  1. Do not repeat the same location in every sentence. State each location once, then describe all its changes together.

  2. Do not drop or add any changes—the content must remain exactly the same.

  3. Only return the rewritten instruction. Use plain text without markdown formatting.

  4. Always write the instruction in English, but keep document content in its original language.

Instruction to refine:

Referential Instruction Generation

Rewrite the given reference-based document modification task into a clear, natural language instruction.

Requirements:

  1. Describe which object needs to be modified and which object is the reference.

  2. The instruction should ask to make the target object consistent with the reference object.

  3. Do NOT specify or guess which attributes to change. Just describe the target and reference objects.

  4. Only return the final instruction text. No markdown, bullets, or headers.

  5. Always write the instruction in English, but keep document content in its original language.

Inferential Instruction Generation

Rewrite the given document modification task into a clear, natural language instruction.
Requirements:
1. Describe which object needs to be modified.
2. The instruction should ask to review the document and infer the appropriate modification for the target object.
3. Do NOT specify or guess which attributes to change.
4. Only return the final instruction text. No markdown, bullets, or headers.
5. Always write the instruction in English, but keep document content in its original language.

PII Filtering

PII detection is performed at both image and text levels using GPT-5.2.

Text PII Detection

Documents are scanned for 8 categories of PII: account numbers, private addresses, private emails, private person names, private phone numbers, private URLs, private dates, and secrets (credentials/tokens/keys). Detected PII text is replaced with category-specific placeholders.

Analyze the following text extracted from a document. Determine whether it contains any PII (Personally Identifiable Information).

Classify each PII finding into one of the following categories:
1. account_number — Financial or government-issued account identifiers.
2. private_address — A physical address specific enough to locate a private residence.
3. private_email — An email address belonging to a specific private individual.
4. private_person — A real person’s name in a private or personal context.
5. private_phone — A telephone number belonging to a specific individual.
6. private_url — A URL that exposes personal information.
7. private_date — A specific date tied to an identifiable individual.
8. secret — Sensitive credentials, tokens, or keys.

Respond with ONLY a JSON object (no markdown):
{"has_pii": true/false, "pii_items": [{"category": "<category>", "text": "<exact PII substring>"}, ...]}

Text to analyze:

Face Detection in Images

Document pages are rendered as screenshots and checked for human face photographs.

Look at this document page screenshot. Determine whether it contains any human faces (photographs, ID photos, headshots, etc.).

Do NOT flag:
- Clip art, icons, or cartoon illustrations
- Logos that happen to contain stylized faces
- Very small, unrecognizable figures in diagrams

Only flag clear, identifiable photographs or images of real human faces.

Respond with ONLY a JSON object (no markdown):
{"has_face": true/false}

Human Annotation

Annotators

We recruit 9 crowdsourced annotators, each of whom is compensated for their work and provides informed consent. The data collection protocol, including human annotation, was approved by the relevant institutional ethics review board. The same 9 annotators perform both data verification and human-baseline editing. Based on annotation results, filtering removed 9 instances for PII, 15 for problematic references, and 12 for unclear instructions (with 4 overlapping between the latter two categories), yielding 32 unique removals and 280 remaining base instances.

Task Allocation

Annotators are divided into 3 groups of 3 (groups A, B, and C), and the 312 candidate base instances are partitioned into 3 balanced subsets of 104 instances. Each annotator completes one instruction variant for every base instance, with 104 tasks under each instruction condition, and never sees multiple instruction variants of the same base instance. The subset-to-condition mapping is rotated across groups, as shown in Table 7, so that group effects are balanced across conditions. Because each group contains 3 annotators, every base-instance–condition pair receives 3 independent human outputs. This yields 2,808 candidate annotations and 2,520 annotations after benchmark filtering, with 840 retained annotations per condition. Tasks are presented in a randomized order.

GroupSubset 1Subset 2Subset 3
AReferentialDescriptiveInferential
BDescriptiveInferentialReferential
CInferentialReferentialDescriptive

Table 7. Counterbalanced assignment of instruction conditions to annotator groups and base-instance subsets.

Editing Time

Annotators face no enforced per-task time limit. Editing time is measured from opening the editable document to declaring the edit complete; time spent on the subsequent evaluation questionnaire is excluded. After benchmark filtering, each instruction condition contains 840 timed annotations, corresponding to 280 base instances completed by 3 annotators. Table 8 reports the resulting time statistics.

ConditionMeanMedian
Descriptive424301
Referential362230
Inferential324230
Overall370253

Table 8. Human editing time in seconds by instruction condition. No per-task time limit is enforced.

Annotation Flow

Each annotation proceeds in two phases: editing and evaluation. We provide the participant-facing editing instructions, questionnaire prompts, and response definitions below. Before starting, annotators see:

Please read the annotation guide (README.docx) before starting. Enter your name and select your assignment to begin.

Phase 1: Editing. The interface displays the task’s editing instruction and the following directions:

Read the instruction below. Click Start Editing to open the document in Word (may take a moment to open). Follow the instruction to modify the document, save the file (Ctrl+S), then return here and click Done Editing.

Clicking “Start Editing” opens a copy of the input document in Microsoft Word and starts the timer. Annotators can use Word functionality such as formatting panels and format painter. They have no access to the reference document during this phase and face no time limit. Clicking “Done Editing” ends the editing timer and opens the evaluation phase.

Phase 2: Evaluation. The interface displays the task’s editing instruction again and provides “Re-open My Submission” and “Open Reference Answer” buttons. It asks annotators to rate the instruction and input document, compare their edited document with the reference, and report their overall confidence. The participant-facing prompts and response definitions are:

Rate the following aspects, then compare your submission with the reference answer.

Instruction Clarity. How clear, understandable, and executable is the editing instruction?

1 — Very Unclear: Completely incomprehensible or impossible to execute.

2 — Unclear: Mostly hard to understand; only a few points are executable.

3 — Acceptable: Somewhat understandable; average clarity.

4 — Clear: Mostly clear and executable; only minor points are unclear.

5 — Very Clear: Perfectly clear with no issues in understanding or execution.

Task Difficulty. How much time and effort is needed to complete this task?

Low: Simple and effortless; requires almost no effort.

Medium: Moderate difficulty; requires a fair amount of time and effort.

High: Very challenging; requires significant time and effort.

Input Document Quality. Rate the visual presentation and richness of the input document.

1 — Very Poor: Almost no substantive content, or extremely messy formatting.

2 — Poor: Thin content or poor visual presentation.

3 — Acceptable: Average content and visual presentation.

4 — Good: Fairly rich content with good visual presentation.

5 — Very Good: Rich content with excellent visual presentation.

Does the document contain PII (personally identifiable information)? e.g. real names, emails, phone numbers, addresses, ID numbers.

Yes: The document contains real PII.

No: The document does not contain any PII.

Unsure: Cannot determine whether certain information is real PII.

Is the reference answer correct? Judge whether the reference itself correctly fulfils the instruction, regardless of your own approach.

1 — Incorrect: The reference has clear errors and does not correctly follow the instruction.

2 — Partially Correct: The reference is partly correct but has omissions or errors.

3 — Correct: The reference correctly fulfils the instruction.

Unsure: Cannot determine correctness (e.g. the instruction itself is unclear).

If not fully correct, what is wrong with the reference?

Is your result consistent with the reference? Compare your submission with the reference, regardless of which is more correct.

1 — Very Different: Your result differs significantly from the reference.

2 — Partially Similar: Your result partially matches the reference with notable differences.

3 — Identical: Your result is identical or nearly identical to the reference.

If not fully consistent, what differs?

How confident are you in your annotations above? Your overall confidence in all the evaluations above.

Low: Uncertain about multiple evaluations; may need re-review.

Medium: Confident in most evaluations; uncertain about a few.

High: Confident in all evaluations.

Common Attribute Errors

We compare attribute errors across all scoreable outputs from the 9 human annotators and 6 agent systems after benchmark filtering. For each output document and attribute, we compute the proportion of its evaluated values that match the reference, and macro-average this accuracy across documents involving that attribute. Table 9 reports the 10 lowest-accuracy attributes for each group. For human outputs, we include attributes evaluated in at least 30 documents. Column width, cell margin, and cell spacing appear among the 10 lowest-accuracy attributes for both groups.

RankHuman AttributeHuman Acc.Human NNAgent AttributeAgent Acc.Agent NN
1Column width20.599Column width36.0174
2Object style22.9297Cell spacing36.161
3Cell margin28.8171Series error bars38.731
4Object alignment37.572Data-table legend key46.4112
5Hyperlink40.6429Chart data table46.4112
6Series line color41.6135Soft-edge radius47.117
7Cell vertical alignment43.0153Cell margin49.3297
8Cell spacing44.436Layout name49.6403
9Data-label position45.854Quick style49.7334
10Cell shading46.6166Minor gridlines50.032

Table 9. Lowest macro-averaged attribute accuracies (%) for human and agent outputs. NN is the number of scoreable output documents in which the attribute is evaluated.

Heuristic Baselines for Inferential Variants

Inferential variants have lower pass rates than referential variants. The core difference is that inferential variants require agents to identify which object in the document serves as the formatting exemplar, whereas referential variants directly provide this information. To characterize how much of this gap may stem from exemplar localization, we design two heuristic baselines that measure the localization ability of naive strategies.

For each target object, both baselines select a same-type candidate from the same document and check whether its attributes exactly match the ground truth. Nearest Neighbor selects the candidate closest in document order. Majority Vote computes a formatting signature for each same-type candidate, selects the one belonging to the most frequent signature group, and falls back to nearest neighbor when no clear majority exists. Nearest Neighbor achieves 50.6% and Majority Vote achieves 45.8%. The superiority of nearest neighbor over majority vote indicates that spatial proximity provides a stronger signal than global format consensus—same-type objects in a document do not always share the same formatting. For example, when the target is a heading paragraph, the majority format among same-type candidates likely corresponds to body text rather than headings, causing majority vote to select incorrectly.

Note that these heuristic baselines only evaluate reference object localization accuracy (whether the selected candidate’s attributes match the ground truth), without performing actual document editing. They are therefore not directly comparable to end-to-end agent pass rates, which additionally require correct execution of attribute modifications.

Visual Attribute Detection Experiment

For each of the 280 document pairs, we first identify the pages where the input and reference documents differ and render both versions as screenshots, retaining at most three pages per document. GPT-5.5 receives these screenshots together with an 81-attribute checklist and selects every attribute that differs visually. We evaluate its selections against ground-truth attribute diffs computed programmatically from document properties. Because the model can directly compare the input with the reference and choose from a fixed checklist, this controlled setup narrows the judgment space and is easier than the actual editing scenario.

The example in Figure 4 reflects the aggregate precision–recall pattern. GPT-5.5 usually identifies visually salient changes such as color and bold weight, but often overlooks subtler differences in line spacing, paragraph spacing, font, and shadow parameters. Its high precision indicates that detected changes are generally correct, whereas its low recall shows that screenshot inspection recovers only about one-third of the actual changes. Because coding agents may also inspect document properties programmatically through COM or XML, these results should be interpreted as evidence of the limitations of screenshot-only attribute detection and vision-only LLM judges, rather than as a complete explanation of agent failures.

Preservation Results

Table 10 reports preservation pass rates for each model and instruction condition. A variant passes only when no differences are detected in any evaluated non-target object.

AgentDesc.Ref.Infer.Avg.
Claude Code (Opus 4.6)92.190.089.690.6
Claude Code (Sonnet 4.6)92.188.686.489.0
Codex (GPT-5.5)65.452.970.062.7
Claude Code (GLM 5.2)91.886.492.590.2
Kimi CLI (Kimi K2.6)84.685.485.485.1
Codex (GPT-5.4)54.651.860.755.7

Table 10. Preservation pass rates (%) across the three instruction conditions, using all 280 tasks per condition as the denominator. The average is computed over all 840 tasks across the three conditions.

References

  1. [1]Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Keunho Jang, and Zheng Hui. 2025. Windows agent arena: Evaluating multi-modal OS agents at scale. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 4874–4910. PMLR.arxiv.org/abs/2409.08264
  2. [2]Anthropic. 2026a. Claude code. https://claude.com/product/claude-code. Accessed 2026-07-28.
  3. [3]Anthropic. 2026b. Claude opus 4.6 system card. https://www-cdn.anthropic.com/0dd865075ad3132672ee0ab40b05a53f14cf5288.pdf.
  4. [4]Anthropic. 2026c. Claude sonnet 4.6 system card. https://www-cdn.anthropic.com/bbd8ef16d70b7a1665f14f306ee88b53f686aa75/Claude%20Sonnet%204.6%20System%20Card.pdf.
  5. [5]Xin-Sheng Chen, Jiayu Zhu, Pei-lin Li, Hanzheng Wang, Shuojin Yang, and Meng-Hao Guo. 2026. Present-Bench: A fine-grained rubric-based benchmark for slide generation. arXiv preprint arXiv:2603.07244.arxiv.org/abs/2603.07244
  6. [6]Common Crawl Foundation. 2026. Common crawl. https://commoncrawl.org/. Accessed 2026-07-28.
  7. [7]Haoyu Dong, Pengkun Zhang, Yan Gao, Xuanyu Dong, Yilin Cheng, Mingzhe Lu, Adina Yakefu, and Shuxin Zheng. 2026. Finch: Benchmarking finance & accounting across spreadsheet-centric enterprise workflows. In Findings of the Association for Computational Linguistics: ACL 2026, pages 10768–10794, San Diego, California, United States. Association for Computational Linguistics.
  8. [8]Chenglong Liu, Haoran Wei, Jinyue Chen, Lingyu Kong, Zheng Ge, Zining Zhu, Liang Zhao, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. 2024. Focus anywhere for fine-grained multi-page document understanding. arXiv preprint arXiv:2405.14295.arxiv.org/abs/2405.14295
  9. [9]Tengchao Lv, Dongdong Zhang, Jiayu Ding, Yilin Jia, Yuzhong Zhao, Yup'an Huang, Wenshan Wu, Xiangyang Zhou, Shaohan Huang, Nan Yang, Li Dong, Lei Cui, and Furu Wei. 2026. Mind the gap: Can frontier LLMs pass a standardized office proficiency exam? arXiv preprint arXiv:2606.10956.arxiv.org/abs/2606.10956
  10. [10]Zeyao Ma, Bohan Zhang, Jing Zhang, Jifan Yu, Xiaokang Zhang, Xiaohan Zhang, Sijia Luo, Xi Wang, and Jie Tang. 2024. SpreadsheetBench: Towards challenging real world spreadsheet manipulation. In Advances in Neural Information Processing Systems, volume 37, pages 94871–94908. Curran Associates, Inc.arxiv.org/abs/2406.14991
  11. [11]Moonshot AI. 2026a. Kimi cli. https://github.com/MoonshotAI/kimi-cli. Accessed 2026-07-28.
  12. [12]Moonshot AI. 2026b. Kimi K2.6: Advancing open-source coding. https://www.kimi.com/blog/kimi-k2-6.
  13. [13]OpenAI. 2025. Introducing codex. https://openai.com/index/introducing-codex/.
  14. [14]OpenAI. 2026a. GPT-5.4 thinking system card. https://openai.com/index/gpt-5-4-thinking-system-card/.
  15. [15]OpenAI. 2026b. GPT-5.5 system card. https://openai.com/index/gpt-5-5-system-card/.
  16. [16]Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, and Ah Totti Nguyen. 2024. Vision language models are blind: Failing to translate detailed visual features into words. arXiv preprint arXiv:2407.06581.arxiv.org/abs/2407.06581

Paper details

vlnc:W_r2wdfcsr
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 6

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents