Word editing agents struggle to infer formatting from document context

WordInferBench: Can Agents Infer How to Edit from Word Document Context?

Artificial Intelligence

Summary

Editing documents involves many small details like fonts and colors, but most AI tools follow exact instructions and don’t figure out formatting on their own. The authors created a new benchmark to test if AI agents can look at an example in a Word document and copy the style without being told every detail. They tested several recent AI coding tools and found that these tools do much better when instructions list every formatting change, but perform worse when they must guess from context. This shows current AI isn’t yet good at understanding and applying document styles just by looking.

What this means in practice

  • For software developers: Develop document editing assistants that automatically infer consistent formatting without explicit instructions, improving user experience in office tools.
  • For ai tool builders: Evaluate and improve AI systems’ ability to infer document style from examples, targeting more natural interactions with text editing applications.

Authors

Jiayu Ding, Yupan Huang, Jinyoung Park, Yilin Jia, Wenshan Wu, Tengchao Lv, Shaohan Huang, Yuzhong Zhao, Xiangyang Zhou, Tao Ge, Sun Mao, Xun Wang, Huitian Jiao, Si-Qing Chen, Lei Cui, Nanning Zheng, Furu Wei

Abstract

Document editing is among the most common knowledge-work tasks, yet existing benchmarks for evaluating autonomous agents on Word document editing provide fully specified attribute-modification instructions. They do not evaluate agents' ability to infer the appropriate attributes from document context, a capability essential in real-world editing where users expect consistency without spelling out every attribute. We introduce WordInferBench, a benchmark comprising 280 base document-editing instances, each paired with three instruction conditions: descriptive (the instruction explicitly lists every attribute change), referential (the instruction points to another object as an exemplar), and inferential (the agent must examine document context to infer what attributes to apply). To evaluate contextual attribute inference, each base instance is constructed so that the target formatting is recoverable from an in-document exemplar. The instances are derived from real Word documents collected from Common Crawl and cover 13 object types and 81 attributes. We evaluate six recent coding-agent systems and find that the best-performing evaluated system, Claude Code with Opus 4.6, achieves a 62.9% task-level pass rate on descriptive variants but only 40.7% on inferential variants, indicating that current systems complete tasks less reliably when the required formatting changes must be determined from document context.