Llm agents benchmarked on complex patent tasks reveal key weaknesses
PatentAgentBench: Benchmarking LLM Agents with Professional Patent Task Suites
Artificial Intelligence
Summary
Patent work is tricky because it mixes technology, law, and language. Some new AI systems called large language models (LLMs) can do certain business tasks and even patent-related work. The authors created a detailed test called PatentAgentBench to see how well these systems do on real patent tasks like searching for new inventions, checking if patents are safe to use, writing patent documents, and responding to official patent office actions. They found that current AI systems still have trouble understanding and performing these patent tasks reliably.
What this means in practice
- •For patent professionals: Evaluate AI tools’ effectiveness across realistic patent tasks including search, drafting, and legal responses to guide automation decisions.
- •For legal technology developers: Develop better AI assistants for patent workflows by testing improvements against a comprehensive, diverse set of real patent challenges.
Authors
Yijie Hu, Wei Li, Junjie Zhou, Zhiqi Jiang, Yi Jin, Licong Xu, Xinkai Sun, Kai Duan, Lisha Zhang
Abstract
Patent practice lies at the intersection of technical, legal, and linguistic work, making it challenging yet crucial to technological innovation. Recently, rapid advances in large language models (LLMs) and agentic systems have demonstrated the potential to complete realistic business workflows and even patent works. This raises a question: can current AI systems perform patent work reliably? Existing benchmarks cover a limited range of patent tasks and simplify evidence acquisition by providing reference materials or candidate documents. In this paper, we introduce PatentAgentBench, a benchmark comprising 510 instances across eight task suites: novelty search, freedom-to-operate (FTO) search, design FTO, invalidity search, claim charting, patent drafting, office-action response, and patent translation. Curated from real patent records and professional materials, the benchmark combines task-specific metrics with expert-reviewed annotations and rubrics. Retrieval tasks such novelty search, FTO, etc., require agents to search and compare patent collections without a supplied candidate pool. The drafting and OAR tasks assess the quality of agents' work given technical disclosure and office action notice. We evaluate seven LLMs across four agent harnesses on PatentAgentBench, finding that current systems still struggle to identify novelty, assess risk, and understand practitioner intent throughout patent workflows.