LLM judges improve AI patent drafting with iterative feedback loops
Vibe Patenting: Evaluating LLM Judges for Professional Patent-Drafting Agents
Artificial IntelligenceMachine LearningMultiagent Systems
Summary
Creating patents is complicated, and using AI to help requires careful checking. This work studies how AI judges—special language models—can evaluate and suggest improvements to AI-created patent drafts. The judges guided the drafts to get better over several rounds, helping simpler AI reach results like more advanced ones. However, when compared to a real patent attorney, the judge’s feedback matched only to a certain extent, showing there are still limits to relying on AI judges for professional tasks.
What this means in practice
- •For patent law firms: Use LLM judges to guide AI tools that draft and refine patent applications, reducing need for costly high-expertise AI agents.
- •For legal technology developers: Develop patent drafting assistants that incorporate iterative LLM judge feedback to improve draft quality efficiently.$Commercial implications: Enables selling AI-powered patent drafting software with built-in evaluation and revision capabilities to law firms.
Authors
Toshiaki Koike-Akino, Vlad Blaykhman, Ye Wang, Jing Liu, Gene V. Vinokur
Abstract
LLM judges are increasingly used to evaluate and improve AI-generated outputs, yet their reliability for complex professional work remains unclear. We study this problem through Vibe Patenting, an end-to-end patent-drafting testbed for AI-agent evaluation. A separately-invoked LLM judge evaluates generated patent drafts and provides structured feedback for iterative revision. Across multiple inventions and drafting-agent configurations, judge-guided revision consistently improves judge-assessed quality, while unguided revision tends to saturate. Notably, iterative judge feedback enables a low-reasoning agent to approach the performance of a substantially more expensive high-reasoning agent. Stronger models and increased reasoning generally improve judge-assessed drafting quality, while domain-specific agentic workflows provide further gains. We validate the judge against independent evaluation by a professional patent attorney and find meaningful but strongly metric-dependent agreement and systematic calibration differences. These results highlight both the utility and limitations of LLM judges as evaluators and optimization signals for complex professional workflows.