Papers for

patent law firms

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

LLM judges improve AI patent drafting with iterative feedback loops

Vibe Patenting: Evaluating LLM Judges for Professional Patent-Drafting Agents

Abstract: LLM judges are increasingly used to evaluate and improve AI-generated outputs, yet their reliability for complex professional work remains unclear. We study this problem through Vibe Patenting, an end-to-end patent-drafting testbed for AI-agent evaluation. A separately-invoked LLM judge evaluates generated patent drafts and provides structured feedback for iterative revision. Across multiple inventions and drafting-agent configurations, judge-guided revision consistently improves judge-assessed quality, while unguided revision tends to saturate. Notably, iterative judge feedback enables a low-reasoning agent to approach the performance of a substantially more expensive high-reasoning agent. Stronger models and increased reasoning generally improve judge-assessed drafting quality, while domain-specific agentic workflows provide further gains. We validate the judge against independent evaluation by a professional patent attorney and find meaningful but strongly metric-dependent agreement and systematic calibration differences. These results highlight both the utility and limitations of LLM judges as evaluators and optimization signals for complex professional workflows.

Fri 11 SeptArtificial IntelligenceMachine LearningMultiagent Systems
The gist
Creating patents is complicated, and using AI to help requires careful checking. This work studies how AI judges—special language models—can evaluate and suggest improvements to AI-created patent drafts. The judges guided the drafts to get better over several rounds, helping simpler AI reach results like more advanced ones. However, when compared to a real patent attorney, the judge’s feedback matched only to a certain extent, showing there are still limits to relying on AI judges for professional tasks.
Open → 2609.13422v1