Llm verification system resolves disagreements in long complex tasks
VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks
Artificial IntelligenceMultiagent Systems
Summary
As language models try to solve big, complicated tasks, checking that their answers are correct is very hard. The paper shows that when multiple model runs disagree, the different answers can help find the truth, while when they agree, mistakes may hide. So the authors created VeriHarness, which helps the model check its own work by examining evidence and testing claims. This method improves how well the final answers are chosen and fixed in five different challenges. VeriHarness can even learn to verify better over time by learning from what went wrong.
What this means in practice
- •For ai system developers: Improve accuracy and trustworthiness of complex AI-generated plans by using disagreement-based verification and evidence checking.
- •For enterprise automation teams: Enhance automated decision systems with multi-step output verification to reduce errors in long processes without needing predefined test answers.
Authors
Caiqi Zhang, Rujun Han, Zifeng Wang, Zoey CuiZhu, Nigel Collier, Tomas Pfister, Chen-Yu Lee
Abstract
As LLM agents undertake increasingly complex, long-horizon tasks, verifying their outputs becomes increasingly challenging. We study how verification capability can be strengthened with a fixed base model, without access to reference answers or grading rubrics at test time. Repeated sampling yields multiple rollouts that can contain complementary correct claims, but we need a reliable verification mechanism to determine which claims to trust. We first find that disagreement often exposes correct alternatives, while consensus can conceal errors. These observations motivate VeriHarness, which turns the underlying LLM a generator uses into an agentic verifier by giving it a workspace, evidence tools, and reusable verification skills. A disagreement resolver checks competing claims against environmental evidence, while a consensus challenger tests shared claims and searches for omitted requirements. Their findings guide the selection and revision of the final artifact. Across five long-horizon workspace benchmarks and two frontier models, VeriHarness achieves the highest selection scores among the evaluated baselines. Evidence-backed revision further improves average performance, bringing gains over a single rollout to 6.2 points with Gemini 3.5 Flash and 6.4 points with Claude Opus 4.8. We further show that verification skills can self-improve from failure feedback, demonstrating VeriHarness as a novel and critical approach for scaling long-horizon agentic verification. We release the full pool of approximately 26,000 rollouts across all five benchmarks and both models, produced at a cost of over $100,000, to support future research on agentic verification.