Harness engineering improves accuracy in converting legal text to XML
A Strong Harness Narrows the Model Gap: A Comparative Evaluation for Legislative Document-to-XML Conversion
Computation and LanguageArtificial Intelligence
Summary
Converting legal documents like laws into computer-readable formats requires perfect accuracy to avoid mistakes that can affect lawyers’ work. The authors show that using carefully designed supporting tools and processes, called harnesses, around AI models helps achieve much better accuracy than relying on the AI model alone. They tested different setups with various models and found that the best harness made even weaker models perform better than the strongest models working alone. The improvements come especially from splitting tasks among specialized agents and using exact lookup tools for things like citations.
What this means in practice
- •For legal technology teams: Develop reliable pipelines that convert legislative texts into exact, verified XML formats for legal research and publishing.
- •For data integration engineers: Implement task-decomposed processing systems with persistent state to improve accuracy when transforming complex hierarchical documents to structured data.
Authors
Palvika Bansal, Rittika Jindal, Ruichong Wang
Abstract
Legal-content reproduction requires matching the source exactly in section hierarchy, verbatim text, citations, and amendment spans; a dropped citation or mislabeled amendment carries into the products lawyers rely on. We report an industrial pipeline converting U.S. state legislation into strict, schema-constrained XML, and hypothesize that its accuracy comes from harness engineering, the deterministic tooling built around a model, more than the model itself. Our controlled ablation crosses three harnesses (zero-shot, a single tool-using agent, and a decomposed multi-agent pipeline) with three Claude models on 153 gold bills from two jurisdictions. The harness narrows the model gap: the weakest model in the full pipeline beats the strongest model alone (0.948 versus 0.806, paired bootstrap p less than 0.001). Because the two agentic harnesses share identical tools, their gap comes from orchestration: decomposition plus the persisted state and per-agent context it brings. Across four scored dimensions, the model spread at zero-shot (0.68 to 0.81) narrows to 0.95 at the full pipeline; the residual sits almost entirely in citation normalization, where the canonical identifier is retrieved from an external authority rather than generated from the weights, so a deterministic lookup tool supplies the exactness that a larger model does not.