Does the Competitive Component of Adversarial Self-Play Improve Legal Reasoning? A Controlled Negative Result
2026-08-03 • Artificial Intelligence
Artificial IntelligenceComputation and LanguageMachine Learning
AI summaryⓘ
The authors tested a training method where a student model argues and an adversary tries to refute it, rewarding the student only if its argument holds up with verified citations. They wanted to see if this competition actually helps improve the student's reasoning compared to training without an adversary. Their multiple tests showed no clear benefit from the adversarial setup, with win rates around chance level and early advantages disappearing with more data. They share these honest negative results to highlight challenges in using adversarial self-play in legal reasoning and suggest that the benefit comes more from building clear verification environments than from competition itself.
Adversarial self-playLegal reasoningCitation verifierSurvival rewardArgument generationAdversary modelRobustness metricMulti-teacher curriculaNegative resultsVerification environment
Authors
Miseog Shawn Kim
Abstract
Adversarial self-play is an appealing recipe for legal reasoning: have a student model draft an argument, have an adversary attack it, and reward the student when its argument survives the attack. We designed exactly such a training signal -- a verifiable "survival" reward in which both the student's cited authorities and the adversary's counter-authorities are checked by a citation verifier, so that survival is decided on verified grounds rather than rhetoric, and fabricated citations are automatically neutralized. We then asked a narrow but important question: does the competitive component itself -- the adversary and the survival reward -- add anything on top of an otherwise identical non-competitive training run? Across four independent tests -- a bootstrap comparison, a two-seed replication, a paired per-case adversarial-robustness comparison, and a blinded head-to-head judgment of generated arguments, plus a follow-up pilot with a deliberately strengthened self-play adversary -- the competitive component produced no reliable benefit. The blinded judgment gave a 49% win rate (binomial p approx. 1.000); the strengthened-adversary pilot gave a 50% win rate (32:32, p approx. 1.000). An early apparent +29% advantage reversed and proved to be a small-sample artifact. We report this as an honest negative result. The value of the paper is reproducibility and the sharing of concrete pitfalls: an initially promising metric that inverted on more data, and an adversarial-robustness metric that silently collapsed to plain recall once the adversary stopped citing the same authorities as the gold answer. This null is consistent with, and reconfirms in the legal domain, the conclusion of the companion coding-domain study (Kim, 2026, arXiv:2607.08255) that the value of multi-teacher curricula arises from constructing a verifiable environment rather than from competition itself.