ArgGYM benchmark measures AI defeasible reasoning with verified tasks
ArgGYM: A Procedural, Engine-Verified Benchmark for Structured Defeasible Reasoning
Artificial IntelligenceComputation and Language
Summary
Figuring out how arguments and decisions change with new information is hard for computers. The authors created ArgGYM, a set of tasks and tests that check if AI models can handle arguments that can be challenged and updated. They use a special engine to make sure the answers are right and break down reasoning into smaller steps. Their tests show which parts AI can do well and where it still struggles. ArgGYM helps train and measure models on this kind of flexible thinking.
What this means in practice
- •For ai training teams: Use ArgGYM to train and reliably evaluate AI models on reasoning that involves changing conclusions as new evidence appears.
- •For nlp developers: Incorporate ArgGYM-generated problems to improve natural language understanding systems’ abilities to handle complex argument structures and contradictions.
Authors
İbrahim Ethem Deveci, Funda Tan Çalık, Barış Deniz Sağlam, Duygu Ataman
Abstract
Recent progress in large language model reasoning has been driven by benchmarks and reinforcement learning environments with automatically verifiable rewards, particularly in mathematics, code, and formal logic. These settings make model accuracy easier to evaluate and optimize, but it remains unclear how far success under fixed problem specifications and stable evaluation criteria transfers to reasoning outside such domains. Real-world reasoning often proceeds under incomplete and revisable information: conclusions may be supported provisionally, defeated by counter-evidence, reinstated by further arguments, or revised when stronger reasons become available. Reasoning of this kind is generally referred to as defeasible reasoning. We introduce ArgGYM, a procedural benchmark and RLVR-compatible training environment for structured defeasible reasoning. ArgGYM decomposes this reasoning into twelve tasks and grounds task-specific scoring in a symbolic argumentation engine that computes the formal states used to evaluate model outputs. It includes a frozen benchmark of 1,440 verified instances across fifteen curriculum configurations, two argument preference orderings (weakest-link and last-link), and two set orderings (elitist and democratic), while the same generators and verifiers can produce fresh instances for evaluation that reduces dependence on static test sets and for verifiable-reward training. On the frozen benchmark, frontier and open-weight models show sharply different reasoning profiles: they can recover substantial parts of structured answers without solving the complete task, and performance declines in later curriculum configurations with longer dependencies and more interacting structures. We release the benchmark, generators, and verifiers for reproducible evaluation and RLVR training.