MADA-RL: Multi-Agent Debate-Aware Reinforcement Learning for Parameter-Efficient Reasoning in Compact Models
2026-07-20 • Machine Learning
Machine LearningArtificial IntelligenceComputation and LanguageMultiagent Systems
AI summaryⓘ
The authors present MADA-RL, a method that improves small language models by splitting them into two roles: a generator that suggests answers and a critic that evaluates them. Their key idea is a new way for the critic to learn by comparing its reward against the generator's accuracy, encouraging the critic to fix errors rather than just agree. Using this approach, they improved math reasoning accuracy on a 1.5 billion parameter model while only fine-tuning a small part of it, making training more efficient. Although it doesn't outperform the biggest models trained on more data, the authors show their method effectively helps critics learn to correct mistakes.
large language modelscompact modelsmathematical reasoningpost-trainingLoRA adapterscounterfactual critic advantagegenerator-critic frameworkfine-tuningmulti-round debateaccuracy-trainable-parameter Pareto front
Authors
Martino M. L. Pulici, Cuong Xuan Chu, Evgeny Kharlamov, Zifeng Ding, Volker Tresp, Yunpu Ma
Abstract
Large language models achieve strong reasoning performance, but often at prohibitive training cost - a challenge that is especially acute for compact models ($\leq 4 \, \mathrm{B}$ parameters) trained under limited budgets. We introduce MADA-RL, a post-training framework that specializes compact models into generator and critic roles and trains them with a debate-aware learning signal, fine-tuning only a small subset of parameters via LoRA adapters. Our central contribution is a counterfactual critic advantage: a dynamic, role-conditioned baseline that redefines the critic's advantage as its reward minus the generator ensemble's per-instance accuracy. This explicitly optimizes critics to improve over generator consensus rather than to merely reproduce a correct answer, yielding more targeted credit assignment than static mean-reward normalization. At deployment, the specialized agents are composed in a lightweight multi-round protocol. Across five mathematical reasoning benchmarks, MADA-RL raises the accuracy of the DeepSeek-R1-Distill-Qwen-1.5B model from $39.9 \, \%$ to $41.9 \, \%$ ($+2.0$ points, $p < 0.001$) using $16$ times fewer trainable parameters than fully fine-tuned baselines, placing it on the accuracy-trainable-parameter Pareto front. It approaches, but does not surpass, the strongest baselines (DeepScaleR, STILL-3), which are trained on substantially larger datasets; we analyse this gap and the associated inference-time cost directly. A controlled study isolates the source of MADA-RL's gains: the counterfactual advantage produces the highest critic improvement rate of any model evaluated, indicating that trained critics learn to correct generator errors rather than to imitate them.