Programs guide fast reasoning in frozen judgement language models
ABACUS: Program-Guided Multistep Reasoning over Frozen Judgment Models
Artificial Intelligence
Summary
Large language models often decide answers by scoring possible options, but combining these small decisions for complex reasoning is tough. The authors present ABACUS, a method that uses fixed "judgment models" together with reusable programs to build step-by-step reasoning processes efficiently. Their approach shows improved accuracy and much faster response times on hard language tasks compared to direct answers from the models. This means complex reasoning can be done more reliably and quickly without changing the underlying model itself.
What this means in practice
- •For natural language engineers: Deploy compact judgment language models combined with reusable programs to achieve fast and highly accurate multi-step reasoning on complex tasks.
- •For automated question answering teams: Use program-guided frozen models to answer complex semantic questions quickly without retraining large language models.
Authors
Yuhao Wang, Zelin Zang
Abstract
Judgment models answer semantic questions by scoring a finite set of candidate outcomes. Can reusable programs turn these local decisions into powerful multistep reasoning? We introduce \method, a framework that jointly constructs judgment schemas and composition programs for frozen judgment models. A generative teacher designs the procedure offline; the judgment model resolves local semantic questions online, and the saved program composes its decisions into an answer. We study three allocations of design effort between humans and the teacher, focusing on construction through a shared interface. We instantiate the framework with Kev. On three selected BIG-Bench Hard tasks, nine independently constructed programs solve all 450 executions of 150 development inputs. A separate deployment study freezes three programs after development-set reference-agreement refinement and evaluates them on 600 held-out questions. These programs achieve near-perfect 99.67\% macro accuracy, versus 64.00\% for direct Kev and 68.80\% for final-answer-only Qwen3.5-4B. They attain comparable observed accuracy to Qwen's unrestricted response policy (99.07\%) with an \textbf{8.43-fold online speedup}: 0.255 versus 2.150 seconds per question. These results show how reusable programs can substantially expand the reasoning capability of a compact judgment model while keeping its parameters fixed.