Compact language models improve reasoning through teacher feedback loops
Collaborative Reasoning Distillation via Cross-Feedback and Coherent Curation
Machine LearningArtificial IntelligenceComputation and Language
Summary
It’s hard to teach smaller language models to think logically without guessing right answers by chance. The authors present a new way where large models act as teachers and critique each other’s reasoning steps, helping the smaller model learn better logical thinking step-by-step. This method also combines teacher strengths to create clear reasoning and trains smaller models efficiently with limited examples. The result is a small model that solves difficult math problems better than previous compact models.
What this means in practice
- •For ai application developers: Build smaller math-solving AI tools that perform well using fewer training examples by integrating cross-teacher feedback during training.$Commercial implications: Offers a way to create efficient math reasoning AI products for education or testing with fewer data and smaller models.
- •For ai infrastructure teams: Reduce computational costs in deploying reasoning-capable language models by training compact models with reasoning quality optimization and teacher collaboration.
Authors
Taehoon Kim, Seunggeun Cho, Dongsu Han
Abstract
Reasoning capabilities are critical for advancing Large Language Models, yet current approaches either require massive computational budgets or struggle to effectively distill reasoning to smaller models. Standard distillation methods rely on outcome-based rewards, failing to distinguish between sound reasoning and lucky guesses. We propose Collaborative Reasoning Distillation (CRD), a framework that enhances reasoning in compact models through three innovations: (1) interactive cross-feedback where teachers iteratively critique each other's reasoning, (2) fine-grained step-wise quality assessment capturing logical validity independent of final answers, and (3) coherence-aware step stitching that synthesizes complementary strengths. Students are trained via Reasoning Quality Optimization (RQO) with budget constraints. Our model, CRD-4B, achieves 97.3% on MATH-500 and 70.3% on AIME'25, surpassing baselines while using only 50K training examples, up to 12 times smaller than the datasets of comparable models.