Multi level modeling improves reasoning in language models
Enhancing Language Model Reasoning with Structured Multi-Level Modeling
Computation and Language
Summary
Language models try to solve hard reasoning problems by thinking through steps, but their plans can go off track, especially if they're small or limited. The authors suggest breaking this process into two parts: a high-level planner that sets goals and a low-level executor that follows those goals with detailed steps. They also created a new training method that teaches the model step-by-step, making it better at sticking to good plans. Their approach works better on tough math and logic tests, and keeps its performance longer on big problems.
What this means in practice
- •For ai system developers: Build language models that generate more reliable step-by-step reasoning in complex tasks by using multi-level planning and execution.
- •For automated tutoring platforms: Improve explanations and solution reasoning in educational AI tools handling math and science problems by adopting structured, multi-level modeling.
Authors
Siheng Xiong, Ali Payani, Faramarz Fekri
Abstract
Inference-time scaling enhances a model's reasoning by extending its chain-of-thought (CoT). However, existing approaches typically rely on a single policy trained with outcome-reward reinforcement learning (RL), which often suffers from long-horizon plan failures where the implicit plan drifts from valid strategies, especially for small LMs with limited capacity. To address this, we propose Multi-Level Reasoning (MLR), which reformulates long-CoT generation as a two-level stochastic process. A high-level planner generates structured step descriptors specifying both the reasoning mode and the semantic subgoal. The low-level executor then produces detailed reasoning conditioned on these descriptors, forming an alternating plan--execute loop. To maintain scalability, we adopt a minimal design where the base model serves as the low-level policy and a lightweight LoRA module implements the high-level policy. For training, we observe that outcome-reward RL provides sparse and delayed feedback for long trajectories (e.g., several thousand tokens), hindering credit assignment. We therefore introduce iterative Step-DPO, a process-level preference optimization scheme that leverages Twisted Sequential Monte Carlo (TSMC) to provide scalable stepwise supervision. This yields more effective training, improved stability, and higher accuracy. Extensive experiments on challenging math, science, and logical reasoning benchmarks show that, under the same reduced data budget (10% SFT and 5% preference relative to the DeepSeek-R1 distillation setup), MLR outperforms both SFT-based distillation and strong RL/preference-optimization baselines across multiple base models and tasks. Moreover, MLR exhibits slower performance degradation on long-horizon reasoning, demonstrating stronger robustness under extended CoT generation.