Heterogeneous fpga and gpu speed up speculative reasoning in large models
HeteroReason: Heterogeneous FPGA-GPU Acceleration for Disaggregated Speculative Reasoning
Hardware Architecture
Summary
Large Reasoning Models use multiple steps to solve problems, but if early steps go wrong, the final answer suffers. This paper proposes a new way to let the system backtrack and fix earlier mistakes, making reasoning more reliable. It also splits work between two types of processors—FPGAs and GPUs—to run these steps faster and save energy. The authors show that their method improves accuracy and reduces processing time compared to doing everything on GPUs alone.
What this means in practice
- •For ai system engineers: Design inference systems that use FPGA-GPU combinations to run reasoning models faster and more energy efficiently by allowing backtracking and parallel workflows.
- •For hardware acceleration designers: Create specialized hardware-software co-designs to offload lightweight model tasks to FPGAs and maintain heavier models on GPUs, improving resource use and reducing latency.
Authors
Zehuan Zhang, Quan Deng, Zibo Ren, Hao Mark Chen, Guoyu Li, Xuchun Hu, Jose G. F. Coutinho, Ce Guo, Wayne Luk, Zhiqiang Que, Hongxiang Fan
Abstract
Large Reasoning Models (LRMs) have achieved state-of-the-art performance in reasoning tasks by utilizing Chain-of-Thought (CoT) reasoning. To achieve fast execution speed, speculative reasoning techniques adopt a lightweight draft model for candidate token generation followed by process reward models (PRMs) for verification and a strong target model for refinements. This paper identifies that the existing speculative reasoning paradigm follows a strictly forward-only reasoning trajectory, which lacks robustness and can lead to severe error propagation if early reasoning steps are suboptimal. Furthermore, executing these disparate inference schemes, including sequential drafting and parallel verification on homogeneous GPU platforms, can lead to severe resource underutilization. To address this, we propose HeteroReason, an algorithm-hardware co-designed heterogeneous FPGA-GPU inference paradigm specifically tailored for LRM speculative reasoning. At the algorithmic level, we introduce a backtracking-enhanced workflow that enables the system to recover from low-quality states and explore alternative reasoning trajectories, significantly improving reasoning robustness. At the system level, the draft model is offloaded to the FPGA while deploying the PRM and target models on GPUs. A specialized workflow is optimized to achieve prefill-decode disaggregation, which exploits shadow synchronization to overlap GPU-side refinements with FPGA-side token updates to effectively hide synchronization latency. To mitigate inherent sequential constraints, we propose a step-ahead speculation and refinement scheduling scheme, transitioning the system from a sequential execution scheme to a parallel pipeline. Experimental evaluations show an average 4.2% accuracy improvement, with 1.01x-1.42x latency speedups and 1.25x-1.57x improvements in energy efficiency compared to homogeneous GPU baselines.