Agentic reasoning caches speed up problem solving on CPUs
CPU-ARC: Agentic Reasoning Caching for on-CPU Reasoning Reuse
Artificial Intelligence
Summary
Solving problems with large language models (LLMs) usually means starting from scratch every time, which wastes time and computing power. The authors show that once a problem’s solution is figured out, it can be reused for similar problems with much less computing effort. They introduce a method to build reasoning caches that run efficiently on ordinary CPUs by storing reasoning steps from LLMs. These caches work as well as top LLMs on hard math and science tasks, are explainable, can improve over time, resist small input changes, and cost much less to use.
What this means in practice
- •For software engineers: Reduce the cost and runtime of complex reasoning tasks by using CPU-executable caches instead of repeated calls to large language models.
- •For cloud service providers: Offer cheaper and faster reasoning services by integrating reasoning caches that avoid running expensive large language models on every request.$Commercial implications: Enables new cost-effective AI reasoning services that run on standard CPUs, making it viable to sell efficient AI-powered APIs.
Authors
Peter Belcak, Greg Heinrich, Saurav Muralidharan, Shizhe Diao, Pavlo Molchanov
Abstract
Large language models reinvent the wheel all the time. They conflate task-solving with the execution of the solutions, which forces them to solve every problem from scratch. We show that once a task has been reasoned through, the application of its solution to a new problem instance can be done very efficiently and with little involvement from language models. We introduce agentic reasoning caching, a method for constructing robust CPU-executable agentic reasoning caches (ARCs) from LLM reasoning traces. ARCs show on-par performance with frontier LLMs across a range of difficult mathematical and scientific tasks, are fully explainable, continually trainable, robust to input perturbations, and 16-291x cheaper to invoke than the source model.