Accuracy drops up to 40 points where large language models split tasks
The Decomposition Tax: LLM Pipelines Lose Up to 40 Accuracy Points at Their Own Interfaces
Artificial IntelligenceComputation and LanguageSoftware Engineering
Summary
When large language models (LLMs) solve tough problems by breaking them into steps, their accuracy can drop a lot at the points where the answers pass between steps. The authors call this loss the "decomposition tax," and it can be as high as 40 points on some math problems. They show that letting each step see the original problem again or reminding steps to keep important relationships reduces this loss. This means designing these multi-step systems needs careful attention to how each step shares and recalls information.
What this means in practice
- •For ai system builders: Improve multi-stage language model pipelines by adding re-grounding steps and preserving relationships to reduce accuracy loss at stage interfaces.
- •For software architects: Design modular AI systems with interface-aware instruction tuning to minimize information loss between processing stages in large language models.
Authors
Tianqi Bu, YuXuan Peng, Junteng Tu, Henghui Xiao
Abstract
A four-stage LLM pipeline gives up as much as 40.5 accuracy points at its own interfaces (gemma-3-12B on MATH-500, Holm-corrected p = 1.66e-19; the largest tax in the primary family). We hold model, problem, stages, stage prompts and completion budget fixed, vary only whether each stage can still see the original problem, and call the accuracy difference the decomposition tax. Across 21 open-weight models from nine organisations, on GSM-Hard and MATH-500 at n = 200 paired items per cell, 70 of 118 primary-family tests survive Benjamini-Hochberg correction and 54 survive Holm. On GSM-Hard, a placebo recovers nothing: it carries at least 60% of the extra tokens and at most one word of the problem. Builders design a pipeline one stage at a time, and its bill arrives at the interfaces between stages. Rewriting one stage's instruction moves gemma-3-12B's tax from 4.5 to 36.5 points, and adding "every relationship stated between them" to a stage that lists the numerical quantities lowers the tax on 9 of 9 models on MATH-500. Re-grounding, which shows a stage the original problem again, belongs after the loss. With one lossy interface, re-grounding the stage after it beats re-grounding the stage before it on 7 of 7 models on both benchmarks; on MATH-500 the earlier repair is worse than none on 7 of 7. Newer models still pay: gemma-4-12B gives up 37.0 points, and the repair holds on all three of the newest models we test. A sealed held-out test refuted a stronger rule we registered, which predicted the paying stage from the interface and receiver types, so we locate the tax by measuring one stage at a time. The prescription has two parts: re-ground the stage after the lossy interface, and if a stage must list the quantities, tell it to keep the relationships.