Improving layer reconstruction does not always boost final model accuracy
Counterexamples to Local Reconstruction Gain as a Proxy for Final Fidelity in Residual Completion
Machine LearningComputation and Language
Summary
This paper looks at a technique called residual completion, which tries to guess parts of a language model’s attention that were left out to speed up processing. The authors found that even when these guesses match the skipped parts better at one layer, it doesn’t always make the whole model’s final output more accurate. In some cases, trying to improve this intermediate reconstruction actually led to worse final results, while perfectly restoring the original attention improved accuracy. Their findings suggest that better local fixes inside a model don’t guarantee improved overall understanding.
What this means in practice
- •For language model engineers: Decide when improving intermediate attention approximations is unlikely to enhance final model outputs during model design or deployment.
- •For natural language processing developers: Avoid relying solely on local reconstruction gain as a proxy metric when tuning sparse attention approximations in transformer-based models.
Authors
Yasuto Hoshi, Daisuke Miyashita, Jun Deguchi
Abstract
Residual completion augments query-aware sparse attention by estimating the contribution of tokens omitted from the exact sparse computation. We ask whether improving a layer's attention-output reconstruction on the same incoming Q/K/V and selected support necessarily improves the fidelity of the final model output. We study training-free RESA and learned Top-K+$φ$ with frozen backbone language models. A prespecified single-layer screen yields two Qwen3-0.6B/Multi-LexSum interventions for which direct-runtime measurements show positive prespecified request-aggregate local reconstruction gain but worse final KL fidelity than the corresponding all-abstain Exact Top-K baseline on both discovery and prompt-token-disjoint holdout requests. Exact restoration at the same layer instead improves final fidelity, showing that the reversal is specific to approximate completion in these cases. In complementary multi-layer experiments, a task-independent local diagnostic often repairs the tested completion estimators, although the repaired models do not consistently outperform Exact Top-K. Together, these results show that better local reconstruction need not translate into better final-model fidelity.