Grounding latent algorithm routing in transformer reasoning
2026-07-27 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied whether transformer models can adapt to different problem types by internally choosing different solving methods without changing the input prompts. They created a test called ROUTEBENCH with scenarios favoring different strategies like global shrinkage or sparsity. Their experiments showed that medium-sized transformers could internally switch solver preferences reliably, even under some input changes and noise. They also found that specific internal model signals relate to these solver choices, but they did not claim this occurs universally in all transformers or real-world language tasks.
transformerin-context learninglatent algorithm routingROUTEBENCHridge regressionlasso regressionHuber regressionk-nearest neighborsactivation patchingout-of-distribution performance
Authors
Xiangbo Zhang, Xiaoxu Ma
Abstract
A central question in the in-context learning literature is whether transformers can organize episode-level adaptation around different inductive-bias families. We study this question in a controlled setting through latent algorithm routing: route-like behavior in which the solver-family preference changes with the latent data-generating regime while prompt form is held fixed, remains stable under nuisance perturbations, and is selectively influenced by targeted activation interventions without large losses in answer quality. We introduce ROUTEBENCH, a diagnostic benchmark whose regimes differentially favor global shrinkage, sparsity, robustness, and locality, operationalized by ridge-like, lasso-like, Huber-like, and kNN-like family representatives. Across dense decoder-only transformers trained from scratch at 44M-612M parameters, a 306M model closes 80.9 percent of the oracle-routing gap and achieves route F1 of 84.1. The effect remains substantial under natural-language renderings, shuffled supports, lexical paraphrases, and a unified four-way routing setting. Stronger adaptive alternatives, including an input-conditioned soft mixture and an unsupervised Gumbel router, narrow the gap but remain below the 306M and 612M models on route F1 and OOD performance. Probe controls and matched activation-patching controls further show that route-relevant internal directions are decodable and functionally involved in solver-family-consistent output behavior. These results provide controlled evidence that dense transformers trained on ROUTEBENCH can develop route-like internal variables, but they do not establish universal routing in pretrained language models or unrestricted natural-language reasoning.