Sensitivity-Aware Thresholding and Token Routing for Activation Sparsification in Large Language Models
2026-07-09 • Machine Learning
Machine LearningComputation and Language
AI summaryⓘ
The authors explored ways to make Large Language Models (LLMs) faster without losing quality by reducing unnecessary calculations. They created a method called SATS to better decide when parts of the model can be skipped based on sensitivity, rather than just simple thresholds. They also developed a token routing system that smartly chooses different processing paths for each word token, instead of treating them all the same. Testing these methods on open-source LLMs, they showed improvements in balancing speed and accuracy compared to older approaches.
Large Language ModelsMLP activation sparsificationSensitivity-Aware Thresholdingthreshold calibrationtoken routinggate thresholdsactivation sparsityquality-throughput trade-offdynamic token processing
Authors
Bishmoy Paul, Youngmin Yi, Hoeseok Yang
Abstract
Efficient inference in Large Language Models (LLMs) requires deciding where computation can be reduced while preserving model quality. We study this problem through multilayer perceptron (MLP) activation sparsification and token-level conditional routing. We first propose Sensitivity-Aware Thresholding for Sparsity (SATS), a threshold calibration method to choose layerwise gate thresholds using a local MLP output sensitivity proxy rather than calibrating thresholds directly from activation percentiles. While SATS retains the existing mechanism of sparsifying MLP activations by thresholding gate activations, it replaces percentile-based calibration with a sensitivity-aware selection rule. We then introduce a lightweight token routing framework that dynamically selects between a base path and a modified path on a per-token basis, rather than applying the modified computation uniformly to all tokens. We evaluate both methods on multiple recent open-weight LLMs. Our results show that SATS improves over the threshold-based sparsification baseline at matched actual sparsity and that token routing yields a more favorable quality-throughput trade-off than static activation modification baselines. Overall, our results suggest that improved threshold calibration and token routing can improve the quality-throughput trade-off in LLMs.