FastE method speeds up large language model embeddings with less computing
FastE: Readout-Triggered Token Compression for LLM Embedding Inference
Artificial Intelligence
Summary
Generating embeddings from large language models (LLMs) can be slow and require a lot of computing power. The researchers found that parts of the model’s early processing steps become less important as data moves deeper into the model. They created FastE, a simple technique that decides when and which parts to shrink during processing, reducing computation without losing much accuracy. This approach works without retraining the model and can be adjusted based on how much speedup is desired.
large language modelstoken compressionembedding inferenceprefix redundancyattention scorescomputational costretrieval systemsclusteringcross-modal retrieval
Authors
Jinsong Shu, Jinyong Wen, Baokun Wang, Zhongle Xie, Lidan Shou, Weiqiang Wang, Gang Chen
Abstract
In this study, we identify depth-dependent prefix redundancy in final-readout LLM embedding models, notably across representative backbones including Qwen3-Embedding and Qwen3-VL-Embedding. We find that removing prefix states is substantially more damaging in shallow layers than at greater depth, showing that prefix states become increasingly compressible as the prefix and readout states propagate through the network. To this end, we introduce FastE, a training-free, plug-and-play method. FastE uses a shared fixed threshold on batch-mean readout-prefix alignment as a lightweight online heuristic for selecting when compression occurs, and ranks prefix states by the attention scores they receive from the readout position to determine which states are retained in subsequent layers. Our evaluations demonstrate FastE's ability to substantially reduce computational costs: on NarrativeQA with Qwen3-Embedding-0.6B, it reduces decoder-backbone FLOPs by 40.11% while retaining 99.53% of Full Forward nDCG@10. Across five text embedding benchmarks, two backbone scales, and three cross-modal retrieval tasks, the quality-efficiency trade-off is directly customizable through the maximum removal ratio without retraining. We believe FastE offers practical value for scalable embedding generation in retrieval, indexing, clustering, and multimodal representation systems.