FastE cuts large language model embedding work by 40 percent
FastE: Readout-Triggered Token Compression for LLM Embedding Inference
Artificial Intelligence
Summary
Large language models create embeddings that help computers understand text and images, but this process can be slow and costly. The authors found that parts of the initial input become less important as they move deeper through the model, meaning some of that information can be safely ignored without losing much quality. They introduce FastE, a simple method that quickly decides which parts to keep and which to remove during processing, making embedding generation faster and more efficient. FastE does this without needing to retrain the model and works well across many tasks and model sizes.
What this means in practice
- •For search engine developers: Reduce computational resources needed to generate document embeddings quickly, enabling faster and cheaper search indexing and retrieval.
- •For image captioning teams: Speed up cross-modal retrieval tasks by cutting down the work for embedding models that connect image and text data.
Authors
Jinsong Shu, Jinyong Wen, Baokun Wang, Zhongle Xie, Lidan Shou, Weiqiang Wang, Gang Chen
Abstract
In this study, we identify depth-dependent prefix redundancy in final-readout LLM embedding models, notably across representative backbones including Qwen3-Embedding and Qwen3-VL-Embedding. We find that removing prefix states is substantially more damaging in shallow layers than at greater depth, showing that prefix states become increasingly compressible as the prefix and readout states propagate through the network. To this end, we introduce FastE, a training-free, plug-and-play method. FastE uses a shared fixed threshold on batch-mean readout-prefix alignment as a lightweight online heuristic for selecting when compression occurs, and ranks prefix states by the attention scores they receive from the readout position to determine which states are retained in subsequent layers. Our evaluations demonstrate FastE's ability to substantially reduce computational costs: on NarrativeQA with Qwen3-Embedding-0.6B, it reduces decoder-backbone FLOPs by 40.11% while retaining 99.53% of Full Forward nDCG@10. Across five text embedding benchmarks, two backbone scales, and three cross-modal retrieval tasks, the quality-efficiency trade-off is directly customizable through the maximum removal ratio without retraining. We believe FastE offers practical value for scalable embedding generation in retrieval, indexing, clustering, and multimodal representation systems.