Efficient prompt compression improves large language model question answering
EffComp: Efficient Prompt Compression via Hybrid Reinforcement & Supervised Learning
Computation and Language
Summary
Long prompts given to large language models (LLMs) can be slow and costly to process. The authors propose EffComp, a way to shorten these prompts by selecting important sentences without inventing new information. They train the system using a mix of teaching and trial-and-error methods to keep answers accurate while making the prompts much shorter. Tests on question-answering tasks show EffComp is faster and more reliable than previous methods.
What this means in practice
- •For software engineers: Reduce input size for LLM-based question answering systems to lower their latency and computational cost without losing accuracy.
- •For customer support teams: Speed up AI-powered answer retrieval by compressing input knowledge documents into concise prompts that maintain response quality.
Authors
Romtham Tungsununtham, Piyawat Lertvittayakumjorn, Atiwong Suchato, Peerapon Vateekul
Abstract
Prompt compression is aimed at reducing input prompt lengths to enable cheaper, faster, and potentially higher-quality LLM predictions. However, existing prompt compression methods are often limited by modest compression gains, a risk of hallucination, and/or high compression latency. We propose EffComp, an Efficient prompt Compression framework for RAG-based question answering (QA). EffComp employs BERT-style document reranker and sentence selector models to allow fast extractive prompt compression at the sentence level. Its extractive nature prevents hallucinations in the compressed prompts. Additionally, the training process using a hybrid reinforcement and supervised learning approach is designed to optimize the compression ratio while preserving LLM accuracy. Experiments on various QA datasets demonstrate that EffComp outperforms state-of-the-art prompt compression methods achieving the best trade-off between compression ratio and accuracy with minimal latency. This makes it practical for real-world applications.