System X improves language model replies using its own memory cache
System X: An LLM Decision Model That Replies from Its Own KV Cache
Artificial IntelligenceComputation and Language
Summary
Making language models choose answers usually involves scoring options, but sometimes they need to reply in words. The authors developed System X, which lets a language model generate word replies by continuing from its own saved memory, avoiding repeating the work of preparing the prompt. This method keeps accuracy similar while reducing errors in confidence estimates and cutting reply time by up to 39%. Some challenges remain with inconsistent replies and unsupported details, and the authors suggest more testing with humans and real data.
What this means in practice
- •For customer support teams: Use System X to generate faster and more confident text replies from language models without repeated prompt processing.
- •For chatbot developers: Implement cache reuse to improve response speed and reply quality in deployed conversational agents.
- •For translation service providers: Enhance routing accuracy for multilingual text support by applying System X’s reply generation with cache memory.$Commercial implications: Enables faster and more accurate multilingual responses which can improve paid language support products.
Authors
Syed Usama Khalid Bukhari, Syed Safwan Khalid, Muhammad Aamir
Abstract
Decision models return probabilities over supplied options, but a deployed agent may also need to reply in words. The present study introduces System X, a large language model (LLM) decision model that adds a text-reply option to the LLM-as-Jev readout. When this option wins, or confidence is low, the same model continues from the decision's key-value (KV) cache, avoiding a repeated prompt prefill. Qwen3-0.6B has been fine-tuned with LoRA on Reason-9k and Bitext reply rows. JevBench accuracy shows no statistically significant change (55.4% to 54.5%, p=0.87), while expected calibration error falls from 0.292 to 0.169. Held-out Bitext routing accuracy reaches 98.3%. Cache reuse saves 15-37% of latency on an Apple M4 Pro, with token-identical replies in the FP32 demonstration cases. Scoring options on copies of the prompt cache matches full-sequence scoring to within 1.4e-6 in probability. Adding dialogue, industry and complaint data (System X+) improves support-task results but lowers IFEval accuracy from 55.5% to 45.0%. System X v4 uses Qwen3-1.7B, rebalanced data and teacher-rewritten reply targets. Against its base LLM, JevBench accuracy rises from 55.0% to 61.0% (p=0.04), with no statistically significant IFEval change (p=0.71). A matched latency test on unadapted Qwen3-1.7B saves 17-39%. One LLM judge prefers v4 replies to an unadapted 1.7B second call (31 to 11 pairs, p=0.003); the other's preference is not significant. These reply comparisons also differ in prompts and generation limits. State conflicts and unsupported reply details remain failure modes. Further studies with several seeds, human raters and real support data are recommended.