Papers for

llm platform engineers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Fairway improves scheduling to speed up large language model responses

Fairway: Load-Aware KV Cache Reuse for LLM Prefill Scheduling

Abstract: LLM-based applications often employ agentic workflows and retrieval-augmented generation (RAG), which build long prompts from shared context. When prefixes are shared across requests, serving engines reuse the key-value (KV) cache to skip parts of the prefill computation. Reuse is typically graded: the uncached work of requests in a queue can differ by up to 100x. In this setting, the order of scheduling determines the time-to-first-token (TTFT) for the queued requests. KV reuse-first scheduling, which prioritizes the requests with the longest KV cache hit, gives the lowest median TTFT but starves expensive requests under load spikes; first-come-first-served (FCFS) scheduling bounds the tail but raises the median TTFT. We present Fairway, a prefill scheduler built on adaptive reuse prioritization: it favors KV cache reuse while queues are short, and shifts toward fair allocation as requests wait longer relative to their own prefill time. Fairway employs a virtual dual lane model: a KV reuse-first Fast lane and a Fair lane atop the serving engine's waiting queue that enables speeding up of cheap reuse requests without starving expensive ones. This model automatically adapts to the workload without any per-workload tuning. Fairway lowers the tail TTFT at no cost to the median in transient overload. Across workloads and load levels, Fairway matches or beats the best baseline, with a p99 TTFT up to 8.6x lower than prior prefix-aware schedulers. During load spikes, it keeps the tail of FCFS at a 6x lower median TTFT.

Sun 1 NovDistributed, Parallel, and Cluster Computing
The gist
Large language models often build responses using parts of previous computations stored in a cache called the key-value (KV) cache. When multiple requests share similar starting points, the system can reuse this cache to work faster. The authors found that prioritizing requests with larger cache reuse reduces wait times for many but can delay more complex requests. They created Fairway, a smart scheduler that balances quick responses and fairness, helping all requests get served faster during busy times.
Open → 2611.02082v1

Agentic lLM serving improves speed by better KV cache eviction

Rethinking KV Cache Eviction for Agentic LLM Serving

Abstract: Key-value (KV) caches impose substantial memory pressure on large language model serving systems, requiring cache eviction when capacity is insufficient. Existing mechanisms typically evict KV from the ends of cached sequences to preserve prefix sharing across requests. However, agent workloads repeatedly reuse long histories across successive turns. Recomputing the KV cache for tokens near the end of a sequence is substantially more expensive than for earlier tokens because their attention computation involves longer histories. To address this problem, we propose preserving shared prefixes and evicting the KV cache from the beginning of the remaining history while retaining the cached suffix. We build Penelope around this strategy by co-designing the eviction policy, KV recomputation, and unified GPU-CPU cache management. Returning requests reuse the KV cache retained in GPU or CPU memory and recompute only missing segments while preserving reuse of the cached suffix. We implement Penelope in SGLang and evaluate it on three Qwen3 model sizes using real agent traces. Compared with SGLang, Penelope reduces mean time to first token by up to 60% and job completion time by up to 37% with a GPU-only cache, and by up to 76% and 60% with a two-tier GPU-CPU cache.

Fri 30 OctDistributed, Parallel, and Cluster Computing
The gist
Large language models use a key-value (KV) cache to remember past conversation tokens, but this cache can get full and slow down the system. Existing methods usually remove tokens from the end of the conversation history, which wastes time because recent tokens rely on all previous context for attention. The authors propose a new way to keep the end of the conversation intact and remove older tokens from the beginning instead. This approach, implemented in their system Penelope, reduces the delay before the first answer and overall response time by efficiently managing the cache on both GPUs and CPUs.
Open → 2611.00391v1

Probeid verifies large language model tools despite changing responses

ProbeID: Verifying MCP Tool Identity for LLM Agents under Registry Drift

Abstract: Agents select tools by copyable descriptions, which cannot establish which implementation runs. ProbeID compares canonicalized responses to schema-valid probes with a trusted admission reference and abstains on ambiguity, so a squatter must copy probe-observable behavior. With the gold present, independent probes and a fixed margin, standard concentration arguments give a shared probe budget logarithmic in the candidate count and bound the joint probability of a wrong decision. On cached MCP-IdentityDrift cases, strict choice reaches \(0.956\) hit@1 while description routers fall near chance, but simpler exact comparisons tie it, MMD better tolerates heavy value noise, and a 32B behavioral judge is similarly accurate at higher cost. Random probes from agent banks and a prospective held-out study of 38 tools test theory-set thresholds. On the agent banks, equality refuses every reachable replacement from 64 probes and the bounded distance from 512, and sharper post hoc gates halve both. Two tools whose drift no bank probe reaches always pass. In an MCP-Bench agent, verification cuts tainted tasks from 15 to 9 of 16 at unchanged task success. Over eleven weeks, 7 of 59 real tools changed responses despite version pins. Verification depends on reference integrity, probe coverage, and response noise.

Sat 24 OctCryptography and SecurityArtificial Intelligence
The gist
Large language model (LLM) agents often pick tools to use based on descriptions that anyone can copy, making it hard to know which tool is really running. The authors propose ProbeID, a method that sends specific test questions called probes to tools and checks their responses against a trusted reference to confirm the tool's true identity. If the answers are ambiguous, ProbeID rejects the tool, so impostors must mimic observable behaviors exactly to pass. Tests show ProbeID can detect changes in tool behavior over time, improving task reliability by filtering out altered tools without hurting performance. The method depends on good reference data and coverage of probe questions.
Open → 2610.33633v1

Attention based method improves detection of indirect prompt injection attacks

Attention-based Classification for Indirect Prompt Injection

Abstract: Indirect prompt injection (IPI) attacks, in which adversarial instructions are embedded in external data processed by a language model, pose a significant threat to the integrity of deployed LLM agents. Attention Tracker (Hung et al., 2025) is a recently proposed white-box detection method that identifies prompt injection attacks by monitoring how a model's attention is distributed between user task tokens and data tokens, claiming dataset-agnostic detection via a scalar focus score threshold. In this paper, we provide the first evaluation of Attention Tracker in a realistic agentic IPI setting using the AgentDojo benchmark, finding that the dataset agnosticism claim does not hold. We find that in agentic environments, long structured tool outputs subvert the fundamental assumptions Attention Tracker makes about attention that the method depends on and produces near-chance detection on AgentDojo across most model families. We propose AttnDA, an extension replacing the scalar threshold with a Fisher Linear Discriminant Analysis classifier over the per-head focus score vector, and evaluate both methods in a cross-dataset setting spanning four benchmarks. AttnDA improves detection performance across agentic and tool-integrated benchmarks, though transfer to AgentDojo remains challenging regardless of classifier choice. A covariance ablation further shows that removing inter-head correlation structure substantially contributes to performance, with complex covariance assumptions causing significant degradation on cross-domain data. These results indicate that the attentional distraction signal underlying focus-score-based detection does not transfer reliably to agentic environments, and that learned linear classifiers over head-level attention features offer improved detection performance, provided training data from the target deployment domain is available.

Tue 20 OctMachine Learning
The gist
Indirect prompt injection attacks sneak harmful instructions into data that language models process, which threatens how these models behave. The authors studied an existing detection method called Attention Tracker and found it struggles with realistic settings where complex tool outputs confuse the model's attention. They propose a new method, AttnDA, that uses a more detailed analysis of attention patterns and performs better at detecting attacks, especially when trained on data similar to where it will be used. However, detecting attacks in all environments remains hard, showing attention patterns alone are not always reliable indicators.
Open → 2610.28988v1

RadixFlow improves large language model latency by better scheduling

RadixFlow: Filling Execution Bubbles in Prefix-Cached Hybrid LLM Serving

Abstract: Prefix reuse improves LLM inference by sharing the KV cache of common prompt prefixes. This helps only if the scheduler can execute a matched branch in the current worker round. Schedulers assume that a matched prefix is ready, which holds for full attention but fails for hybrid LLMs. There, a branch (the consumer) may need recurrent state that another branch (the producer) is still computing in the same round. The native scheduler stops its waiting-queue scan at this blocked consumer, and the resulting head-of-line blocking creates \emph{execution bubbles}: idle batch slots that independent work could fill. We introduce RadixFlow, which decouples prefix matching from state readiness. Its state-ready traversal turns a global scan barrier into a local wait: only the consumer is blocked until the next round, while feasible independent branches keep filling the batch under the native state check. On five hybrid models, RadixFlow lowers mean latency by 3\% to 24\% and worker rounds by 11\% to 44\% at almost unchanged reuse. On the larger Qwen3.5 models, ablations show that most of the gain comes from traversal, which adds little on a full-attention control. Heavier rounds limit how much of this saving reaches latency.

Tue 13 OctArtificial Intelligence
The gist
Large language models (LLMs) often reuse parts of previous work to speed up responses, but this can cause delays when some parts depend on work still in progress. The authors found that current schedulers often wait unnecessarily, leaving some processing slots idle. They created RadixFlow, a new scheduling method that lets independent tasks proceed without waiting for dependent ones to finish. This method reduces the average delay and the number of processing rounds needed, making hybrid LLMs faster without losing efficiency in reuse.
Open → 2610.18143v1