Papers for

llm service operators

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Large language models vulnerable to resource draining chatter attacks

Chatter Attack: Resource Consumption Attack for Large Language Models

Abstract: LLMs are powerful but incur substantial inference cost, which can be exploited by malicious users to induce overly long outputs, increasing latency and operational expense. Prior resource consumption attacks either coerce repetition or suppress the end-of-sequence token by lowering its logit value. However, with autoregressive, sampling-based generation, the search space grows exponentially with length and small deviations or unstable trajectories can derail optimization, limiting attack effectiveness and stability. To address this issue, we introduce Chatter Attack, a novel class of resource consumption attacks that optimize adversarial prompts to induce predefined short target phrases, thereby avoiding search-space explosion while still eliciting long responses. Specifically, Chatter Attack triggers self-doubt using short doubt-inducing phrases as the target, and sustains this state throughout generation via inference attention loss and global entropy loss. To extend to black-box settings, we further propose a novel Bayesian Prompt Optimization (BPO) method, which models the objective function globally by constructing a discrete token kernel, efficiently exploring the entire search space using prior information. Extensive experiments across multiple models and datasets show that our method outperforms baselines, with maximum response length increases to 31.5$\times$, and revealing practical resource-depletion risks for real-world LLM inference services.

Sun 8 NovMachine LearningArtificial IntelligenceCryptography and Security
The gist
Large language models (LLMs) can be forced to produce very long responses by attackers, which slows down the system and uses a lot of resources. Prior attacks tried to do this by making the model repeat itself or never stop generating text, but these methods are unstable and hard to control. To fix this, the authors designed a new attack called Chatter Attack, which tricks the model into doubting itself with short phrases and keeps it going longer. They also developed a special optimization method to find these attack phrases even when they only have limited access to the model. Their tests show this approach can make responses over 30 times longer, exposing practical risks for real-world applications that rely on LLMs.
Open → 2611.11261v1

Intent aware caching improves large language model conversations speed

Intent-Aware Caching for Efficient LLM Serving

Abstract: Modern LLM serving systems use prefix caching to accelerate multi-turn conversational workloads. They cache the key-value states of previous conversation prefixes and reuse them when subsequent requests share the same prefix, to avoid redundant prefill computation and reduce serving latency. However, achieving a high cache hit rate remains challenging because cache reuse depends on whether a conversation continues and how soon the next turn arrives. Our analysis of real workloads shows that user intent is a useful signal for this behavior, as different intents exhibit different continuation probabilities and inter-turn delays. Based on this key insight, we propose Intent-Aware Caching (IAC), an online eviction policy that classifies requests by intent, learns lightweight reuse statistics from recent serving logs and leverages these statistics to guide an oracle-inspired eviction strategy. Extensive evaluation on real workloads (WildChat and ShareChat) shows that IAC reaches a 34.6% cache hit rate. This substantially outperforms both the previous learning-based approach (18.8%) and traditional methods such as least recently used (LRU, 12.2%), while reducing average serving latency by up to 46.1%.

Fri 6 NovArtificial IntelligenceMachine Learning
The gist
When people chat with large language models in multiple turns, computers try to save time by remembering parts of the conversation. But it's hard to guess which parts will be reused because people might stop or pause for different reasons. The authors found that knowing what the user intends to do helps predict if and when the conversation continues. They designed a new method that uses this idea to decide what conversation parts to keep in memory. This method made the model respond faster and reused conversation parts more often than older methods.
Open → 2611.09715v1

Caching and configuration settings impact large language model serving speed

A Controlled Experimental Evaluation of Configuration and Cache Architecture in LLM Serving Engines

Abstract: Multi-agent LLM applications repeat most of their prompt tokens across requests, and serving engines reuse the cached state of shared prefixes. vLLM and SGLang reuse it at different granularities, and benchmarks of their default deployments report large performance gaps, usually credited to the cache. Such comparisons rarely control for other configuration differences. One of them, whether a long prompt is prefilled in chunks, is set by each engine's defaults from the hardware and the release, and it confounds the comparison. We built a reproducible testbed on Kubernetes and a seeded multi-agent workload generator, and compared the two engines under vendor defaults and with chunked prefill matched. SGLang's configuration is the same in both arms, which makes it an internal control across independently provisioned deployments. Under defaults, vLLM's median time to first token is up to 64x SGLang's. Matching one setting removes 78% of the capacity gap where that gap is largest and 30 to 67% of the latency gap. Near saturation the remaining latency gap reflects queue growth. A fluid queue model driven only by a 4.4% throughput shortfall reproduces it, and shows that such a gap scales with the length of the benchmark run. What survives matching is a tail-latency advantage of 2.4 to 6.4x at equal throughput, consistent with cache-aware ordering of waiting requests. We derive six recommendations for comparing serving engines.

Wed 4 NovDistributed, Parallel, and Cluster ComputingMachine LearningPerformance
The gist
Large language models often receive many requests that start with the same words, so servers try to reuse stored data to respond faster. This paper compares two such servers, called vLLM and SGLang, to see if differences in their settings or caching cause speed differences. The authors controlled for configuration and found that much of vLLM's speed advantage comes from how it manages long prompts in chunks. After matching certain settings, the remaining speed difference comes from how each server handles waiting requests and caching. They offer recommendations for fairer comparisons of such servers.
Open → 2611.05991v1

Prompt caching errors in large language model tools cause hidden costs and delays

Silent and Costly: Characterizing and Detecting Prompt-Caching Mis-Integration in LLM Agents

Abstract: Large language model (LLM) services process massive volumes of user requests, resulting in substantial inference costs. To reduce both cost and latency, providers have introduced prompt caching. However, correctly integrating prompt caching is non-trivial due to heterogeneous provider-specific APIs and continuously evolving caching semantics. To understand prompt-caching mis-integration in downstream applications, we conduct the first empirical study of 120 prompt-caching-related pull requests from 10 popular open-source agent systems. Our findings show that prompt-caching mis-integrations arise in both request-side misconfiguration and read-side usage miscount. More importantly, most mis-integrations fail silently without causing observable runtime failures. Motivated by these findings, we design CACHELINT, which models a prompt-caching mis-integration as an invocation site together with the preconditions required for correct cache integration. CACHELINT locates caching invocation sites with CodeQL and discharges each precondition either against a dated provider capability oracle or with an LLM guard check. CACHELINT achieves 0.71 precision and 0.78 recall in identifying the 120 labeled mis-integrations across the 10 studied agent systems. Our evaluation further demonstrates the effectiveness of CACHELINT in real-world settings, identifying 32 mis-integrations across 571 agent projects with 0.86 precision. These findings have received 21 developer confirmations and led to 18 upstream fixes, including one in Anthropic's official SDK.

Mon 26 OctSoftware EngineeringArtificial Intelligence
The gist
Many apps use large language models (LLMs) to answer questions, which can be slow and expensive. To speed things up and save money, these apps try to reuse previous answers through a process called prompt caching. But the authors found that many apps make mistakes when using prompt caching, and these errors often don't cause obvious problems, making them hard to notice. The authors created a tool named CACHELINT to detect these hidden errors, helping developers fix them and reduce costs.
Open → 2610.34817v1

Adaptive resource aware sampling boosts large language model reasoning

ATEST: Adaptive Resource-Aware Test-Time Scaling for LLM Inference

Abstract: Test-time scaling (TTS) improves large language model (LLM) reasoning by sampling multiple answers, but fixed sampling budgets ignore both query difficulty and the changing energy cost of inference under GPU contention. We measure up to a 4.1-fold increase in energy for the same four-sample query as GPU load rises. We present ATEST, a resource-aware TTS framework that jointly considers the expected value of additional samples and their current GPU cost. ATEST-Online, its default mode, requires no offline calibration: it incrementally generates samples, estimates the stability of the plurality answer, monitors live GPU conditions, and learns step energy cost online to decide when to stop. ATEST-Calibrated instead selects a per-query budget using offline accuracy and cost models, favoring deployments with stable loads. We evaluate both modes on four reasoning and question-answering datasets using Phi-3-mini on an NVIDIA A100 under controlled varying load, and test transfer to an NVIDIA L40S and Qwen2.5-3B. Across the four A100 datasets, ATEST-Online answers every query without exceeding the energy budget, while achieving the highest or statistically tied accuracy and reducing sampling as load increases. On GSM8K at high load, it achieves 60.0% accuracy with no deferrals, versus 44.0% accuracy and 42.4% deferrals for ATEST-Calibrated. Ablations show that online cost learning drives adaptation to load. Including offline profiling, ATEST-Online uses 42 times less GPU time and 48 times less energy than ATEST-Calibrated for the evaluated workload. Online control trades higher tail latency and lower throughput on short tasks for robustness under varying load; offline calibration remains competitive under stable conditions.

Sun 18 OctMachine LearningArtificial IntelligenceComputation and Language
The gist
Large language models can answer questions better by trying multiple answers and picking the best, but always doing the same number wastes energy, especially when computers are busy. The authors designed a system called ATEST that decides how many answers to generate based on how hard the question is and how busy the GPU is, saving energy while keeping accuracy high. ATEST can learn these decisions on the fly without prior information or use prepared settings when loads are stable. They tested it and found it answers questions more accurately and uses less energy than fixed methods, adapting well to changing workloads.
Open → 2610.25519v1

Kv-cache offloading in llm serving trades predictability for latency

Characterizing Predictability-Latency Trade-offs of KV-Cache SSD Offloading in LMCache for LLM Serving Systems

Abstract: KV-cache offload is widely used to stretch GPU memory for LLM serving, but its storage behavior has not been characterized at the block-device level. In this paper, we study LMCache through realworld multi-session workloads that span same/different context x same/different prompt, using over 100 stateless requests per workload. Our comparison is observational rather than factorial: it contrasts two realizable deployment snapshots -- LMCache 0.3.0 with buffered I/O and the kernel page cache, and LMCache 0.3.16 with prefix-aware deduplication and O_DIRECT. Key findings define the paper. First, the legacy stack appears almost read-free at the SSD layer in all four workloads, but a flush_ram experiment shows that this is conditional on page-cache warmth: once the cache is evicted, the same path issues about 1 GB of sequential reads in 683 ms. Second, deduplication delivers a dramatic write reduction only for the exact-repeat workload: one set collapses from 1.62 GB to 11.75 MB of writes, while the other three sets remain in the 2-5 GB range. Third, O_DIRECT converts a hidden and bimodal read cost into an explicit and stable one: on one set, warm TTFT rises from 27 ms to 93 ms because each of the 99 warm requests re-reads the same ~12 MB partial chunk from SSD. The main conclusion is therefore not "newer is better," but a system trade-off: predictability versus average-case latency.

Thu 15 OctOperating Systems
The gist
LLM serving systems often save GPU memory by storing large key-value cache data on SSDs, but the way these SSDs behave has not been well understood. The authors study two versions of LMCache software with different storage techniques to see how they influence read/write behavior and speed. They find that newer techniques trade stable and predictable response times for sometimes slower average performance. This helps system builders choose between consistent speeds or faster responses depending on their needs.
Open → 2610.21249v1

Student guided distillation improves efficient large language model task routing

Student-Guided Teacher Distillation for Efficient LLM Task Routing: Positioning Against Jev-Style System-1 Classifiers

Abstract: Zero-shot classifiers are useful for routing user requests to specialized LLM tasks, but scoring every request against a large candidate set is expensive: a zero-shot NLI classifier must evaluate one premise-hypothesis pair per label, so cost scales linearly with taxonomy size. We study a student-guided teacher distillation pipeline for a fixed taxonomy of 60 LLM task categories: a compact ModernBERT classifier predicts the full category distribution in one forward pass and retrieves a small top-k candidate set, and a larger DeBERTa-v3 zero-shot NLI classifier reranks only those candidates rather than all 60 labels; the resulting teacher labels iteratively improve the student, which produces sharper candidates for the next round. Unlike generic embedding retrieval or clustering-derived shortlists used in extreme multi-label classification, our candidate generator is trained end-to-end on the target taxonomy and is the same model serving production traffic, distinguishing it from LLM-routing work that routes between candidate models, and from concurrent System-1 encoder-classifier proposals (e.g. TypeSafe AI's Jev and the open-source Laya project) whose training methodology is undocumented or RL-based. Our best student checkpoint reaches 77.5% teacher agreement on a 200-example evaluation set, and preliminary coverage measurements show Coverage@16 of 91-100%, suggesting top-k sets retain most of the teacher's decision-relevant information. We further show truncated top-k teacher scores should not be treated as full 60-class soft targets for KL distillation: zeroing untruncated classes destroys the dark knowledge soft-label distillation depends on, introducing systematic bias rather than a harmless sparse approximation. A complete evaluation, including coverage at multiple k on a held-out set, an embedding-retrieval baseline, and a larger human-reviewed test set, remains in progress.

Thu 1 OctMachine LearningArtificial Intelligence
The gist
Routing user requests to the right AI task can be slow and expensive when the system has many categories to check. The authors study a way to speed this up by training a smaller model (student) to quickly suggest a few likely categories, then having a bigger, more powerful but slower model (teacher) refine those choices. This process repeats, improving the smaller model's suggestions over time. They find that using the teacher's full knowledge is important when training the student and that their method keeps almost all important decision info while being cheaper.
Open → 2610.02516v1

KV cache management cuts latency and costs in AI reasoning agents

Capture the lifecycle: KV Cache management in ReAct Agents with KVTether

Abstract: Efficient serving of long-context reasoning-and-acting (ReAct) agents relies on KV cache reuse to reduce large language model (LLM) prefill latency and monetary cost. However, a semantic gap exists between agent harnesses and the underlying serving stack. Through context mutation, tool execution, and subagent coordination, context messages may become actively engaged, permanently discarded, and temporarily unused, while the serving stack only observes accesses to the corresponding KV cache. This lifecycle blindness prevents recency-only policies such as LRU from reclaiming dead KV promptly and from preserving older KV that will be reused sooner than newer entries. We present KVTether, a lifecycle-aware KV cache management framework for ReAct agents. By tracing semantic primitives embedded in agent harnesses, KVTether captures runtime lifecycle semantics during highly dynamic execution. KVTether then translates message-level semantics into KV-level lifecycle states and uses these states to drive state-prioritized cache management without exposing physical complexities to agent harnesses. After reclaiming dead KV, KVTether preferentially preserves live-but-idle KV that is waiting for reuse, reducing premature eviction before reuse. Across agent benchmarks and production workloads, KVTether reduces end-to-end request latency by up to 26.3% and 17.4% relative to LMCache and MORI, respectively, and lowers estimated task cost by 40.0% and 33.2% on average.

Wed 30 SeptOperating Systems
The gist
Large language models that reason and act need to reuse stored information efficiently to save time and money. The authors point out that current systems don't understand when cached data is still useful or can be thrown away, leading to wasted effort. They developed KVTether, a method that tracks how the data is used during the AI’s decision process to better manage this cache. This approach speeds up responses and lowers costs compared to previous methods by keeping important information longer and removing useless data sooner.
Open → 2609.39819v1

Honeypot reveals real attacks on exposed large language model infrastructure

OllamaDrama: Designing and Deploying a Honeypot to Measure Attacks on Exposed LLM Infrastructure

Abstract: Publicly exposed large language model (LLM) infrastructure creates a growing attack surface, yet real-world targeting remains poorly understood. We present Ollure, a low- and medium-interaction honeypot that emulates the Ollama API without a backend LLM. Spanning four deployments across cloud and university networks, Ollure operated for 84 days and recorded 290,887 interactions from 2,793 unique source IP addresses. Most of the activity consisted of automated discovery, fingerprinting, and model enumeration. However, we also observed concrete exploitation attempts against both the infrastructure and LLM layers. These included model management abuse, path traversal and SSRF probes, RCE and cryptocurrency mining payloads, resource exhaustion attempts, prompt injection, information extraction, and agent-oriented tool use. Our results provide empirical insight into real-world threats against exposed, self-hosted LLM services.

Thu 24 SeptCryptography and Security
The gist
The authors studied how hackers try to attack computer systems that run large language models, like those used in AI chatbots. They created a fake system called Ollure that looks real but doesn’t actually have the AI behind it. Over nearly three months, they recorded almost 291,000 interactions from nearly 2,800 different places on the internet. Many attempts were automated probes, but some were serious attacks trying to take control or steal information. This helps us understand the kinds of threats real AI services face when they are public.
Open → 2609.29757v1

Tool unifies AI inference costs across Kubernetes and provider bills

Who Pays for the KV Cache? Attributing Shared AI Inference Spend Across Kubernetes and LLM Provider Bills

Abstract: Organizations pay for AI through disconnected ledgers: Kubernetes allocations for self-hosted inference, gateway logs, and per-token bills from API providers. We present unalloc, an open-source tool that joins OpenCost, LiteLLM, OpenAI and Anthropic cost data into one exact ledger and reports the share of spend with no owner, and use it to study where attribution breaks at the seams between these systems. Five case studies run inference for real or simulate it: a vLLM-style serving simulator with paged KV memory and prefix caching; a PyTorch transformer serving a multi-tenant trace with a real KV cache; tensor- and pipeline-parallel inference on torch.distributed; the unmodified CLI against mock provider APIs; and four downstream use cases. At the seams, in a constructed multi-pod deployment scenario -- one month of synthetic OpenCost allocations, not observed billing data -- owner labels set only on LeaderWorkerSet leader pods leave 66% of that deployment's GPU bill unowned, and the natural fallback key assigns 61% of it to a Helm chart name while the headline unallocated share falls to 4%; enabling every source double counts all gateway spend; and reading one page of a billing API reports a quarter of spend. Inside a shared inference server the metering rule decides who pays: on an NVIDIA H100 running vLLM, a token meter assigns a retrieval-heavy tenant 12-14 percentage points more of the bill than an equal time-share meter at every load tested, while GPU utilization reads 97-99% across configured loads of 2 to 16 requests per second (3.7 to 26.9 completed requests per second; the configured rate counts session-initial arrivals only) and power draw tracks load. Neither meter is a ground truth; we position these results against recent Shapley-based energy attribution. Code, raw data, captured evidence, figures and the paper regenerate from the repository.

Mon 21 SeptDistributed, Parallel, and Cluster ComputingPerformance
The gist
Organizations often struggle to track who pays for parts of AI computing because the costs come from different places, like their own servers and external AI services. The authors created a tool called unalloc that brings together all these separate bills into one clear report showing who is responsible for what spending. They tested this tool in various scenarios and found that existing billing often leaves big parts of the cost unassigned or double counted. Their work highlights how complex it is to fairly assign AI costs among teams using shared resources.
Open → 2609.24991v1

Adaptive chunk sizing improves prompt handling in large language models

Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving

Abstract: Continuous batching improves large language model (LLM) serving throughput, but long prompt prefills can delay decode iterations and violate inter-token latency objectives. Chunked prefill mitigates this interference, yet its chunk size is normally fixed: small chunks protect decode latency but repeatedly pay launch overhead, while large chunks improve prefill efficiency but create latency spikes. We introduce SLOWeave, an online scheduling method that selects the largest prefill chunk predicted to finish before the earliest active decode deadline. The decision requires no workload-specific chunk-size tuning and is computed by a logarithmic-time search over a monotone iteration-cost model. We prove that, whenever a decode-only iteration is feasible and the cost predictor is accurate, SLOWeave maximizes immediate prefill progress among decisions that preserve every active request's next-token deadline. We evaluate the method in a reproducible event-driven simulator and an iteration-level GPU runtime across chat, mixed-context, long-context, and bursty workloads. Under a 25ms time-per-output-token objective, SLOWeave improves goodput over the strongest fixed-chunk baseline by 39% on mixed requests and 38% on long-context requests. Under a stricter 10ms objective, the gains rise to 3.3$\times$ and 2.4$\times$, respectively. These results isolate adaptive chunk sizing as a useful serving primitive and provide an implementation-ready controller for integration with iteration-level LLM runtimes.

Mon 7 SeptComputation and Language
The gist
Large language models take some time to get ready for generating text, which can delay responses when handling many requests quickly. The authors introduce SLOWeave, a way to split this preparation into smart chunk sizes based on how quickly the model needs to respond to each request. This approach balances speed and efficiency so the model works faster without making users wait longer at important moments. Testing shows it delivers more results on time compared to fixed chunk sizes.
Open → 2609.07883v1