Papers for

llm deployment engineers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Tool improves reliability of gpu kernel results in large language models

God Does Not Play Dice: Enforcing Batch Invariance in Agentic Kernel Generation

Abstract: Large language model (LLM) serving systems dynamically batch user requests for GPU execution, so the same request may be processed alongside different requests across runs. If the GPU kernels lack batch invariance (BI), this changing batch context can alter the numerical output for an unchanged input, undermining reproducibility. Existing kernel generation agents focus on correctness and efficiency but \emph{overlook BI}. We find 19 confirmed BI violations among 83 batch-applicable released artifacts. To address this limitation, we propose \tool, which combines contract-guided static dependency analysis with dynamic validation of bitwise consistency across batch contexts. \tool is designed as a plug-in component for general kernel agents, allowing their original optimization capabilities to be reused while adding BI enforcement. We integrate it as successive acceptance gates in the optimization loop, rejecting structurally risky or dynamically non-invariant candidates before performance benchmarking and returning feedback to the agent. Experiments across three agent frameworks under baseline and BI-aware generation on real world LLM serving tasks show that \tool improves the BI pass rate from 67.6\% to 91.9\% while retaining 91.8\% of the baseline geometric mean speedup. These results demonstrate that \tool can be incorporated into kernel optimization by agents as an additional reliability objective alongside correctness and speed.

Tue 27 OctArtificial IntelligenceMachine LearningSoftware Engineering
The gist
When large language models process many user requests at once on GPUs, the order or grouping of these requests can change the output even if the input stays the same. This problem, called batch invariance violation, makes results unreliable. The authors found many existing GPU kernels with this issue and created a tool called tool that detects and rejects kernels causing inconsistent results while keeping performance high. This tool can be added to existing systems to improve result reliability without slowing them down much.
Open → 2610.36623v1

Budget-safe evaluation method speeds up large language model tests

Dynamic Budget Allocation for LLM Evaluation under Hard Resource Constraints

Abstract: We evaluate large language models (LLMs) in multi-turn interactions through their time-to-event: the number of interaction steps required to produce an event of interest, such as a successful jailbreak or agentic task completion. Under limited compute, interactions may be terminated before the event occurs, so that event times are only partially observed (censored). Existing allocation methods for calibrating time-to-event bounds satisfy the budget only in expectation and can exceed the available budget on a particular evaluation run. Enforcing a hard constraint is particularly challenging as the cost of a trajectory is initially unknown. We introduce Hard-budget Allocation with Reflow for Predictive calibration (HARP), a budget allocation that satisfies hard resource constraints and adaptively reallocates unused budget. We show how to use HARP to construct lower predictive bounds (LPBs) on the time-to-event and to estimate evaluation metrics such as the jailbreak rate on a fixed benchmark. Although HARP induces dependence in acquisition decisions across different trajectories, we prove that HARP never exceeds the target budget, that its LPBs have finite-sample coverage guarantees, and that its metric estimates are unbiased. Experiments on agentic task success, LLM jailbreaks, toxic content generation, and RAG hallucinations show that HARP achieves coverage close to the nominal level with low variance, while never exceeding the given budget.

Mon 5 OctMachine Learning
The gist
Testing large language models often takes a lot of computing time and resources, especially when waiting for rare events to happen during interactions. The authors developed a new method called HARP that carefully manages a fixed computational budget so evaluations never run out of resources mid-way. HARP adapts how much time to spend on each interaction and still provides guaranteed reliable results about model performance. Their tests showed that HARP keeps costs under control and produces stable, unbiased measurements for things like safety checks and task success.
Open → 2610.07362v1

GPU memory sharing boosts large language model tuning and inference speed

MOLT: A Fine-Grained GPU Memory Sharing System for LLM Serving with Opportunistic Fine-Tuning

Abstract: Large language model (LLM) serving scales its replica count with the request load, yet GPU memory still stands idle inside the replicas. Adding a replica takes minutes, while the memory that a replica needs changes within seconds. Even instant autoscaling could not return this idle memory, because the smallest unit that it can remove is a whole replica. Colocating parameter-efficient fine-tuning (PEFT) with inference can use this memory, but inference must be able to reclaim it within seconds, before requests that wait for memory exceed their latency service-level objective (SLO). Existing colocation systems either keep the tuning memory resident or let inference reclaim it at the coarse granularity of a whole training sample. Each such reclamation also discards the running tuning step. To address these limitations, we present MOLT, a fine-grained memory sharing system that lets inference reclaim the memory of individual activations that a running tuning step has saved for its backward pass. The step continues, and its backward pass recomputes those activations. Inference reclaims only memory that no in-flight GPU work can still access, even under CPU--GPU asynchrony and tensor parallelism. On four model deployments (24B--70B) across H100 SXM and B200 GPUs under trace-driven workloads, MOLT keeps inference SLO attainment at or above 99.7% and completes 1.9--3.3x the tuning work of discard-based memory sharing.

Mon 5 OctDistributed, Parallel, and Cluster Computing
The gist
Running large language models (LLMs) on GPUs can waste memory because it’s allocated for entire model copies, even when requests change quickly. The authors designed MOLT, a system that lets the model reuse small pieces of this memory between running inference tasks and fine-tuning steps. This reuse happens without stopping the tuning process, saving time and fitting more tuning work into the same hardware. Tests showed that MOLT keeps response times reliable and speeds up fine-tuning by nearly three times compared to older methods.
Open → 2610.05748v1

Large language model routing improves by learning reusable capabilities

Pretrain Once, Route Anywhere: Towards a Foundation Model for LLM Routing

Abstract: Large language model (LLM) routing aims to assign each query to the most suitable model from a heterogeneous candidate pool, improving the quality--efficiency trade-off of LLM inference. Existing routers are typically learned through local fitting: a router is optimized for a particular query workload and candidate pool, and often requires additional supervision or retraining as the routing environment changes. We ask whether LLM routing can instead be approached from a foundation-model perspective, learning a reusable routing capability that generalizes across tasks, candidate models, and deployment conditions. To this end, we introduce RouteFM, which learns to characterize anonymous candidate models from behavioral context and infer their target-specific capabilities, rather than binding routing decisions to fixed model identities or a single environment. Through episodic pretraining across heterogeneous routing environments, this capability can be reused by a frozen router and adapted to new environments through context alone. Experiments demonstrate transfer across changes in domains, modalities, candidate pools, and context budgets, with the largest gains when behavioral evidence is limited. On MMR-Bench, which is excluded from pretraining, RouteFM outperforms the strongest baseline by 2.23 quality points with only eight observations per candidate. These results support moving LLM routing from repeated local fitting toward a pretrain once, route anywhere paradigm. Our code is publicly available at https://github.com/LAMDA-Model-Reuse/RouteFM.

Tue 29 SeptArtificial Intelligence
The gist
Large language model (LLM) routing decides which AI model best answers a question to balance quality and speed. Current routing systems learn only for specific tasks and need retraining when models or situations change. The authors propose RouteFM, which learns general routing skills from many environments, so it can handle new tasks and models without retraining. Their experiments show RouteFM works well even with few examples, outperforming other methods on new tests.
Open → 2609.37362v1

Kv cache compression method retains long and off context query strength

Cartridges++: KV Cache Compression without Off-Context Derailment

Abstract: Serving long documents to a Large Language Model (LLM) repeatedly is expensive: computations grow with context length, and the memory footprint of the key-value (KV) cache balloons. Compressed KV (CKV) representations aim to mimic the cache of a document and are typically computed once and for all, ahead of inference time. Methods to obtain CKVs range from drop mechanisms that reduce their number of columns, to learned approaches. Among the latter, Cartridges have emerged as a leading compression method, learning compact KV representations through distillation on relevant Q/A pairs. While existing evaluations focus primarily on whether Cartridges and other CKVs yield approximately similar responses to document-related, on-context queries, we investigate the crucial deployment question of whether they can handle off-context queries, something the native KV representation is particularly good at, thanks to the mechanics of attention. We observe a fundamental trade-off: while Cartridges perform better for on-context queries, heuristic-variants preserve better the original LLM's ability to operate off-context. We measure this through their capability to avoid context contamination in their response, retain general knowledge, and follow instructions. We propose Cartridges++, simple modifications to cartridges that retain off-context abilities at small or negligible cost. The router variant decides at inference time whether the query should use the learned long-context memory, while the data-mixing variant allocates a small fraction of training Q/As to queries outside the reference long document. Our study shows that assessing CKVs on document utility alone can mask substantial degradation in broader model capabilities, yet those issues can be fixed with benign changes to CKV inference or training.

Mon 28 SeptMachine Learning
The gist
Working with long documents on large language models uses a lot of computer memory and time, so people try to compress the memory it needs. The authors study a popular compression method called Cartridges, which works well for questions about the original document but struggles when questions are unrelated to that document. They find a trade-off between remembering document details and handling general queries. They propose Cartridges++, which modify the compression to keep the benefits for document queries while also preserving the model’s ability to answer off-topic questions well.
Open → 2609.35621v1

Dual-precision memory boosts large language model serving throughput

DPS: Dual-Mode Precision LLM Serving with Semi-Unified Memory

Abstract: Existing LLM serving systems virtualize and optimize KV-cache memory, but treat model-weight memory as fixed throughout execution. Recent work on multi-precision model representations challenges this design by allowing a single stored model to support both full-accuracy and lower-precision execution, making the effective weight footprint runtime-dependent. This creates an opportunity under bursty workloads, where temporary spikes in KV-cache demand often determine throughput and SLO compliance. We present DPS, a dual-precision LLM serving system that turns weight memory into an elastic resource: under normal load, DPS serves the full-accuracy model; under KV pressure, it switches to a nested, lower-precision variant and repurposes unused weight memory for KV cache blocks. DPS is built on Semi-Unified Memory (SUM), which partitions the weight region into a persistent lower-precision sub-region and a shared region that alternates between residual weight tensors and KV-cache blocks, preserving compatibility with paged KV-cache management. We implement DPS on top of vLLM and evaluate it across both dense and MoE models and various production workload traces. Our results show that \sysname improves sustained throughput by $2.1$--$3.3\times$ and effective pass@1 by up to $+41$\,pp over Static FP16, while preserving FP16-class accuracy.

Mon 28 SeptDistributed, Parallel, and Cluster ComputingArtificial IntelligenceEmerging Technologies
The gist
Running large language models uses different kinds of memory for storing both the model and temporary data. This paper presents a system that changes how memory is used depending on workload: it normally runs the model at full accuracy, but when the system is busy, it switches to a simpler version to save memory and use that space for temporary data. By doing this, the system can handle more requests faster without losing much accuracy. The authors tested this approach and found it can more than double throughput while keeping accuracy close to usual standards.
Open → 2609.34380v1

Py-kvcache improves large language model caching performance with nvme ssds

Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs

Abstract: Prefix caching can reduce the time to first token (TTFT) of long-context LLM requests by reusing previously computed key-value (KV) states, but for short prefixes or fast GPUs, recomputation can be faster than loading from an external cache. We characterize this tradeoff in vLLM across GPU, CPU, and NVMe tiers using synthetic workloads, long-context benchmarks, production traces, and find that cache performance depends on transfer granularity, intermediate memory use, and when transfers enter the request schedule, not only on device bandwidth. These findings motivate py-kvcache, a vLLM KV Offload connector with asynchronous direct I/O, bounded shared staging, and scheduler-aware preloading, which starts disk reads while requests are still waiting, overlapping with compute. At 80k tokens, py-kvcache loading from disk is 2.0x faster than LMCache, with preloading contributing 1.34x. With GPU, CPU, and disk caching enabled, it is 1.23x faster than LMCache and within approximately 4% of the native vLLM KV Offload implementation. LongBench and SCBench show that these benefits extend to irregular prefix chains and multi-turn workloads. Bailian trace replays improve TTFT on a weaker GPU, but on an H100 the average request falls below the break-even point and GPU memory alone retains enough prefixes. External KV caching should therefore be treated as a setup specific admission decision. The py-kvcacheimplementation is available at: https://github.com/atlarge-research/py-kvcache.

Thu 10 SeptDistributed, Parallel, and Cluster ComputingMachine Learning
The gist
Long conversations with artificial intelligence models can be slow because they need to remember everything said before. The authors studied how saving and reusing parts of these conversations, called key-value states, can speed up responses. They built a tool called py-kvcache that stores these states on fast NVMe disk drives and cleverly starts loading them before they're needed, making the process faster. They found that py-kvcache can speed up the time to start answering by up to twice compared to earlier systems, but using external cache depends on the setup and hardware.
Open → 2609.11744v1