Papers for

llm deployment teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Microscaling cuts hardware costs in large language model acceleration

MIAO: Microscaling Intermediate Accumulation Optimization

Abstract: Microscaling (MX) quantization enables lower-precision large language model (LLM) inference with limited accuracy degradation, reducing memory requirement and multiplication cost. However, despite reduced operand precision, accumulation is often retained at high floating-point precisions, leaving a substantial hardware cost that operand quantization alone does not address. In particular, systolic arrays, as used in tensor accelerators such as Google TPU and AWS Trainium, distribute accumulation across distinct processing elements but conventionally use uniform accumulator precision throughout the array. This spatial organization offers an opportunity to reduce hardware cost by tailoring precision to individual accumulation stages. In this work, we present MIAO, a hardware-software co-design framework that jointly selects product precision and stage-dependent accumulator widths to minimize hardware area and energy cost under an end-to-end perplexity constraint. MIAO first calibrates MX block-scale shifts to reduce intermediate magnitudes while preserving operand quantization accuracy, then assigns narrower accumulators to earlier stages and progressively wider accumulators to later stages. To efficiently explore the resulting design space, MIAO combines hardware cost estimates derived from logic synthesis with a calibrated accuracy predictor to identify promising precision configurations, then validates the accuracy of promising candidates through bit-exact end-to-end LLM evaluation. Across MXFP8, MXFP6, and MXFP4, MIAO reduces processing element (PE) array area by up to 38.7% and energy by up to 36.1% relative to the all-BF16 accumulation baseline, with at most 1.60% additional perplexity degradation over operand-only MX quantization across six LLMs.

Sun 15 NovHardware Architecture
The gist
Running big language models usually requires a lot of computing power and memory, especially in parts that add up numbers. The authors found a way to make these adding parts use less space and energy by adjusting the precision at different stages. They created a tool called MIAO that picks how detailed numbers should be at each step, saving hardware and energy while keeping the models almost as accurate. This approach works well on multiple language models and different precision levels.
Open → 2611.19722v1

Circe uses model internals to check large language model answers

Circe: Training-Free LLM Self-Evaluation via Component Interaction Consistency

Abstract: As large language models (LLMs) are increasingly deployed in real-world applications, evaluating the correctness of their responses is crucial for reliable use. However, such evaluation is challenging because reference answers are rarely available in practice. Existing reference-free methods mainly rely on output signals, which may assign high scores to incorrect responses and thus do not necessarily reflect correctness. This motivates exploring the model's internal computation for complementary correctness signals. Recent studies exploit internal states, but they mainly analyze hidden-state trajectories across layers, which record only the combined updates of attention and feed-forward networks (FFNs) and overlook how these components interact. We therefore propose Circe, a training-free self-evaluation method that assesses a response using both internal and external signals, without reference answers or repeated sampling. For internal signals, we first introduce component interaction consistency (CIC), which measures the directional alignment between attention updates in earlier layers and attention-induced FFN output changes in later layers. We further incorporate representation geometry (RG) to characterize normalized displacement and angular changes along layerwise hidden-state trajectories. For the external signal, we convert the mean negative log-likelihood of generated tokens into a bounded likelihood confidence (LC) score. Finally, we fuse CIC, RG, and LC into a unified correctness score. Experiments on five LLMs and six benchmarks across four domains show that our method distinguishes correct from incorrect responses more effectively than existing baselines, and generalizes to multilingual, multimodal, and open-world settings.

Thu 5 NovArtificial Intelligence
The gist
Checking if large language models give correct answers is hard when you don't have the right answers to compare against. The authors present Circe, a new way to tell if a model’s answer is likely right by looking inside how the model thinks, without needing extra training or reference answers. Circe examines how different parts of the model work together and how the model’s internal states change, plus how confident the model is in its answer. Tests show Circe is better than existing methods at spotting right and wrong answers in many kinds of tasks and languages.
Open → 2611.07948v1

Tri-level method improves AI preference learning under shifting conditions

TPO: Tri-level Distributionally Robust Learning for OOD Direct Preference Optimization

Abstract: Direct Preference Optimization (DPO) minimizes empirical risk over observed preference comparisons, implicitly assuming that the training preference distribution matches the deployment preference distribution. In practice, this i.i.d. assumption often fails, as preference distribution shift exhibits a natural hierarchical structure across three levels: group-level shift, prompt-level shift, and preference-level shift. Such multi-level distributional mismatch poses a critical challenge for safe LLM deployment, yet existing robust DPO methods typically either model uncertainty over the joint preference distribution or focus on individual sources of shift, leaving the hierarchical structure of OOD preference shifts underexplored. We propose TPO, a tri-level distributionally robust framework for OOD preference alignment, which jointly models all three levels of distributional uncertainty through a hierarchically structured rectangular ambiguity set. By instantiating it with Wasserstein distance, we derive a tractable and interpretable objective that combines a closed-form preference-level margin penalty, prompt-level adversarial embedding perturbation, and worst-case group reweighting, each governed by an independent radius. We establish finite-sample convergence guarantees and empirically validate the effectiveness of TPO over existing robust DPO methods.

Tue 20 OctMachine LearningArtificial Intelligence
The gist
Preference learning helps AI like language models understand what users like based on comparisons. However, user preferences can change in complex ways across different groups, contexts, and individual choices, making it hard for AI to stay accurate. The authors present a new method called TPO that carefully models these three layers of preference shifts to make AI's learning more robust to changes. Their approach proves to be reliable and improves performance compared to existing methods.
Open → 2610.28229v1

Distributed subliminal learning shares knowledge without sending model updates

Distributed Subliminal Learning: Replacing Model Updates with Random-Carrier Outputs

Abstract: Collaborative learning typically exchanges model parameters: federated clients communicate updates, while independently adapted foundation models are combined by exchanging adapters or checkpoints. This makes communication scale with model size and requires local specializations to be reconciled in weight space, where interference is common. We ask whether knowledge can instead be shared through model behavior on task-unrelated inputs. We introduce Distributed Subliminal Learning (DSL), a collaborative learning primitive in which participants adapt a common model locally, probe it with task-unrelated inputs, and transmit only the resulting carrier outputs. A coordinator pools these outputs and distills them into a shared model. The primitive supports one-shot foundation-model composition through carrier completions and iterative federated learning through carrier logits, without transmitting model updates or requiring task-related proxy data. In LLM composition, compared with LoRA averaging, DSL achieves higher preference retention (94.56% vs. 87.76%) and a larger GSM8K gain over the base model (22.0 vs. 0.6 points), while reducing upload by 30.6-49.0$\times$. In federated classification, DSL reaches 96.83% on MNIST with 8.9$\times$ less uplink than FedAvg and provides lower-communication operating points on CIFAR-10 and Tiny ImageNet. These results establish random-carrier outputs as a practical communication primitive for knowledge sharing across distinct collaborative learning paradigms.

Sun 4 OctMachine LearningArtificial Intelligence
The gist
Collaborative AI training usually requires sharing parts of models, which can be large and cause conflicts. The authors propose a new way called distributed subliminal learning, where participants send only how their models behave on unrelated random inputs instead of actual updates. A central coordinator uses these responses to create a shared model, reducing communication needs and avoiding direct weight conflicts. They tested this method on language and image tasks and found it achieves good results while sending much less data.
Open → 2610.05378v1

Adversarial requests trick length predictions to jump scheduling queue

Jumping the Line: Exploiting Length Predictions in LLM Scheduling

Abstract: Efficient request scheduling is increasingly important for reducing completion time in large language model (LLM) serving. Size-based policies such as Shortest Job First prioritize shorter requests, but output lengths are unknown before generation, so practical schedulers rely on predicted lengths. We introduce JIL, an attack on prediction-based LLM schedulers that manipulates the scheduling signal to obtain higher priority and reduce completion time. Using TRAIL as a case study, JIL optimizes an adversarial suffix that causes a lightweight output-length probe to underestimate a request's length. We evaluate JIL on two datasets and four LLMs across varied request profiles and deployment configurations. JIL reduces predicted output lengths by up to 83.4 percent, and adversarial requests complete up to 1.53 times faster on average in end-to-end serving experiments. The reduction in predicted length is substantially larger than the change in actual output length, revealing a mismatch between the scheduler's estimate and the request's realized size. Response utility varies across models and tasks, exposing a trade-off between scheduling advantage and response quality. We also evaluate scheduler-side defenses and find that grouping length predictions into coarse intervals reduces JIL's scheduling advantage and mitigates delays to benign requests.

Fri 2 OctArtificial Intelligence
The gist
Large language models often decide which user requests to handle first based on how long the response will be. The authors show that by adding special endings to their input, attackers can fool the system into thinking their responses will be shorter than they really are. This trick helps these requests get faster processing at the expense of others. The authors also test ways to reduce this problem by making length predictions less precise.
Open → 2610.03430v1

Agent evaluations depend on tasks and scaffolds for reliability

Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard

Abstract: Agent evaluations are increasingly used to compare LLMs and inform deployment decisions, yet ranks can reflect not only the model but also the effects of the evaluation conditions such as the scaffolds or tasks. This makes reliability claim-dependent: an evaluation that reliably ranks deployed systems may not reliably rank underlying models. We ask which conclusions current agent evaluations reliably support and what additional evaluation would improve them. We develop a Bayesian variance-decomposition framework for sparse, imbalanced agent leaderboards and apply it to 22 benchmarks from the Holistic Agent Leaderboard and Harbor Index. The framework separates signal, performance differences relevant to the intended claim, from noise, irrelevant variation that can still change rankings. We find: (1) Reliability depends on the measurement goal. Fixed model-scaffold systems are ranked reliably (0.935-0.994), while underlying-model reliability is substantially lower (0.148-0.841). (2) Scaffold choice can change conclusions. Inter-scaffold reliability measures whether scaffolds preserve model rankings, showing that scaffold effects vary substantially across evaluations. (3) More tasks cannot resolve all uncertainty. Even infinitely many similarly constructed tasks improve model-ranking reliability of a benchmark by at most 0.097 when uncertainty is dominated by limited scaffold coverage. (4) Pooling diverse benchmarks can improve cross-task rankings at lower cost. For rankings across diverse agentic tasks, pooling benchmarks raises projected reliability from 0.44 to 0.75 at the same task budget and can reduce projected cost by up to 83\%. Evaluation design should follow the intended claim: identify what a score or ranking should mean, diagnose what limits its reliability, and spend evaluation budget on the sources of uncertainty that matter.

Wed 30 SeptArtificial IntelligenceMachine Learning
The gist
Evaluating AI agents to see which is best can be tricky because scores depend not just on the agents but also on how the tests are done. The authors developed a new way to measure how much of the score differences come from actual agent quality versus random noise or test setup. They found that just adding more test tasks doesn't always make rankings more reliable if the way tasks are structured varies a lot. Combining different sets of tests can give better and cheaper comparisons.
Open → 2610.00651v1

Galahad cuts repeated reading costs for large language models

Working Around the Compute Ceiling: Byte-Exact Memory in Galahad Makes LLM Reading a One-Time Cost LLM Reading a One-Time Cost

Abstract: A transformer language model performs a bounded amount of computation per token, and recent work by Vishal Sikka, former CEO of Infosys, argues that this bound limits which tasks a model can carry out or verify (arXiv:2507.07505). We ask how much of the budget beneath that ceiling is spent on work the model has already done. Serving is stateless across requests: a model that answers a second question about a document recomputes the document's attention state from the first token. On seven real-world datasets, 98.7% of prompt tokens were text the model had already read. We present Galahad, a memory layer for vLLM, SGLang and llama.cpp that makes this reading a one-time cost. Taliesin saves the model's key-value (KV) state for a block of text and loads it on the next request that contains the same bytes, instead of recomputing it. Blaise keeps the documents themselves and passes the model only the section a question needs. On a recall test with 100 facts hidden in a 97,000-token corpus (Gemma 4 31B), Taliesin alone let the model attend to the whole corpus and answered 98 of 100 on llama.cpp at 3.0 s and 572 J per question, against 10 of 100, 9.3 s and 2,754 J for the same model without Galahad, which could hold only the last 12,000 tokens. With Blaise added, the model read about 668 tokens per question and answered 100 of 100 on all three runtimes at 0.59-0.64 s and 200-213 J; a tuned RAGFlow pipeline answered 77. Storing the corpus is a one-time cost of about 100 s and 28 kJ, whose energy is recovered after 13 questions. Restored state is bit-identical: all 262,144 output logits matched after restart, rehydration and hot-load. Galahad worked with all 30 models we tested under vLLM, and it fails closed: any load that does not pass its checks is recomputed. Together these results move LLM serving from stateless to stateful inference.

Wed 30 SeptComputation and LanguageArtificial IntelligenceInformation Retrieval
The gist
Large language models (LLMs) often redo the same reading work when answering multiple questions about the same document, wasting time and energy. The authors designed Galahad, a tool that saves the internal reading state of the model so it only needs to read a document once. This approach drastically speeds up answering questions from large texts and reduces the computational energy needed. Their tests show Galahad can handle long documents more efficiently than previous methods, making LLM usage faster and greener.
Open → 2609.39358v1

Language models recover meaning from corrupted inputs spontaneously

Spontaneous Context Restoration: How Language Models Recover from Corrupted Inputs

Abstract: Language models sometimes produce correct outputs even when their inputs are corrupted by deletion, replacement, or misspelling. We study the internal processes accompanying this behavior, which we call context restoration, in controlled attention-only transformers and five pretrained LLMs (1B-32B parameters) across arithmetic, reading comprehension, and multiple-choice reasoning tasks. In the attention-only transformers, restoration emerges spontaneously despite training exclusively on clean sequences, without corruption training or an explicit denoising objective. We find that context restoration follows a two-phase process: early layers localize effects associated with repair at corrupted positions, while later layers accumulate these effects at uncorrupted positions through the residual stream and ultimately concentrate them at the output position. Repair outcome is predictable from hidden states: cosine alignment with the clean state is highly predictive in attention-only models, while linear probes recover additional information in pretrained LLMs. A linear probe using only the corrupted prompt's first-block hidden state predicts failure with mean ROC-AUC 0.78. This enables failure triage under matched or even partially shifted deployment conditions and may reduce unnecessary verification or computation. Failed examples also show substantially greater nonlinearity along corruption directions. Moderate-corruption finetuning increases corruption tolerance while simultaneously reducing displacement-normalized linearization error, associating improved robustness with a more nearly linear response to corruption.

Mon 28 SeptComputation and LanguageArtificial Intelligence
The gist
Sometimes language models give the right answer even if their input has mistakes like missing words or typos. The authors studied how this happens inside the model and found a two-step process where early parts spot errors and later parts fix them. This repair ability appears naturally, even wenn the models were only trained on perfect, clean text. They also made tools that can predict when the model will fail on bad inputs, which could help avoid wasted work.
Open → 2609.35475v1