Papers for

ai model deployers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

StrataFlux improves low-bit LLM inference accuracy and speed

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference

Abstract: Quantization reduces the memory and bandwidth demands of large language model (LLM) inference but can degrade accuracy; offloading higher-precision data can instead stall generation. We introduce StrataFlux, a framework that challenges this tradeoff by recovering accuracy lost to low-bit weight and key-value (KV) cache quantization while retaining quantization's memory and speed advantages. StrataFlux co-designs refinement representation, selection, and delivery to make host RAM and solid-state drives useful sources of additional precision. Its selective precision streaming reduces transfer volume, overlaps delivery with computation, and reuses fetched precision across decoding steps so generation need not wait for every transfer. We evaluate 2-4-bit weights and 2-3-bit KV-cache on dedicated GPUs and unified-memory platforms. For 2-bit Llama-3.1-8B-Instruct weights, StrataFlux reduces WikiText-2 perplexity from 9.46 to 8.40 while maintaining 95.4% of the quantized base's throughput on RTX 5090. With 2-bit weights and KV, StrataFlux raises Qwen3-8B's AIME 2024/2025 accuracy from 11.0% to 47.1% at unchanged GPU memory use. At 128K context, Llama-3.1-8B-Instruct with 3-bit weights and 2-bit KV achieves comparable perplexity and speed to vLLM's 4-bit AWQ weights and 8-bit KV on RTX 5090, using approximately half the GPU memory. These results show that GPU-resident precision need not set the accuracy ceiling, nor offloading-tier latency dictate decoding speed.

Thu 5 NovMachine LearningDistributed, Parallel, and Cluster Computing
The gist
Running large language models on computers usually needs a lot of memory and fast hardware, but cutting down the details (called quantization) can make the models less accurate. The authors propose StrataFlux, a way to use extra precision from computer memory and storage without slowing things down. Their method smartly fetches better data precision only when needed, keeping the model fast and memory-efficient while improving the results. They tested it on various GPUs and found it can get better quality and speed than prior approaches using very low bit levels.
Open → 2611.08363v1

Compression reveals how much fine-tuning data a model needs

The Behavioral Rate of Fine-Tuning: Compression as a Measure of Useful Information

Abstract: Low-Rank Adaptation (LoRA) adapts a frozen language model through a compact learned update, making it a natural setting for asking how much information fine-tuning needs to write. Yet adapter rank, parameter count, or numerical reconstruction error do not directly tell us how many bits are required to preserve the acquired behavior. We therefore define an operational behavioral rate $R^\star$ as the smallest serialized size of a compressed adapter that, once reloaded into the frozen base model, retains a fixed fraction of its held-out fine-tuning gain. Across LoRA fine-tuning experiments, this rate increases with distinct training content at fixed optimization budgets, but is not determined by intuitive metrics like source description length, fine-tuning gain, or weight-space reconstruction error. Most importantly, the same datasets exhibit substantially different, and even reordered, rates across frozen base models, showing that fine-tuning compressibility is a property of the dataset--receiver pair rather than of the dataset alone. However, we show that correction spectra measured on the data are able to rank these rates within a receiver. Finally, we study a striking noise-reducing effect of compression: adapters trained on deliberately corrupted datasets fall far below the frozen model, yet after compression score above it (23.7% to 82.9% on GSM8K, against 77.6% for frozen Qwen2.5-7B), consistently across tested families and sizes. The damage is unevenly concentrated in the adapter's leading singular directions, which compression appears to filter. Away from its training task, the compressed corrupted adapter also generalizes better than a clean fine-tune: on harder math it retains 97% of base accuracy, versus 44% for the clean adapter.

Wed 4 NovMachine Learning
The gist
When computer models like language AIs are fine-tuned, the amount of information needed to keep their improved behavior is unclear. The authors use a method called LoRA, which changes models in a small way, to measure how tiny a compressed update can be without losing performance. They find that the amount of useful information depends on both the data used to train and the original model, not just on simple factors like size or error. Finally, compressing these updates can even help remove noise and improve how well the model works on new tasks.
Open → 2611.06598v1

Pruned large reasoning models fail to stop but can be fixed

To Reason but Not to Stop: Understanding and Recovering Pruned Large Reasoning Models

Abstract: Weight pruning can degrade the accuracy of large reasoning models (LRMs), yet it remains unclear whether it damages reasoning itself. We find that much of this degradation stems from a failure to stop rather than to reason. As sparsity increases, pruned LRMs increasingly fail to terminate within the token budget, often after the correct answer has appeared in their reasoning, instead falling into repetitive loops, even though they still recognize where the dense model would stop. We further find that the hidden-state difference between terminating and looping continuations is consistent across pruned checkpoints, suggesting a shared direction associated with committing to an answer. Building on this, we propose Termination-Guided Activation Steering (TGAS), a training-free method that adds this direction to the residual stream at inference time. Across six LRMs, three pruning pipelines, and math and code benchmarks, TGAS improves termination and overall accuracy while reducing repetitive reasoning and generation length, showing that this failure is actionable.

Wed 14 OctComputation and Language
The gist
Large reasoning models are often made smaller by pruning their weights, but this can cause them to get stuck repeating themselves instead of stopping after finding the correct answer. The authors discovered that these repeated loops happen even when the model knows it should stop, so the problem isn’t reasoning but stopping. They identified a specific pattern in the model’s internal signals linked to deciding to stop, which stays consistent across pruned versions. Using this insight, the authors created a simple method to guide the model to stop at the right time during use, improving its accuracy and reducing repetitive outputs without retraining.
Open → 2610.19934v1

Joint smoothing and scaling improve NVFP4 quantization accuracy for AI models

OSFP4: Joint Optimization of Diagonal Smoothing and Block Scales for NVFP4 Quantization

Abstract: NVFP4 is an attractive datatype for large language model (LLM) inference, offering compact storage and native tensor-core acceleration. However, preserving accuracy using NVFP4 requires careful quantization. In this work we develop a novel quantization scheme called Optimized Smoothing and Scaling for NVFP4 (OSFP4). For each linear projection it uses a diagonal smoothing matrix whose entries are optimized to minimize the squared matrix-product quantization error under NVFP4, taking into account the rounding procedure that is used (either round-to-nearest, or GPTQ-style successive interference cancellation). This requires performing joint optimization on the smoothing entries as well as the block scales, which is facilitated by analyzing a multiplicative-dither FP4 quantizer instead of the fixed deterministic one. Experiments show that OSFP4 achieves the highest average accuracy among the evaluated competitors in the corresponding quantization settings, while retaining approximately 94-97\% of vendor NVFP4 prefill throughput on the measured workloads. Our code is available in https://github.com/neriahbd/OSFP4

Tue 6 OctArtificial Intelligence
The gist
Large language models need to shrink how they store and compute numbers to run efficiently. The authors developed a new way to carefully adjust number scales and smoothness to better keep accuracy when using a compact number format called NVFP4. They optimize these adjustments together to reduce errors caused by rounding numbers in the model's calculations. Their tests show this method keeps model accuracy high while running almost as fast as the vendor’s standard method.
Open → 2610.08231v1

Grammar guided watermarking improves code generation detection with minimal errors

Grammar-Guided Code Watermarking with Green Temperature

Abstract: Large language model watermarking embeds detectable statistical signals during decoding, but the resulting changes to token probabilities can degrade generation quality. This trade-off is particularly important for code, where small changes in token selection can break syntax or alter program behavior. Existing code watermarking methods mitigate this risk through entropy-based insertion or syntax-aware token selection, but they do not directly construct the watermark over the set of continuations admitted by the current grammar state. We propose Grammar-Guided Code Watermarking with Green Temperature (GTCW), which integrates grammar-constrained decoding with probability-aware watermarking. At each decoding step, GTCW restricts the candidate set to grammar-admissible tokens and partitions this support into keyed green and red subsets. At eligible high-entropy positions, green temperature reweights the green tokens according to the model's relative preferences, strengthening the watermark signal while retaining the grammar constraint. Across five models and five benchmarks spanning four programming languages, GTCW achieves a mean AUROC of 73.61%, compared with 67.83% for the strongest baseline, while maintaining a mean Pass@1 of 59.18% versus 59.58% for unwatermarked generation. Our implementation is available at https://github.com/hyundong98/GTCW .

Sun 4 OctCryptography and SecurityArtificial Intelligence
The gist
Embedding hidden signals in AI-generated code can help identify if code was machine-made but often causes small mistakes that break the program. The authors propose a way to guide the watermark process using programming language grammar rules, so the watermarks don’t create syntax or behavior errors. Their method, called GTCW, identifies where watermarks can safely be added and adjusts probabilities only among grammar-allowed options, making the watermark stronger but code still works well. Tests on multiple programming languages and AI models show the watermark is easier to detect without harming code quality.
Open → 2610.05323v1

Weight decay shapes neuron behavior in two layer neural networks

Weight Decay and Neuron Condensation: A Three-Stage Analysis of Two-Layer ReLU Networks

Abstract: Weight decay is widely used as a regularization technique in neural network training, yet its role in neuron condensation (parameter direction alignment) remains unclear. Starting from a parameter initialization in the neural tangent kernel regime, we characterize training dynamics under weight decay through three stages: rapid fitting, amplitude compression, and neuron condensation. Using a two-layer ReLU network, we analyze a residual correlation field that governs both neuron amplitudes and directions. During rapid fitting, the residual approaches a quasi-static equilibrium maintained by weight decay while the tangent kernel remains nearly unchanged. In the early stage of amplitude compression, kernel decay amplifies the residual correlation field, whose isolated attracting extrema provide candidates for condensation directions. As neuron amplitudes stabilize, we bound the drift of attracting extrema and demonstrate contraction of neuron directions around them, leading to neuron condensation. This staged analysis provides a dynamical understanding of how weight decay promotes a condensed representation, beyond reducing parameter norms.

Sat 3 OctMachine LearningArtificial Intelligence
The gist
Weight decay is a common technique used to keep neural network parameters small, but how it guides neurons to align or 'condense' was not clearly understood. The authors study a simple two-layer neural network and find that training with weight decay happens in three stages: fast learning, shrinking of neuron strengths, and finally neurons aligning their directions. This alignment, called neuron condensation, emerges from the dynamics of a mathematical field that controls neuron behavior. The work shows weight decay does more than just shrink parameters—it actively encourages neurons to cluster in direction.
Open → 2610.04533v1

SparseOPD improves efficiency of on-policy distillation with selective corrections

Look Before You Select: Rethinking Vocabulary Sparsification in On-Policy Distillation

Abstract: On-policy distillation (OPD) uses teacher correction on student-generated responses. Full-vocabulary correction can provide important corrections even for tokens that the student assigns low probability, but backpropagating through all token logits becomes memory-intensive for long sequences. Existing memory-saving approaches estimate corrections from sampled tokens or restrict supervision to the student's TopK tokens, introducing sampling noise or changing the full-vocabulary correction. We introduce \textbf{SparseOPD}, which uses full-vocabulary teacher correction to determine which corrections matter before selecting the token logits to differentiate. SparseOPD first constructs the full-vocabulary correction without retaining its backward graph, then selects tokens by correction magnitude rather than student probability. Signed residual compensation preserves the total promoting and suppressing correction mass, while correction-aware budget allocation distributes the sparse support across positions. Finally, the update backpropagates only through the selected token logits. Across six task--scale settings spanning mathematics, chemistry QA, and multimodal reasoning, SparseOPD outperforms Sampled Token and TopK in task-average accuracy and matches or exceeds Full Vocabulary. Gradient cosine similarity reaches 99\% on 4B mathematics, while 8K full-parameter profiling shows 70.5\% lower backward memory.

Mon 28 SeptComputation and LanguageMachine Learning
The gist
When teaching AI models to generate better answers, a method called on-policy distillation uses corrections from a teacher model on all possible words. But checking every word takes a lot of memory for long answers. The researchers introduced SparseOPD, a smarter way that first looks at all corrections to pick only the important words before updating the model. This reduces memory use dramatically while keeping or improving the quality of the AI’s answers.
Open → 2609.34386v1

MpFA boosts long-context AI speed on NVIDIA Blackwell GPUs

MpFA: Hardware-Efficient Train-Free QK4V8 FlashAttention Kernels on Blackwell GPUs

Abstract: Long-context LLM inference pushes modern GPU serving stacks into an attention-bound regime, where both compute and memory are dominated by the softmax-GEMM pipeline. On NVIDIA Blackwell GPUs, FP4 Tensor Cores offer high matmul throughput, but we find that fully FP4 attention often fails to translate this throughput into end-to-end speedups due to non-matmul costs: online quantization after softmax, tensor/shared-memory data movement, and contention on the softmax path. We present MpFA, a training-free FlashAttention kernel optimized for Blackwell. Guided by hardware characterization, MpFA uses mixed precision: NVFP4 for QK and FP8 for PV (QK4PV8). This preserves low-bit QK throughput while avoiding the conversion and scaling overheads of FP4 PV. To recover accuracy without further stressing the softmax pipeline, MpFA introduces rank-one smoothing compensation implemented as an additional Tensor Core MMA. MpFA further improves performance with a fine-grained asynchronous pipeline, tensor-memory reuse, and adaptive parallel partitioning across prefill and decode. On an NVIDIA B200 and across 16K-128K contexts, MpFA improves prefill throughput over state-of-the-art BF16/FP8 baselines and increases end-to-end output throughput by 2.81$\times$ over BF16 FA4 across Llama-3.1-8B and Qwen3-14B. Across five benchmark suites and two models, rank-one compensation recovers 62.5% of the accuracy loss with about 2.0% kernel overhead.

Sun 27 SeptDistributed, Parallel, and Cluster Computing
The gist
Long-context AI models require a lot of memory and computing power, slowing down their operations on GPUs. The authors introduced MpFA, a new method that mixes different precisions of math operations to speed up a key part of these models without needing retraining. They also found a way to fix accuracy lost from this change using a clever smoothing step. Overall, MpFA makes AI models run faster when handling very long inputs on certain NVIDIA GPUs.
Open → 2609.33135v1