Papers for

cloud ai infrastructure teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Factored optimizer speeds up large language model training efficiently

Factored One-sided Adam-Moment for Practical and Scalable SOAP

Abstract: Second-order optimizers (e.g., Shampoo, SOAP) have emerged as compelling alternatives to Adam/AdamW and promise faster per-step convergence than AdamW in transformer training. However, they still remain costly in practice, and even SOAP still pays substantial optimizer-state memory and eigendecomposition overhead. We ask whether the benefits of SOAP can be retained at nearly the efficiency of AdamW. In this paper, we answer yes with a lightweight SOAP variant, Factored One-sided Adam-Moment (FOAM). The path to practical impact is twofold: first, FOAM employs input-only preconditioning, dropping the output-side Kronecker factor to reduce memory and remove one eigendecomposition. The Kronecker structure of the Hessian shows that this is exact for a diagonal output-side curvature and nearly lossless for a near-diagonal one, as we observe empirically. Second, FOAM replaces SOAP's dense rotated second moment with a factored Adam-moment estimator based on row-by-column statistics. Under the K-FAC Fisher assumption with a diagonal output curvature, this estimator exactly recovers SOAP's dense second moment, and we measure only a 15-20% K-FAC violation, with negligible effect on the update. Our empirical results support the claim that FOAM matches strong SOAP and Muon baselines in validation loss across scales of large language model (LLM) pre-training, uses half of SOAP's optimizer state, runs at Muon-class wall-clock time, and retains robustness across a three-fold learning-rate range. The results hold from 124M to 760M parameters, demonstrating that SOAP's gains need not require SOAP's cost. Code: https://github.com/naver-ai/foam

Thu 12 NovMachine Learning
The gist
Training large language models usually requires powerful optimizers, but the best ones can be slow or use a lot of memory. The authors introduce FOAM, a simpler and lighter version of a strong second-order optimizer called SOAP. FOAM keeps most of SOAP’s benefits while using less memory and running faster, making it easier to train big language models without extra hardware costs. Their tests show FOAM works well on models with hundreds of millions of parameters and matches top-quality results.
Open → 2611.16905v1

Modality aware pruning saves memory in multimodal large language models

Modality-Aware Expert Pruning for MoE-Based Multimodal Large Language Models

Abstract: Mixture-of-Experts (MoE) enables multimodal large language models (MLLMs) to scale capacity efficiently, but expert parameters dominate memory even though only a few experts are activated per token: they account for over 90% of model size in modern MoE-MLLMs (e.g., 54 GB out of about 58 GB in BF16 for InternVL3.5-30B-A3B and Qwen3-VL-30B-A3B). Expert pruning has emerged as an effective compression strategy, but existing methods aggregate importance uniformly across tokens, ignoring that the same expert may serve different roles for vision versus language processing. To address this, we propose MAEP (Modality-Aware Expert Pruning), a training-free framework that decomposes expert importance by modality in a single forward pass. MAEP introduces (1) Delta-H Output ($ΔH$), a redistribution-aware metric measuring the MoE output change when an expert is removed, and (2) low-attention vision tokens as a vision-side pruning signal that is dataset-stable: its expert-importance ranking remains stable across different datasets. Text-side importance serves as a preservation signal while vision-side importance serves as a pruning signal, and experts are ranked globally across layers. Experiments on three MoE-based MLLMs across multimodal and text benchmarks show that MAEP outperforms modality-agnostic baselines on the majority of multimodal benchmarks while remaining competitive on text-only tasks, reducing the BF16 expert weight footprint by about 27 GB (about 47% of total memory) so that the full weight footprint of an approximately 58 GB model fits within a single 40 GB GPU. Code is available at https://github.com/nota-github/MAEP.

Wed 4 NovComputer Vision and Pattern RecognitionComputation and Language
The gist
Large language models that handle both images and text use many expert components, which take up a lot of memory. The authors propose a way to shrink these models by carefully removing experts based on whether they are more important for image or text processing. Their method does this without needing extra training and keeps the model working well, especially on tasks involving both pictures and words. This makes it possible to run big models on smaller hardware.
Open → 2611.05927v1

MoE language models gain accuracy with symmetric FP4 reinforcement learning

Aligning Training and Rollout Policies for End-to-End FP4 RL on Mixture-of-Experts LLMs

Abstract: In this work, we study end-to-end FP4 reinforcement learning (RL) post-training for Mixture-of-Experts (MoE) large language models. Recent work on FP4 RL has introduced effective recovery recipes that restore accuracy under aggressive low-precision quantization for dense models, but we find that the same approach yields no improvement over the naive FP4 baseline in the MoE setting. We conduct an in-depth study to understand why. Our analysis identifies the root cause in the routing mechanism unique to MoE: because the router makes discrete expert-selection decisions, training-inference policy alignment becomes acutely sensitive to whether the same recovery technique is applied symmetrically on the training and inference sides. When an asymmetric recipe is used-including the dense-model approach, which corrects only the rollout-training-inference alignment deteriorates sharply, as measured by the Pearson correlation coefficient between training and inference probabilities, and router selection matching rates. Building on this finding, we propose MoeResQ, a quantization-recovery recipe for MoE RL post-training that enforces symmetry between the training- and inference-side corrections. MoeResQ restores training-inference alignment and substantially closes the accuracy gap to full precision, establishing symmetric recovery as the key principle in FP4 RL for MoE models.

Tue 3 NovMachine LearningArtificial Intelligence
The gist
When training certain big language models called Mixture-of-Experts (MoE), using low-precision numbers makes the models less accurate. The authors found that a recovery method that helps models with regular structures does not work for MoE models because MoE has a special way of choosing which expert to use. They discovered that making the correction method symmetric between training and using the model fixes the problem and improves accuracy. Their new method, MoeResQ, helps MoE models get closer to the performance of full-precision models.
Open → 2611.04376v1

Mixed ADC resolution improves energy efficiency and accuracy in analog AI computing

SMART-AIMC: Sensitivity-Aware Mixed ADC Resolution with Fine-Grained Tunability for Analog In-Memory Computing

Abstract: Analog In-Memory Computing (IMC) is a promising paradigm for accelerating GEMM-based AI workloads by executing computations directly within the memory array in the analog domain. While IMC delivers substantial energy and throughput improvements, it can suffer from significant task accuracy degradation due to analog-to-digital converter (ADC) quantization noise, which occurs at the GEMM output and is thus different from weight/activation quantization, which occurs at the GEMM input. Mixed ADC resolution allows IMC modules to choose between high- and low-resolution ADC modes with different energy costs, offering a compelling accuracy-energy trade-off. However, selecting ADC resolutions across IMC units while respecting hardware granularity constraints remains challenging. In this paper, we propose SMART-AIMC, a Sensitivity-aware Mixed ADC Resolution framework with fine-grained Tunability for Analog In-Memory Computing. SMART-AIMC correlates the algorithmic task loss to hardware ADC quantization noise in the form of an ADC Channel Sensitivity (ACS) metric. Using this metric, SMART-AIMC systematically, and in a disciplined manner, allocates ADC resources to critical computations. Furthermore, we introduce a sensitivity-based channel reordering scheme that overcomes the hardware granularity constraints with no runtime overhead and approaches the optimal ADC quantization possible for the model. Evaluated across diverse workloads, SMART-AIMC unlocks a superior accuracy-energy Pareto frontier, providing (1) higher task accuracy at matched energy budgets and (2) fine-grained tuning across efficiency targets, from aggressive energy savings to near-digital inference accuracy. At the latter operating point, SMART-AIMC achieves $6.99\times$ higher modeled system-level energy efficiency for linear projections than the INT8 digital baseline with negligible accuracy degradation.

Sun 1 NovHardware ArchitectureArtificial Intelligence
The gist
Analog in-memory computing (IMC) speeds up AI tasks by doing math directly inside memory, but it can lose accuracy due to noisy signal conversions. The authors developed SMART-AIMC, a way to smartly pick different precision levels for these conversions based on how sensitive each part of the calculation is, improving accuracy and saving energy. They also reorder computations to better use hardware capabilities without slowing down. Their method achieves much better energy efficiency than standard digital AI systems while keeping accuracy very close.
Open → 2611.01707v1

RankLane speeds up mixture-of-experts model prefill on AI hardware

RankLane: Efficient MoE Prefill on Accelerators with Ranked Expert Capacities

Abstract: Mixture-of-experts (MoE) models expand model capacity by activating only a subset of experts for each token. On accelerators with static execution plans, expert capacities and placement are fixed before routing reveals each expert's load. Uniform worst-case capacities waste work on padding; smaller capacities tied to expert identities can overflow. We present RankLane, which ranks each device's experts by current load and assigns them to fixed-capacity lanes. Runtime indices select experts and token rows while tier populations, shapes and device placement remain fixed. An aggregation graph preserves device ownership and reduces each token's local contributions on one core. Routing calibration and measured execution costs guide tier selection; overflowing chunks rerun at full capacity. We evaluate full-model prefill of Qwen3-30B-A3B and Gemma 4 26B-A4B with MXFP6 weights on four Cloud AI 100 Ultra SoCs, using 128-2048-token chunks and including overflow detection and in-program recovery support. Against a dense baseline, RankLane achieves end-to-end prefill speedups of 1.27-4.18x on Qwen3 and 1.93-3.46x on Gemma.

Fri 30 OctMachine Learning
The gist
Large language models sometimes use mixture-of-experts (MoE) to handle more data by activating only some parts called experts for each word. The authors found that existing methods waste time on padding or overflow because expert capacities are fixed and don’t adjust to current workload. RankLane rearranges how experts are assigned during processing to better match their load without changing device setup, reducing wasted work. This approach leads to faster model prefill times on specific AI accelerator chips compared to traditional methods.
Open → 2611.00421v1

TranSAGE boosts transformer model speed and energy use with spatial pipelines

TranSAGE: Automating Spatial Accelerator Generation for Large-Scale Transformer Models

Abstract: Large-scale Transformers have achieved remarkable success across NLP, computer vision, and robotics, but their growing computational and memory demands challenge efficient inference. Spatial accelerators offer advantages in on-chip bandwidth and energy efficiency, yet designing them for large models remains costly and time-consuming. Existing accelerator generators typically focus on single-operation support, overlooking the benefits of multi-operation pipelining, which limits throughput and energy efficiency. We propose TranSAGE, a compiler-based framework that jointly optimizes scheduling for large-scale Transformers under multi-operation pipelining and generates corresponding accelerator implementations. Our approach integrates a design space exploration (DSE) engine with multi-operation dataflow optimization and a dynamic programming-based graph partitioning algorithm to efficiently identify pipeline-friendly operation clusters and their optimal dataflow parameters. We further design spatial architecture primitives and lowering passes to automatically generate efficient RTL implementations. Evaluations on four representative LLMs demonstrate that TranSAGE achieves up to 2.41x higher throughput and 13.04x higher energy efficiency than commercial GPU/NPU baselines. Compared with state-of-the-art accelerator generators, TranSAGE achieves up to 11.33x higher throughput and 6.07x higher energy efficiency. Our code is available at https://github.com/sjtu-zhao-lab/TranSAGE.

Fri 23 OctHardware Architecture
The gist
Large Transformer models are powerful but need a lot of computing power and memory, making them hard to run efficiently. The authors created TranSAGE, a system that automatically designs special hardware to run these models faster and use less energy by cleverly organizing multiple operations to work in a pipeline. TranSAGE uses smart algorithms to find the best way to split and schedule the model's tasks, then generates hardware designs automatically. Tests show TranSAGE can speed up processing and save energy much better than existing GPUs and similar tools.
Open → 2610.32279v1

Vision language model reduces parameters by looping layers with adapters

Retrofitting Recurrent Vision-Language Models from Pretrained Backbones

Abstract: Recent work shows that depth recurrence can be retrofitted into pretrained language models. In this work, we adapt this paradigm for visual understanding, using depth recurrence to reduce the parameter footprint of a pretrained vision-language model (VLM). To this end, we prune layers from the language backbone of the VLM and loop over a subset of the remaining layers. At each step of the loop, a linear adapter merges the block's output with the token representations from the preceding layers and projects the fused representation back into the block. As the pruning impacts the original model performance, we propose a three-stage recovery training: first, we leverage a language-only pretraining to recover the performance of the language backbone, followed by an alignment of the vision-language projector, and a full fine-tuning of the model via visual instruction tuning. In the first and the last stage, we distill the outputs of the pretrained non-recurrent model into the recurrent model, which further improves the recovery. We evaluate the proposed approach on the TinyLLaVA-1.4B backbone without introducing any new vision-language data during retrofitting. Our experiments demonstrate that the retrofitted recurrent model outperforms the unpruned baseline trained with the same recovery data while storing 23.3% fewer language parameters. We further show that the proposed retrofitting recurrence outperforms the setup of converting a recurrent LM into a VLM on the same data. Additionally, our ablations show how design choices, such as the adapter between recurrences and a larger recurrent block, lead to the improved performance of our approach. Finally, we find that our proposed model gains on reasoning-oriented benchmarks, whereas its performance on basic visual understanding benchmarks remains on par with the baseline.

Thu 15 OctComputer Vision and Pattern Recognition
The gist
Large vision-language models combine visual and text understanding but can be very big and slow. The authors show how to make these models smaller by cutting some layers and reusing the remaining ones repeatedly, using small adapters to keep information flowing correctly. They train the smaller model in steps to get back most of the original accuracy without needing extra data. This approach saves memory and even improves reasoning skills, while keeping basic visual understanding just as good as before.
Open → 2610.22075v1

Static sparse memory boosts pretrained model performance efficiently

STAPLE: Static Sparse Upcycling of Pretrained Checkpoints via Embedding Modules

Abstract: Sparse computation can scale the parametric capacity of LLMs without proportionally increasing active computation, but pretraining sparse architectures remains expensive and data-hungry. Existing sparse upcycling methods mainly target Mixture-of-Experts models, where expert homogeneity, sample inefficiency, and system overheads can limit practical scalability. Static sparse memory offers predictable access and token-specific updates that can help address these challenges, but naively integrating it into pretrained computation yields poor returns. In this work, we introduce STAPLE, a recipe built upon STEM for integrating static sparse memory into pretrained models. Whereas STEM replaces FFN up-projections with token-indexed embeddings, STAPLE retains them and adds the embeddings as a residual pathway, preserving pretrained computation while learning complementary token-specific information in a scalable manner. On OLMo-2 1B, STAPLE improves over continued dense training by 4.27 points on HumanEval, 2.80 on MBPP, 2.27 on GSM8K, and 1.65 on DROP. Average performance increases with static capacity, and improvements persist on the larger OLMo-2 7B backbone. STAPLE also outperforms static sparse baselines such as STEM and an Engram-style model, and achieves a higher average score than an MoE upcycling baseline while decoding 2x faster with less than half its GPU model memory and measured GPU energy per token in our throughput-oriented decoding evaluation. Importantly, despite introducing billions of parameters late in training, our host-offloaded implementation keeps generation throughput within 1.5% and training throughput within 3% of the dense baseline, while caching only the ~5% most frequent embeddings on GPU. These results show that static sparse memory can provide a sample-efficient, scalable, and sustainable path for expanding pretrained models without sparse pretraining from scratch.

Thu 8 OctMachine Learning
The gist
Large language models can be made bigger and better without slowing down by using sparse parts that only activate when needed, but training these sparse parts from the start is very costly. The authors introduce STAPLE, a way to add a new sparse memory that remembers specific tokens alongside an existing pretrained model, improving performance without losing existing knowledge. Their method works well on different model sizes and tasks, is faster and uses less memory than other sparse methods, and keeps speed almost as good as the original model. This offers a practical way to expand language models later, without expensive retraining from scratch.
Open → 2610.12599v1

Looped transformers cut memory use with low-bit residual quantization

ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals

Abstract: Looped Transformers improve parameter efficiency by repeatedly applying shared Transformer blocks over multiple recurrent loops, increasing computational depth without increasing the parameter count. However, KV cache memory still scales with the number of loops, becoming a key memory bottleneck that limits batch size and inference throughput. KV cache quantization can alleviate this bottleneck, but existing methods often suffer substantial accuracy degradation at aggressive low-precision regimes. We observe that looped Transformers offer a unique opportunity: KV states across loops are highly similar. Based on this observation, we propose ResidualQuant, which uses the final-loop KV states as a reference and represents the remaining loops with low-precision residuals. Our method further combines least-square scaling and rotations applied to the residuals, as well as loop-wise mixed precision, to enable accurate quantization down to INT2 while retaining efficient reconstruction. Across multiple looped Transformer models and mathematical reasoning and code generation benchmarks, ResidualQuant consistently improves the accuracy-memory tradeoff over state-of-the-art rotation-based KV quantization. In particular, our method retains accuracy close to BF16 under mixed-precision settings while reducing theoretical KV storage by 80.7%, achieving up to 13.0% higher accuracy than the rotation-based baseline at the same memory budget. On an RTX 5090, the reduced KV memory traffic improves fixed-batch decode throughput by up to 2.73x, while the smaller memory footprint enables up to 2x larger batches, improving peak throughput by up to 4.15x.

Wed 7 OctMachine Learning
The gist
Storing memory in certain advanced AI models called looped Transformers uses a lot of space, which slows down their work. The authors noticed that parts of this memory are very similar across repeated steps. Using this, they developed a method called ResidualQuant that saves the differences in very low detail instead of the whole memory, greatly shrinking the storage needed. This approach keeps the AI’s accuracy high while speeding up processing and allowing more work at once.
Open → 2610.10381v1

Looped language models save memory and computation on easy tokens

Enabling Dynamic Computation in Looped LMs

Abstract: Looped LMs are parameter efficient and promise dynamic computation (saving memory and FLOPs on easy tokens). However, state-of-the-art open Looped LMs trained with this dynamic computation capability (Ouro models) do not realize it in practice as each loop iteration (depth) requires its own level of KV-cache, necessitating all loop computations. Moreover, Ouro's early-exit prior is enforced on each token equally, which results in static lower-depth like processing of all tokens regardless of difficulty. In this work, we propose a simple "best-available" KV caching strategy that works out-of-the-box, creating a new frontier in the performance vs depth space. Our approach enables up to 30% reduction in FLOPs and KV memory while retaining full-depth performance, showing the true flexibility of Looped LMs. Furthermore, training looped LMs with awareness about this KV caching strategy improves performance and efficiency. Finally, we apply a small but effective fix to the early-exit prior enforcement objective that makes tokens exit at truly heterogeneous depths based on effort. Our findings are validated on Ouro models as well as smaller looped LMs pre-trained from scratch.

Tue 6 OctArtificial Intelligence
The gist
Language models that repeat calculations multiple times can be efficient but often end up doing all the work for every word anyway. The authors explain why previous models didn’t save much effort and suggest a new way to reuse stored information from earlier repeats, cutting down on memory and computation by up to 30%. They also improved how the model decides when to stop working on each word, so it only works as hard as needed. Their ideas were tested both on existing big models and smaller ones trained from scratch.
Open → 2610.09013v1

Activation denoising improves large language model quantization speed and accuracy

Activation Denoising: A Robustness View on Parallel vs Sequential LLM Quantization

Abstract: Post-training quantization is a powerful tool for compressing large language models. The most scalable methods quantize every layer in parallel, but quantization errors then compound through the residual stream, as no layer corrects for the errors of the layers before it. Sequential quantization accounts for this error compounding by re-calibrating each layer on the already-quantized outputs of its predecessors, yielding stronger results but at the cost of a serial schedule that becomes a bottleneck at scale. As a solution, we propose parallel quantization with activation denoising, which recovers much of the sequential benefit while keeping quantization fully parallel. Rather than re-calibrating layer-by-layer, we take a robustness perspective and model the upstream error as noise, regularizing to be robust to it through a preprocessing step followed by metric-weighted rounding. Applied at every layer, this regularization forms a depth-compounding smoothness penalty that dampens how strongly quantization errors amplify through the model. Unlike orthogonal rotations commonly used in quantization, which must preserve the model's function, we multiply the weights by a more general linear transformation. We find that the two are complementary and their effects compound. Empirically, our robustness regularization recovers a significant part of sequential quantization's benefit in a single parallel pass, at a fraction of its time. Overall, by treating compounding quantization errors as a robustness problem, we offer a principled foundation for more efficient and accurate LLM quantization at scale.

Mon 5 OctMachine Learning
The gist
Large language models need to be compressed to run efficiently, but this compression can introduce errors that add up as data moves through the model. The usual way to fix this is to adjust each layer one by one, which is slow. The authors propose a new method that treats these errors as noise and trains the model to be robust against them all at once. This method achieves much of the quality of the slower approach while running much faster, helping make large models easier to use at scale.
Open → 2610.07522v1

Scaling up machine learning models boosts jet classification performance

How to scale your HEP ML models: A recipe for robust architecture comparisons at scale

Abstract: Much of the recent progress in machine learning domains such as language models has come from scaling laws that predict performance as a function of training effort. In high-energy physics (HEP) similar behavior has now been observed. To aid further study, we present a systematic procedure to derive robust scaling laws and compare design choices on the relevant budget axes for HEP tasks. We first validate the full scaling trajectory on toy problems and then apply the procedure to multi-task transformers on the ~11 billion-jet ATLAS JetSet2 dataset, in both the compute- and data-constrained regimes. For the latter, we predict, to the best of our knowledge for the first time, the jointly optimal model size, training horizon, learning rate and batch size under early stopping. At compute-optimal scaling, we recover a near-equal $\sqrt{C}$ dependence of model and dataset size, and find that auxiliary objectives lower the primary jet-classification loss at equal compute budget. Expanding the inputs toward lower-level data systematically lowers the loss while leaving the scaling exponent nearly unchanged. The onset of the power-law regime is itself set by scale: below a threshold in dataset size the loss carries little information about high-compute scaling, underscoring the value of large, high-quality full-simulation datasets as a foundation for scaling studies and the development of foundation models in HEP.

Mon 5 OctMachine Learning
The gist
The authors found a way to understand how increasing the size of machine learning models and the amount of data affects how well these models work on high-energy physics problems. They tested their method on simple examples and a very large dataset related to particle jets from the ATLAS experiment. Their results show how to balance model size, training time, and data to get the best results efficiently. This helps guide building better models with limited computing resources. They also found that using extra tasks and more detailed input data improves model accuracy.
Open → 2610.06784v1

QATFactory improves quantized large language model accuracy for production use

QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs

Abstract: Large language model (LLM) inference is increasingly moving toward lower precision to realize the throughput of hardware accelerators, but aggressive post-training quantization (PTQ) can degrade model quality. We present QATFactory, an open-source framework for deployment-aligned quantization-aware distillation (QAD) and reinforcement learning (QARL). QATFactory simulates deployment-time quantization while performing matrix multiplications in BF16, allowing models to adapt to quantization noise without requiring training hardware that natively supports the target format; for example, it supports NVFP4 training on H100 GPUs, which lack FP4 Tensor Cores. The framework supports NVFP4, MXFP4, and llama.cpp's Q4_K format; dense and mixture-of-experts models; and both full-parameter and LoRA-based training. It exports checkpoints directly to vLLM and llama.cpp without an additional lossy conversion step or added inference overhead. With QATFactory, we conduct extensive experiments on models ranging from 8B to 230B parameters and evaluate exported checkpoints in production inference engines. Across models and formats, QAD consistently improves deployed-model quality over strong PTQ baselines. On Qwen3.5-9B, QAD achieves average benchmark accuracies of 68.9% under NVFP4 and 66.0% under MXFP4, outperforming the best PTQ results of 65.4% and 56.4%, respectively. Through our experiments, we found that although both FP4 formats quantize weights and activations at deployment, the best training strategy is format-dependent: NVFP4 generally performs better when only weights are quantized during training, whereas MXFP4 benefits from quantizing both weights and activations. At a fixed training token budget, training on fewer 32K sequences improves average accuracy by 1.9 points over training on more 4K sequences. We release the complete QATFactory training code and the resulting checkpoints.

Wed 30 SeptMachine Learning
The gist
Running very large language models on computers is faster when using simpler math, but this can make the models less accurate. The authors created QATFactory, a tool that helps train these models to work well with simpler math formats before deployment. This tool can simulate how the model will behave when using these simpler math types, even if the training computers don’t natively support them. Their tests show that QATFactory-trained models perform better in real-world environments than models just adjusted after training.
Open → 2609.39223v1

LLM inference system ensures consistent outputs across runs

Vosti: Specifying, Implementing, and Verifying Deterministic LLM Inference

Abstract: LLM inference systems may vary batch composition, prompt chunking, prefill/decode execution, and KV-cache reuse, eviction, or recomputation. These optimizations should not affect system outputs. Production systems, including vLLM's batch-invariant mode and SGLang's deterministic mode, target this goal but lack a formal system-level specification. We formalize deterministic LLM inference: under a fixed model and deployment configuration, requests with the same prompt and initial sampler state produce bitwise-identical logits at corresponding output positions across executions. Our tests find that these production modes produce different logits under some execution variations. To address this limitation, we present Vosti, an inference engine designed and verified against this specification. Vosti chooses kernels independently of runtime engine state and ties cached KV values to their logical token prefixes. Its proof decomposes at the engine/GPU kernel boundary: a Verus inductive proof establishes that scheduling and the paged, prefix-sharing KV-cache preserve output logits, while a Triton analyzer proves bitwise-equal selected kernel outputs across batches, query lengths, and paged KV-cache layouts. Vosti produces bitwise-identical logits across every tested execution variation and achieves performance comparable to vLLM's batch-invariant mode on decode-heavy workloads, while providing a stronger, formally verified determinism guarantee.

Wed 30 SeptDistributed, Parallel, and Cluster ComputingProgramming Languages
The gist
In large language models, running the same prompt multiple times can sometimes produce different outputs due to how the system handles batching and caching. The authors found that existing tools aiming for consistent results still had differences in some conditions. They created Vosti, a system that guarantees identical outputs every time by carefully managing how data is processed and cached. This work includes a formal proof showing why their approach reliably produces the same results across all tested variations.
Open → 2609.38981v1

Delta-matching enables reliable native 8-bit training for large language models

Delta-Matching: Closing the Final Gap of Native 8-bit Training for LLMs

Abstract: Reliable FP8 attention remains a barrier to fully native 8-bit large language model training. We derive how forward-backward inconsistencies produce stale delta and empirically show how it distorts training dynamics. Our stale-delta hybrid runs show a modest loss gap at 569M parameters but substantial loss increases and downstream degradation at 1.67B and 5.29B. QK normalization, NoPE (no positional encoding), and lower-learning-rate context extension mitigate or delay degradation without eliminating it. This pattern suggests accumulated optimization error that smaller models and short runs can conceal. We propose Delta-Matching, proving that it restores the softmax gradient's zero-row-sum invariant under the stated numerical assumptions. It enables native block-scaled FP8 in every forward and backward attention-core matmul without architectural changes, smaller global batches, or auxiliary forward outputs. Across tested architectures, scales, and training stages, Delta-Matching matches BF16/FP32 mixed-precision training loss and overall downstream performance. We will release our implementation, trained models, and data recipes.

Tue 29 SeptMachine Learning
The gist
Training large language models using very low-precision (8-bit) math has been difficult because of errors in how the math is done, especially in an important part called attention. The authors found that a certain type of error called stale delta messes up training for bigger models. They created a technique called Delta-Matching that fixes this error without needing complex changes to the model or training process. Their method lets 8-bit training match the results of higher-precision methods across different models and sizes.
Open → 2609.37852v1

Looped mixture of experts improves model efficiency by reshaping experts

How to Loop MoE: Flatten the Experts, Untie the Attention

Abstract: Looped Transformers reuse one block of layers several times: by spending extra computation they push a model of fixed size further, and so use its parameters more fully; while sparse mixture-of-experts (MoE) models activate only a few of many experts for each token. Looped MoE bridges these two design philosophies and gives MoE models new potential for better expert usage, but it raises a question: how to loop a MoE? We answer it with Foil. With the expert parameters and the expert compute per token held fixed, Foil (1) flattens the experts, halving the expert layers, doubling the experts per layer and doubling the passes, so that every routing decision chooses from a larger pool, and (2) unties the attention, giving each pass its own attention parameters while the experts and routers stay shared. Experiments show that Foil clearly outperforms the unflattened looped baseline: at 20B tokens every Foil model has lower pretraining loss than the baseline; at 100B tokens the loss improves monotonically with the degree of flattening, the most flattened Foil ending 0.012 nat below the baseline at equal parameters and compute, with downstream accuracy on par or better; untying the attention also yields more balanced and more confident routing at equal shape. Our ablations analyse why Foil works and turn the findings into design guidance for looped MoE: the returns of looping and of widening the expert layers amplify each other, routing confidence tracks healthy expert use better than load balance, and a sparse looped MoE should therefore use more experts per layer and more passes. Code and configurations are available at https://github.com/SR-A-W/how-to-loop-moe.

Mon 28 SeptMachine LearningArtificial IntelligenceComputation and Language
The gist
Large AI models sometimes reuse parts of themselves repeatedly to do more with less. This paper shows a way to reshape and reorganize these reused parts, called experts, so the system can choose from a bigger and better pool each time. The authors found that this approach leads to better learning and more balanced use of the parts, without increasing the model size or computation needed. This means AI models can perform better by simply looping through a clever arrangement of their own components.
Open → 2609.35751v1

Kv cache compression method retains long and off context query strength

Cartridges++: KV Cache Compression without Off-Context Derailment

Abstract: Serving long documents to a Large Language Model (LLM) repeatedly is expensive: computations grow with context length, and the memory footprint of the key-value (KV) cache balloons. Compressed KV (CKV) representations aim to mimic the cache of a document and are typically computed once and for all, ahead of inference time. Methods to obtain CKVs range from drop mechanisms that reduce their number of columns, to learned approaches. Among the latter, Cartridges have emerged as a leading compression method, learning compact KV representations through distillation on relevant Q/A pairs. While existing evaluations focus primarily on whether Cartridges and other CKVs yield approximately similar responses to document-related, on-context queries, we investigate the crucial deployment question of whether they can handle off-context queries, something the native KV representation is particularly good at, thanks to the mechanics of attention. We observe a fundamental trade-off: while Cartridges perform better for on-context queries, heuristic-variants preserve better the original LLM's ability to operate off-context. We measure this through their capability to avoid context contamination in their response, retain general knowledge, and follow instructions. We propose Cartridges++, simple modifications to cartridges that retain off-context abilities at small or negligible cost. The router variant decides at inference time whether the query should use the learned long-context memory, while the data-mixing variant allocates a small fraction of training Q/As to queries outside the reference long document. Our study shows that assessing CKVs on document utility alone can mask substantial degradation in broader model capabilities, yet those issues can be fixed with benign changes to CKV inference or training.

Mon 28 SeptMachine Learning
The gist
Working with long documents on large language models uses a lot of computer memory and time, so people try to compress the memory it needs. The authors study a popular compression method called Cartridges, which works well for questions about the original document but struggles when questions are unrelated to that document. They find a trade-off between remembering document details and handling general queries. They propose Cartridges++, which modify the compression to keep the benefits for document queries while also preserving the model’s ability to answer off-topic questions well.
Open → 2609.35621v1

Efficiently select experts to speed up large AI models

EAT: Expert Account Tracker for Efficient MoE Inference

Abstract: Mixture-of-Experts (MoE) models have emerged as a revolutionary method to scale Transformer models. However, traditional MoE architecture still suffers from inefficiency since a large number of experts are unnecessarily activated. Existing approaches for reducing the number of activated experts often overlook the historical performance of each expert. In this paper, we propose EAT, a novel method called Expert Account Tracker (EAT), which utilizes history-awareness metrics and adaptive thresholding to dynamically select the most important experts, thereby reducing the activated expert number while effectively maintaining the model performance. Experiments show that EAT outperforms the existing baseline Top-P method across multiple models and datasets, achieving over 25% an average reduction compared to the vanilla method in the number of activated experts and performing better token generation speed compared to the baseline. Furthermore, the performance of pruned models can be efficiently recovered via OPD using only 9K data. Additionally, through ablation studies, we find that excessively reducing the number of activated experts can significantly harm model performance, and the importance of experts varies across layers, with higher-level experts being generally more critical.

Sun 27 SeptArtificial Intelligence
The gist
Large AI models called Mixture-of-Experts (MoE) use many smaller modules called experts to make decisions, but activating too many experts wastes time and resources. The authors introduced EAT, a method that keeps track of how well each expert has performed in the past to smartly choose only the most useful ones for each task. This reduces the number of experts that need to be activated by over 25%, making the model faster without losing accuracy. They also show that the model’s performance can be recovered quickly if experts are pruned too much.
Open → 2609.33614v1

Oled moe speeds up large language model inference with smarter expert memory use

OLED-MoE: Accelerating MoE-Based dLLM Inference via Inter-Iteration Locality-Aware Expert Offloading

Abstract: Semi-autoregressive diffusion large language models (dLLMs) improve decoding parallelism through iterative block-wise denoising, but scaling them with mixture-of-experts (MoE) layers introduces a large expert parameter footprint that exceeds memory-constrained GPU capacity. Expert offloading is a natural remedy, yet existing MoE serving systems target autoregressive decoding and rely on intra-iteration layer-wise prefetching: while computing one layer, they predict and load experts for subsequent layers. Under dLLM inference, block-wise routing expands the active expert working set within each iteration, making such prefetches difficult to complete in time and costly when mispredicted. Consequently, existing prefetch-based solutions often degenerate into on-demand expert loading with high decoding latency. We propose OLED-MoE, an expert offloading system that shifts the optimization target from intra-iteration prefetching to inter-iteration expert retention. Its key insight is that adjacent denoising iterations exhibit strong expert routing overlap, and token confidence indicates which experts are likely to be reused. OLED-MoE uses confidence-guided inter-iteration prediction to retain high-value experts in GPU memory without introducing extra prefetch traffic. It further compensates unavoidable cache misses through CPU-GPU cooperative execution, jointly considering dynamic expert computation load and predicted future reuse. Across diverse dLLM workloads, OLED-MoE reduces time per output token (TPOT) by 1.23x-7.93x and improves expert cache utilization by 1.44x-4.23x over state-of-the-art offloading systems. Notably, OLED-MoE approaches full-residency performance while using only 40% of the expert GPU memory, incurring merely 23% higher TPOT despite a 60% reduction in expert memory footprint. OLED-MoE's source code is publicly available at https://github.com/flashserve/OLED-MoE.

Sun 27 SeptDistributed, Parallel, and Cluster ComputingComputation and Language
The gist
Large language models that use many expert components can become too big to fit into standard graphics cards, slowing down their response time. The authors found existing methods for managing these experts during decoding often miss timing targets, causing delays. They designed OLED-MoE, a system that smartly keeps frequently reused experts in fast memory across consecutive steps, using token confidence to predict reuse. This approach reduces waiting time significantly and uses memory much more efficiently while keeping performance close to ideal full-memory use.
Open → 2609.33385v1

On policy training improves efficient transformer attention on long tasks

On-Policy Attention Linearization

Abstract: Hybrid transformer architectures that replace most softmax attention layers with linear attention offer transformer-level quality at a fraction of the memory cost. Rather than pretraining such models, a growing body of work distills them from already trained full-attention transformers. However, these distilled models often collapse on long-context retrieval and reasoning tasks, particularly when operating in thinking mode, where the efficiency gains of hybrid architectures matter most. Since linear attention layers must compress context into a fixed-size state, their errors compound over long sequences. As off-policy distillation never teaches the student model to recover from this drift, tasks that necessitate longer sequence lengths become especially challenging. We introduce On-Policy Attention Linearization (OPAL) in which the hybrid attention student samples its own long-context trajectories and receives dense supervision from the frozen full-attention teacher. Applying OPAL to Qwen3-4B and MiMo-7B-RL-0530, we recover $87$--$94\%$ of full-attention performance on commonsense reasoning, $100\%$ on needle-in-a-haystack (NIAH) retrieval, and $83$--$93\%$ on mathematical reasoning with only 3B training tokens. We achieve these results without supervised fine-tuning (SFT) or reinforcement learning with verifiable rewards (RLVR). Compared with the strongest prior linearization method, which recovers $68\%$ of its teacher's retrieval performance and $21.6\%$ absolute average mathematical reasoning accuracy, OPAL fully recovers retrieval and achieves $67.6$--$72.2\%$ on math reasoning.

Fri 25 SeptMachine Learning
The gist
Transformers often use a type of attention called softmax attention that requires a lot of memory and computing power, so some newer models use linear attention to be faster and use less memory. However, these faster versions often struggle with tasks that need understanding of long pieces of information, especially in complex reasoning. The authors introduced a new way to train the faster model by letting it learn from its own long attempts while being guided by a bigger model. This method greatly improves performance on tasks like reasoning, finding tiny details, and math problems without needing extra complex training.
Open → 2609.31947v1

Hybrid sparse attention improves long-context processing efficiency

HySparse2: Hybrid Sparse Attention with Two-Level KV Sharing

Abstract: Long-horizon and multi-turn agents typically generate short actions and process long observations from tools and environments. This growing context demands efficient prefill, compact KV-cache storage, and accurate long-context retrieval. To meet these demands, we introduce HySparse2, a hybrid sparse attention architecture with two-level KV sharing. At the outer level, KV Bridging adopts a YOCO-style self-decoder and cross-decoder structure, but bridges only full-attention layers. The self-decoder uses hybrid sliding-window attention (SWA), while the cross-decoder uses hybrid sparse attention. The KV caches for full-attention layers in the cross-decoder are generated from the hidden states of full-attention layers in the self-decoder. At the inner level, HySparse2 retains HySparse's core KV Reuse design with two refinements. First, it replaces block-level sparsity with token-level sparsity for finer long-context retrieval. Second, it removes the separate SWA branch from sparse layers and instead forces a sliding window of recent tokens into the sparse selection. This two-level KV sharing allows all cross-decoder KV caches to be constructed from self-decoder hidden states. Prefill can therefore exit after the self-decoder, skipping all cross-decoder layers. On an 80B-A3B MoE model, HySparse2 outperforms HySparse and Hybrid SWA on long-context retrieval and multi-turn agentic tasks, while substantially reducing prefill computation and KV-cache storage.

Tue 22 SeptComputation and Language
The gist
Long-running AI agents often need to understand and remember lots of information while working on many small tasks. The authors introduced HySparse2, a new way for AI models to handle this big amount of information more efficiently. It combines different attention methods and shares intermediate data cleverly to save memory and speed up processing. Their approach helps AI remember more details over longer conversations or actions while using less computing power.
Open → 2609.26368v1