Papers for

ai hardware engineers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Heterogeneous 3d chip speeds up large language model inference fivefold

Tri-Stack-PIM: A Heterogeneous Multi-Die 3D-Integrated Processing-in-Memory Architecture for Efficient LLM Inference

Abstract: Large language model (LLM) inference during autoregressive decoding is fundamentally memory-bandwidth-bound. Existing three-dimensional (3D) stacked processing-in-memory (PIM) architectures alleviate the off-chip bandwidth wall, yet remain constrained by several limitations: physical layer (PHY), controllers, and direct memory access (DMA) engines compete with arithmetic units for advanced-node base-die area; hybrid-bonding keep-out zones fragment the base-die floorplan and impede physical design; compute tiles built in a dynamic random access memory (DRAM) process are restricted to low logic density, capping achievable throughput; and nonlinear operations offloaded to the host reintroduce a communication bottleneck. We present Tri-Stack-PIM, a heterogeneous three-die architecture that resolves these tensions by dedicating a top DRAM die to storage, a middle buffer die on a mature process node to PHYs, DMA engines, and near-memory compute units, and a bottom base die on an advanced node exclusively to single-instruction-multiple-data (SIMD) multiply-accumulate (MAC) arrays. We further introduce a Unified Buffer-Aware Ready-based (UBAR) hardware out-of-order micro-operation scheduler that enforces cross-die dependency correctness at runtime while maximizing issue-slot utilization through a multi-term priority function, recovering up to 90% of the cycles that an in-order policy would forfeit. A joint architecture-dataflow co-exploration identifies Pareto-optimal configurations that reduce the energy-delay product (EDP) by 33-66x over naive parameter selection. Evaluated across multi-head, grouped-query, and multi-query attention (MHA, GQA, MQA) mechanisms with batch sizes 1-32, Tri-Stack-PIM achieves geometric-mean speedups of 4.95x, 2.57x, and 1.68x across all three attention mechanisms and batch sizes over a graphics processing unit (GPU) baseline, AttAcc, and H2-LLM, respectively.

Tue 10 NovHardware Architecture
The gist
Running large language models (LLMs) can be very slow because moving data between memory and processors is a bottleneck. The authors propose a new chip design called Tri-Stack-PIM that stacks three different layers of memory and processors to work together more efficiently. This design dedicates each layer to a specific task, improving speed and saving energy. They also created a smart scheduler to better coordinate tasks across these layers, recovering most lost cycles from simpler methods. Their tests show significant speed gains compared to current graphics processors and other specialized designs.
Open → 2611.13644v1

Comparing vector and CGRA chips for AI nonlinear functions

To SIMD or Not to SIMD: Comparing RVV vs. CGRA SoCs for Non-Linear Functions in AI

Abstract: Non-linear functions are an important class of operations in modern large language models. However, chip vendors building for AI have so far relied on traditional vector or scalar processing units for non-linear functions. The performance of these units has not kept pace with matrix multiplication units (MMUs), which can be found in both high-performance and edge-computing devices from almost every vendor. Reconfigurable units such as FPGAs and CGRAs have been proposed in the literature, but have not been systematically compared with traditional units backed by MMUs. In this paper, we build two SoCs: one using an RVV vector processor and the other using a latency-insensitive (\emph{elastic}) CGRA, both coupled with an MMU and driven by a single-core CPU. We then propose mappings for common feed-forward networks on our SoCs and find that the CGRA reaches a maximum speedup in clock cycles of 7.12x over RVV on the activation stage of simple point-wise non-linear functions like SiLU, whereas the RVV reaches a maximum speedup of \textbf{9.8x} over the CGRA on complex non-linear functions such as $\mathrm{softmax}$. Over the enclosing feed-forward block the corresponding maxima are 3.16x and 4.24x.

Mon 9 NovHardware ArchitectureArtificial Intelligence
The gist
Non-linear functions play a big role in AI models, but the common chips used to run them haven’t improved as fast as those for matrix math. The authors built and tested two types of computer chips to see which works better for these functions: one using a common vector processor and the other using a more flexible compute design called CGRA. They found that CGRA chips are quicker on simpler functions like SiLU, while vector processors perform better on complex functions like softmax. Overall, each chip type has strengths depending on the function complexity in AI tasks.
Open → 2611.12435v1

Phantom accelerates diffusion image generation with temporal caching

Phantom: Temporal Caching for Diffusion Models

Abstract: Diffusion models have become the dominant paradigm for high-quality image generation, but their iterative inference across tens of timesteps incurs substantial computational overhead. Prior accelerators exploit temporal similarity in diffusion models via differential quantization or output sparsity, yet suffer from increased memory footprint, design complexity, and image degradation. In this work, we present Phantom, an algorithm-architecture co-designed accelerator built on temporal caching. We view temporal similarity as differential sparsity: between adjacent timesteps, only a small fraction of tokens (outliers) change significantly. Phantom recomputes only these outlier tokens and reuses the rest from a temporal cache. Phantom runs on standard systolic arrays with lightweight additions: a Phantom Engine at the global buffer that detects outliers and updates the cache as activations stream through, and a condensed pipeline that maps the resulting sparse updates onto dense systolic computation for FC, convolution, and attention layers. Evaluations on modern diffusion models show that Phantom achieves an average 4.0x speedup and 5.5x energy saving over GPUs and prior accelerators while preserving generation quality.

Wed 4 NovHardware Architecture
The gist
Generating images with diffusion models takes a lot of computing power because the process repeats many steps. The authors see that changes between these steps mostly affect only small parts of the data. Their system, called Phantom, saves and reuses parts that don’t change, only recomputing what’s different. This approach lets it run about four times faster and use less energy while keeping picture quality high.
Open → 2611.06019v1

Analog and hybrid computing improves energy efficiency and specific computations

Analog and hybrid computing in the 21st century

Abstract: Analog computers solve problems through physical representations of mathematical variables and relations, often using the dynamics of electronic circuits and their associated signals. Many analog computers directly represent nonlinear dynamical systems; they encode their solutions in the signals that they generate. Although most modern scientific computing relies on digital architectures with inherently discrete-valued representations and algorithmic approaches, the design constraints of such architectures increasingly limit both performance and energy efficiency. Analog systems can circumvent some of these constraints; they execute specific computations very efficiently by mapping them directly onto physical processes using smaller and simpler computing elements than the ones in digital architectures. In this article, we provide an interdisciplinary perspective on modern analog and hybrid (i.e., analog-digital) computing. We discuss how modern scientific computing workflows can leverage analog approaches in various applications, such as in simulating dynamical systems, performing optimization, and enabling more energy-efficient implementations of artificial intelligence.

Fri 30 OctArtificial IntelligenceEmerging Technologies
The gist
Digital computers use switches that are either on or off to perform calculations, but this approach can limit speed and use a lot of energy. The authors explain that analog computers work differently by using continuous physical processes to solve problems directly. They show that combining analog and digital computing can make some tasks, like simulating complex systems or running AI, more energy-efficient and faster. This paper looks at how modern computing methods can include analog approaches alongside digital ones.
Open → 2611.00446v1

RankLane speeds up mixture-of-experts model prefill on AI hardware

RankLane: Efficient MoE Prefill on Accelerators with Ranked Expert Capacities

Abstract: Mixture-of-experts (MoE) models expand model capacity by activating only a subset of experts for each token. On accelerators with static execution plans, expert capacities and placement are fixed before routing reveals each expert's load. Uniform worst-case capacities waste work on padding; smaller capacities tied to expert identities can overflow. We present RankLane, which ranks each device's experts by current load and assigns them to fixed-capacity lanes. Runtime indices select experts and token rows while tier populations, shapes and device placement remain fixed. An aggregation graph preserves device ownership and reduces each token's local contributions on one core. Routing calibration and measured execution costs guide tier selection; overflowing chunks rerun at full capacity. We evaluate full-model prefill of Qwen3-30B-A3B and Gemma 4 26B-A4B with MXFP6 weights on four Cloud AI 100 Ultra SoCs, using 128-2048-token chunks and including overflow detection and in-program recovery support. Against a dense baseline, RankLane achieves end-to-end prefill speedups of 1.27-4.18x on Qwen3 and 1.93-3.46x on Gemma.

Fri 30 OctMachine Learning
The gist
Large language models sometimes use mixture-of-experts (MoE) to handle more data by activating only some parts called experts for each word. The authors found that existing methods waste time on padding or overflow because expert capacities are fixed and don’t adjust to current workload. RankLane rearranges how experts are assigned during processing to better match their load without changing device setup, reducing wasted work. This approach leads to faster model prefill times on specific AI accelerator chips compared to traditional methods.
Open → 2611.00421v1

Ising machine chip cuts energy for context selection in AI models

p-DCIM: A 28nm Probabilistic SRAM DCIM based Ising Machine for Near-Memory Context Selection in LLM-RAG Pipeline

Abstract: We present a fabricated SRAM DCIM Ising machine that performs set-level context selection for large language model (LLM) with retrieval-augmented generation (RAG). In RAG, the evidence-selection stage is increasingly delegated to repeated language-model reranking, which adds substantial energy to the inference pipeline. We reformulate evidence selection as a quadratic unconstrained binary optimization (QUBO) problem and solve it on a dual-mode digital compute-in-memory (DCIM) macro: the same multiply-and-accumulate datapath computes the embedding-similarity scores that populate the QUBO coefficients and evaluates the per-spin Hamiltonian differences that drive the anneal. The annealing temperature is set by a digital plane-count control along with the SRAM pseudo-read random bit flips, giving graded cooling entirely from on-chip digital configuration. The macro is taped-out in a 28 nm process [0.16 mm2, 0.41 mW] and is proposed to be integrated as part of the logic base-die in hybrid high-bandwidth memory (HBM) and high-bandwidth flash (HBF) stacks in a GPU package, where only the selected passage identifiers return to the host. Replacing the language-model selection pass with the near-memory Ising solver yields a macro normalized selection-energy estimate more than six orders of magnitude lower than the LLM-based baseline and reduces the evaluated total energy per query by 67%, while maintaining comparable token-level F1 on the evaluated examples.

Tue 27 OctHardware Architecture
The gist
Large language models that use retrieved information need to pick the most useful pieces of context before answering questions. This picking process usually takes a lot of energy because it involves running the language model multiple times. The authors created a tiny computer chip that solves this selection problem more efficiently by using a special memory circuit that performs calculations directly inside its own cells. Their chip uses way less energy—over a million times less for selection—and speeds up the process without losing accuracy, which could help devices run smarter AI with less power.
Open → 2610.37033v1

Analog memory stores multiple vectors in single array for ai models

Superposed analog memory for vector storage in neural networks

Abstract: The growing memory demands of artificial neural networks pose a major challenge for energy-efficient computing, particularly for models that rely on large collections of high-dimensional weight and embedding vectors. Conventional memory architectures store these vectors individually, requiring many memory cells per vector and limiting achievable storage density. Here we demonstrate an analog memory architecture that encodes multiple vectors in a distributed superposition within a single memory array. Individual vectors are recovered through in-memory matrix-vector multiplication, allowing the degree of superposition to be tuned to balance storage density against recall fidelity. This approach introduces a new scaling dimension for analog memory, in which effective storage density can be increased without a corresponding increase in the number of physical memory cells. The distributed representation also provides resilience to device-level noise. We implement the architecture on a prototype phase-change memory chip and experimentally evaluate it using embedding tables from deep learning recommendation models and large language models. The results demonstrate inference accuracy comparable to full-precision software baselines, establishing analog superposition as a promising route towards dense, energy-efficient memory for artificial intelligence.

Mon 19 OctEmerging TechnologiesHardware Architecture
The gist
Artificial intelligence models need a lot of memory to store their data, which can be energy-consuming. The authors show a new kind of analog memory that can hold many data vectors all mixed together in one place and still recover each one accurately. This means more data can fit in the same physical space, saving energy and resisting noise. They tested this on special memory chips using examples from language and recommendation models and found it worked as well as traditional digital methods.
Open → 2610.26118v1

Transformers stay stable and robust with dimensionless scaling rules

Neural Dimensional Analysis: Foundations of Dimensionless and Scale-Consistent Computation

Abstract: Modern Transformers rely on many scale-control mechanisms, including normalization, residual scaling, attention temperatures, loss weighting, and adaptive updates. To explain how these mechanisms maintain scale consistency, we introduce Neural Dimensional Analysis (NDA), a unified framework based on dimensional consistency. We define neural units that specify how tensors rescale and which quantities can be added, just as physical units do in physics. Our rules match residual units, nondimensionalize attention and gate inputs, combine losses across units, and convert gradient units into parameter units. We implement NDA through dimensionless Unit Violation Indices (UVI) for diagnosing scale mismatches within Transformers, a unit-consistent multi-loss objective, and the Gauge-Canonical Transformer (GCT) with normalized coordinates and orthogonalized residual additions. In language-model experiments spanning 51M to 1B parameters, GCT remains stable during training across a broad sixteenfold learning-rate range, outperforming Pre-LN, LLaMA-style, and QK-Norm in 124M validation loss at tuned learning rates. GCT's normalized residual stream also shows improved robustness to quantization. At 1B parameters, calibrated per-tensor int8 quantization increases GCT's validation loss by only 0.009 nats, compared with up to 5.51 nats for the baselines. Vision and physics-informed neural network experiments further confirm NDA's benefits beyond language modeling. These results establish dimensional consistency as a neural network design principle.

Tue 13 OctMachine LearningNeural and Evolutionary Computing
The gist
Transformers, a type of AI model, use many tricks to keep their calculations balanced during training. The authors present Neural Dimensional Analysis (NDA), a way to treat these calculations like physical units so their sizes stay consistent. Using NDA, they build a new Transformer variant that trains well over a wide range of learning speeds and handles tiny number approximations better. Their experiments show this approach works not only for language AI but also in vision and physics tasks. This helps explain and improve how Transformers handle different sized values internally.
Open → 2610.17969v1

Quantized pretraining scaling laws improve model size and precision choices

Mixed Precision Scaling Laws for Quantized Pretraining

Abstract: Since scaling laws were introduced for modern LLMs (Kaplan et al., 2020), several works have extended them to account for quantization, tracking the emerging low-precision data types supported on AI accelerators. We develop a more accurate scaling law for quantized pretraining with mixed precision across the forward and backward passes, grounded in optimization theory and validated empirically. Our formulation separates the effects of quantization on model capacity and on optimization efficiency, and explicitly models the persistent optimization error from forward and backward quantization identified through convergence analysis. To select the regimes for fitting and evaluating our law, we first decompose existing 4-bit training recipes into their forward and backward components, isolating those that most affect loss. Our Qwen3 experiments under selected 4-bit regimes, together with published Llama-style experiments, show that our law predicts loss with up to $11\times$ lower MSE than existing laws as model size and token count scale. Finally, we extend our law to layer-wise mixed precision and identify the optimal quantized-pretraining recipe, i.e., model size, training tokens, and per-layer forward/backward precision, under a hardware-aware compute budget. Unlike prior precision scaling laws, our law predicts that the optimal precision increases with the compute budget.

Tue 13 OctMachine Learning
The gist
Training large AI language models uses a lot of computing power, and lowering the precision of calculations can save resources but might hurt learning quality. The authors create a new mathematical law that predicts how lowering precision in both forward and backward training steps affects model training accuracy. Their model separates the impact of precision on a model's ability and on how well it learns, and it matches observed results better than previous laws. This helps find the best combination of model size, training amount, and precision per layer to make training efficient given hardware limits.
Open → 2610.17637v1

Spiking language models improve meaning with fewer spikes and better math

Rethinking the Tradeoff Between Temporal Encoding and Nonlinear Computation in Spiking Language Models

Abstract: Spiking language models face a tradeoff between representing continuous semantic features over short temporal windows and retaining costly nonlinear attention operations. We introduce Spora, which jointly designs spike encodings and attention operators. Binary temporal weights let $T$ spikes represent compositional values with up to $T$ bits of capacity, compared with $O(\log_2 T)$ bits for spike-count readout. Unipolar Binary Spiking (UBS) uses thresholds and spike-triggered residual decay to produce non-negative integer codes; Bipolar Binary Spiking (BBS) separates sign and magnitude and learns a scale for signed activations. These representations support accumulation-and-shift dot products and integer-exponent mappings in attention. With four time steps, Spora achieves 76.6 average GLUE score and 44.1 CoLA MCC, improving over SpikeLM by 1.2 and 6.2 points, respectively. Extending BBS to six steps raises these scores to 78.2 and 47.4. Conditional-decay analysis, matched-budget activation-quantization comparisons, event-workload statistics, and fixed-point evaluation further characterize the connection between encoding fidelity and computational cost.

Wed 7 OctMachine LearningArtificial Intelligence
The gist
Understanding language with spiking models is tricky because they have to balance quickly representing meanings and using complex math that costs time. The authors designed a new system called Spora that uses smart ways to encode information in spikes and handle attention calculations more efficiently. This lets the model understand language better with fewer spikes and steps. Their approach improves performance scores on standard language tests compared to previous spiking models.
Open → 2610.10933v1

Loop flow transformers improve image generation with less computation

LiFT: Loop Flow Transformers

Abstract: We introduce Loop Flow Transformers (LiFT), a family of looped generative models that scales computation by repeatedly applying a shared Diffusion Transformer (DiT) core, with only light changes to the standard architecture. Rather than asking every recurrent step for the final prediction, LiFT trains each step with a single regression target: a point on a straight path from the model's initial estimate to the flow-matching target. Because we index these targets by a continuous depth coordinate, a trained model can loop far beyond its training depth with no retraining, early exits, or other modifications. In our experiments, these longer rollouts improve generation, so inference computation can grow without adding parameters. On ImageNet at 256x256, LiFT-L/2 achieves an FID 3.34 points lower than our dense DiT-XL/2 baseline while using approximately 60% fewer parameters, 32% fewer training FLOPs, and 52% fewer inference FLOPs.

Sun 4 OctMachine LearningArtificial Intelligence
The gist
Generating high-quality images with AI often requires a lot of computing power. The authors present a new model called Loop Flow Transformers that repeatedly uses the same core process multiple times during generation. Instead of predicting the final image all at once, their method gradually improves the image over several steps, each trained to move closer to the target. This approach allows the model to generate better images with fewer parameters and less computation.
Open → 2610.05538v1

Loss scale imbalance affects optimizing language model quantization

The Devil Is in the Reconstruction Loss Scale: Rethinking Optimization in LLM Quantization

Abstract: Post-training quantization (PTQ) methods typically use sequential quantization that partitions a pre-trained LLM into a series of units (e.g., transformer blocks), with one unit quantized at each stage. State-of-the-art PTQ methods are predominantly learning-based, optimizing auxiliary quantization parameters (e.g., scaling factors, rotation matrices, clipping thresholds, and adapters) via gradient descent to minimize a reconstruction loss. A common practice is to use mean squared error (MSE) as the reconstruction loss function, yet its induced optimization behavior remains largely unexplored. In this work, we take a holistic view of sequential quantization and systematically investigate how optimization evolves from the first quantization stage to the last, aiming for a deep understanding of optimization in learning-based PTQ schemes. Through extensive empirical studies spanning representative learning-based PTQ methods, LLM families, model scales, architectures, quantization settings and various tasks, we consistently uncover Optimization Imbalance: reconstruction loss magnitudes vary dramatically across stages, accompanied by highly uneven gradient magnitudes and parameter updates under MSE. We term the cross-stage range of loss magnitudes the reconstruction loss scale, and reveal that MSE translates the unexpectedly large reconstruction loss scale into highly uneven gradient magnitudes, which in turn lead to uneven optimization strength across quantization stages. This finding suggests a general principle for improving learning-based PTQ: optimization strength across stages should be decoupled from the reconstruction loss scale. Theoretically, we show that root mean squared error (RMSE) variants defined at the sample, channel, token, and element levels naturally realize this principle through implicit gradient normalization, outperforming MSE significantly as a drop-in replacement.

Thu 1 OctComputation and LanguageArtificial Intelligence
The gist
Quantizing large language models (LLMs) helps make them run faster and use less memory, but it can be tricky to keep their accuracy. The authors found that the usual way to measure error during quantization, using mean squared error (MSE), causes uneven optimization across different model parts because it creates very different error levels at each stage. They discovered that using root mean squared error (RMSE) instead balances out these differences, leading to better and smoother optimization. This finding can improve how LLM quantization methods are designed and applied.
Open → 2610.00983v1

Looped transformers quantization fails without feedback and calibration fixes

Quantizing Looped Transformers: Feedback Exposure and Calibration Blindness

Abstract: Looped transformers reuse weights across recurrence steps, making low-bit quantization especially attractive. We identify two distinct failure modes of standard post-training quantization. On Huginn-3.5B, per-channel INT4 fails primarily at the non-residual loop-entry adapter, while quantizing the residual core is much less damaging. We call this feedback exposure: a quantized layer perturbs the recurrent state without an identity path, and the resulting error is fed back at later steps. Controlled experiments on linear filters and Mamba state-space models show that feedback exposure also occurs outside transformers. Grouped INT4 reveals a separate failure, calibration blindness: our one-step GPTQ baseline builds its Hessian from step-0 activations, leaving input directions used later in the recurrence nearly unweighted. Across nine checkpoints from seven looped architectures, one-step GPTQ is worse than round-to-nearest (RTN) on the primary task metric for five checkpoints. Accumulating the GPTQ Hessian across recurrence steps outperforms both one-step GPTQ and RTN on all nine checkpoints and recovers bf16-level accuracy on Huginn. These results separate two questions for PTQ on looped models: where quantization error enters the recurrence, and which states calibration sees.

Fri 25 SeptMachine LearningComputation and Language
The gist
Quantizing looped transformers—in which the same weights are reused repeatedly—can cause errors that get worse over time. The authors found two main problems: one where errors build up because a key part lacks a direct 'identity path' to keep things stable, and another where the calibration method misses important later signals. They showed that by improving how calibration collects information across multiple steps, this error can be greatly reduced, matching higher-precision results. These insights help make very low-bit quantization of looped models more reliable.
Open → 2609.30820v1

Efficient programming of AMD XDNA NPUs boosts attention model speed

Programming AMD XDNA NPUs with Open-source Compiler Tools: A FlashAttention Case Study

Abstract: Spatial NPUs such as AMD XDNA place compute tiles beside small local memories and leave data movement between them to software. Mapping a multi-stage workload onto such a device is largely a question of where the intermediate tensors live. We report what we learned making those choices for FlashAttention with the open-source IRON and MLIR-AIR flows. We compare four reference designs on XDNA 1 and XDNA 2: one runs each operator separately, two stream between operators on chip, and one fuses all three attention stages into a single kernel. The fused kernel holds the $\boldsymbol{QK}^{\mathsf T}$ scores in compute-tile local memory and reduces partial results over the cascade interconnect, so the scores never return to shared MemTile memory. On XDNA 2, it reaches 3.62 TFLOP/s over complete end-to-end execution, twice the IRON design, with 5.3 to 7.2 times the energy efficiency of the integrated GPU on the same chip at 2K tokens and above. It covers twelve LLM configurations, from BERT to DeepSeek, up to 128K tokens. Roofline analysis at each memory level explains this result and shows when to stop. XDNA 1 has lower ridge points, so streaming on chip already reaches the compute-bound regime: the same fusion that doubles throughput on XDNA 2 is nearly wasted on XDNA 1. Comparing a mapping's operational intensity against each level's ridge point predicts which case applies before writing any code. Fuse until the mapping clears that ridge point, then stop. We release the reference designs as maintained open source.

Fri 18 SeptHardware ArchitectureMachine Learning
The gist
Running large, complex AI models on special chips called NPUs requires deciding where to store data for best speed. The authors studied how to organize and run parts of an important AI technique called FlashAttention on AMD’s XDNA NPUs. They compared different designs and showed that combining all steps into one fused program runs twice as fast on newer XDNA chips and uses much less energy than integrated GPUs. Their results cover many versions of large language models, and they share their working code as open source.
Open → 2609.21264v1

Semantic aware error recovery improves AI data movement latency

SAGE: Semantic-Aware Geographic Error Recovery for AI Data Movement

Abstract: AI interconnects typically protect and replay packets uniformly, yet numerical bit faults differ sharply in consequence: a low-order mantissa flip may resemble quantization noise, while a high-significance exponent flip can produce a catastrophic outlier or non-finite value. We present SAGE, a semantic-aware geographic error-recovery architecture that decouples whether a detected fault merits replay from where replay restarts. For BF16-like data, a workload-calibrated contract separates catastrophic Class-H faults from bounded Class-M and precision Class-L damage. It first applies a Class-H silent-delivery constraint, then ranks admissible policies by quality-normalized terminal latency, $Ψ_{\rm del}$. Independently, a source-local region table adapts checkpoint intervals to fault geography, shortening recovery segments in noisy regions. Detected Class-H failures may trigger protected negative acknowledgments and full-flit replay; Class-M and Class-L outcomes do not trigger default network replay. We implement SAGE's endpoint and replay protocol in gem5 Garnet and synthesize its fully pipelined checker in ASAP7. At a stable synthetic operating point, a ten-seed contention-faithful direct-Garnet campaign shows that SAGE reduces $Ψ_{\rm del}$ by 30.1% relative to fixed 34-hop recovery, combining 28.0% lower mean latency with improved delivered semantic quality. Under higher-BER synthetic stress at the same offered load, SAGE maintains bounded queues while the fixed baseline accumulates backlog. Application-derived DeiT-S communication traces also show lower mean and p99.5 latency at the evaluated nonzero BERs. Within the qualified operating envelope, CRC32 decoder trials yield a simultaneous 95% per-original Class-H silent-delivery upper bound of $3.18\times10^{-7}$.

Wed 9 SeptHardware Architecture
The gist
Data errors happen during AI communication, but not all errors are equally serious. The authors developed SAGE, a method that recognizes which errors matter most and fixes only those, speeding up recovery. SAGE also adapts to where errors occur, making the whole system faster and more reliable. This approach reduces delays and keeps AI data quality higher during movement.
Open → 2609.10126v1

Hybrid scheduling improves mixture-of-experts processing on 3D memory chips

HDA-MoE: Hybrid Parallelism and Dynamic, Adaptive Scheduling for Mixture-of-Experts with 3D Near-Memory Processing

Abstract: Mixture-of-Experts (MoE) architectures have become a key technique for scaling Large Language Models (LLMs), enabling high model capacity with reduced computational cost. However, this efficiency comes at the expense of increased memory capacity and bandwidth demands. Recent 3D Near-Memory Processing (NMP) architectures, which vertically integrate memory and compute through hybrid bonding, provide high internal bandwidth and energy efficiency, making them attractive for accelerating MoE inference. Nevertheless, the distributed memory and compute organization of NMP systems introduces new challenges for mapping MoE workloads. Existing parallelization strategies, such as Tensor Parallelism (TP) and Expert Parallelism (EP), suffer from either high communication costs or unbalanced computation utilization, leading to inferior efficiency. In addition, the dynamic routing behavior of MoE models further complicates efficient deployment. To address these challenges, we present HDA-MoE, a framework that optimizes MoE execution on NMP architectures through hybrid parallel deployment and runtime scheduling. HDA-MoE integrates an offline hybrid parallel mapping algorithm with an online dynamic and adaptive scheduling mechanism to reduce communication overhead while improving computation utilization. Experimental results show that HDA-MoE achieves a speedup of 1.1x--3.4x over TP, 1.1x--1.5x over EP, 1.1x--3.7x over the Hybrid TP-EP compute-balanced baseline, and 1.1x--1.3x over HD-MoE. Source code is available at https://github.com/PKU-SEC-Lab/HDA-MoE-TCAD26.

Tue 8 SeptHardware Architecture
The gist
Large language models use a technique called mixture-of-experts to handle complex tasks efficiently, but this method requires a lot of memory bandwidth and capacity. The authors explore special computer chips that stack memory and processing units closely together to speed up this technique. They introduce a new way to organize and schedule work on these chips that balances communication and computation better than previous methods. Their approach makes the processing faster while using resources more effectively.
Open → 2609.08682v1

WIDER method improves latent reasoning diversity in language models

Think Wider: Mitigating Latent Rank Collapse in Implicit Chain-of-Thought Reasoning

Abstract: Chain-of-thought (CoT) reasoning improves the reasoning ability of large language models by introducing intermediate computation, but explicit rationales increase decoding length, latency, and context cost. Implicit CoT offers a more efficient alternative by moving intermediate reasoning into continuous latent states. However, latent reasoning can be unstable: successive latent states may become overly similar and collapse toward a shared dominant direction, reducing the diversity of the reasoning trajectory. In this work, we identify $\textit{latent rank collapse}$ and propose $\textbf{WIDER}$, a lightweight spectral regularizer for implicit CoT. During training, WIDER estimates the shared direction of each latent trajectory and penalizes projections onto this direction, encouraging latent states to span a broader representational subspace. The method is plug-and-play and leaves the backbone model, latent schedule, and inference-time decoding procedure unchanged. We further formulate this collapse as a geometric bottleneck in implicit reasoning, casting its mitigation as a training-time regularization problem rather than an inference-time decoding change. Extensive experiments show that WIDER improves matched implicit CoT baselines, while mechanistic analyses reveal higher effective rank, lower dominant-direction energy, and reduced redundancy among latent steps. These results highlight latent subspace utilization as an important factor for efficient continuous reasoning, providing a geometric perspective for analyzing and improving implicit CoT. Code is available at https://github.com/whitesweater/WIDER.

Mon 7 SeptMachine Learning
The gist
Large language models can think step-by-step to solve problems, but writing out each step takes time and space. The authors found that when these reasoning steps are kept inside the model's memory (latent states) instead of written out, these steps can become too similar and less useful. To fix this, they created WIDER, a training trick that encourages these memory states to be more varied and spread out. This makes the model’s internal thinking clearer and improves its problem-solving without slowing it down.
Open → 2609.07406v1