hardware system designers

Abstract: Unrolled variational networks achieve state-of-the-art performance for accelerated MRI reconstruction, but standard end-to-end training becomes prohibitively memory intensive as the number and width of cascades increase. We propose a memory-efficient training framework that updates a randomly sampled cascade while retaining supervision from the final reconstruction loss. The error signal is propagated from the end of the chain to the sampled cascade using an activation-light backward pass that avoids storing full chain activations, thereby decoupling training memory from chain length. A time-conditioned network shared across cascades further keeps the parameter count independent of chain length. The backward pass is approximate only through frozen normalization statistics, and a trained exact gradient control shows the approximation leaves reconstruction quality unchanged. The framework remains trainable at chain lengths and cascade widths where standard end-to-end backpropagation exhausts GPU memory. On fastMRI single-coil knee and multi-coil brain reconstruction, our method consistently outperforms other local training approaches. On high-SNR knee slices, our method surpasses E2E-VarNet at $8\times$ and matches it at $4\times$, while on the complete brain validation set it surpasses E2E-VarNet at $8\times$ and achieves comparable performance at $4\times$, with fewer parameters, lower peak memory, and faster per-iteration training. Code is available at https://github.com/emehillaj/scalable-unrolled-mri.

Abstract: We study structural properties of polynomial ideal (PI) codes and additive Reed--Solomon (aRS) codes in the folded Hamming metric (equivalently, the Hamming metric over a larger alphabet). We first determine the maximum possible length of PI codes, prove that the dual of every PI code is a generalized PI code, and deduce that PI codes are dually QMDS together with an explicit formula for their weight distribution. We then show that PI codes defined by irreducible polynomials correspond to aRS codes with pairwise nonconjugate evaluation points. For these aRS codes, the previous results on PI codes apply (duality, QMDS property and weight distribution). We show that, asymptotically, aRS codes with nonconjugate evaluation points achieve the same length as PI codes. We also compute the permutation group of such aRS codes. We also extend the family of PI codes to consider arbitrary polynomial modules (not necessarily pairwise coprime), and study when these and general aRS codes satisfy the QMDS and dually QMDS properties.

Abstract: Residual connections propagate information through a network by additively accumulating layer outputs. Recently proposed alternatives, such as Manifold-Constrained Hyper-Connections (mHC) and Attention Residuals (AttnRes), instead use more expressive depth aggregation mechanisms that improve language modeling performance at scale. In this work, we consider the mixing of representations over depth as a problem of storing and retrieving information across layers. Formalizing this perspective, we introduce Depth Memory (DM), a unified framework that describes the standard pre-layer-norm (PreLN) transformer, mHC, and AttnRes in terms of streams that carry representations through the network. One such stream, for example, could be an explicit memory stream that is queried to enrich the input to every layer, and whose output is subsequently written back into the memory for use by later layers. For AttnRes, this memory stream is a growing cache of explicit layer representations. For mHC, it is an expanded residual stream. Building on this view, we propose Linear Depth Memory (LDM), which formulates information retrieval across depth as a regression problem solved using recursive least squares. The resulting architecture propagates three streams across depth: the standard residual stream, a linear associative memory stream, and an inverse covariance stream. For high-throughput training and inference, we propose a hardware-aware implementation of LDM that reduces peak activation memory and memory traffic. LDM consistently achieves the lowest perplexities among PreLN, mHC and AttnRes baselines at the 400M and 1.5B parameter scales, alongside competitive zero-shot accuracy.

Abstract: We perform the first comprehensive experimental characterization of power consumed by real HBM2 DRAM chips. Our extensive characterization yields 1) the empirical basis for developing new, accurate HBM power models or validating existing power models and 2) key insights into structural and data-pattern-dependent variation of power in HBM2 chips. Out of our 13 new findings and 6 key takeaway messages, we highlight: 1) a read operation consumes significantly higher current than any other operation and 2) the toggled data bus signals in a data transfer burst significantly affect current draw where toggling all HBM2-internal bank group bus signals causes 29.4% higher power consumption than keeping them constant. Leveraging our empirical insights into HBM2 power consumption, we engineer an access pattern that yields 23.6% higher power consumption on average across 36 tested HBM2 chips than the highest-power-consuming JEDEC-specified current measurement loop. Our results have important implications for HBM-based system design and validation. Relying solely on JEDEC- or vendor-specified test loops to bound HBM power could cause system failures and/or thermal throttling in real-world high-power AI/ML workloads. To improve 3D-stacked DRAM power modeling in open source architectural simulation infrastructures, we develop a new HBM power model. Our model predicts the power of 36 real HBM2 chips across diverse workloads (including a real LLM decode workload) with 7.6 -- 8.1% mean absolute percentage error, substantially more accurately than state-of-the-art HBM power models. The power consumption predicted by our model for a power-intensive microbenchmark closely matches (with 2.4% to 5.3% error across HBM3E's specified supply voltage range) the average HBM3E memory power consumed by three tested NVIDIA H200 GPUs. We publicly release our power model at https://github.com/CMU-SAFARI/HBM-Power.

Abstract: Analog computing with microwave signals can perform linear transformations directly in the analog domain, as the signals propagate through a microwave network. A fundamental question is which transformations can be computed with a given set of microwave components. In our previous work, we answered this question for networks of hybrid couplers and phase shifters by deriving a necessary and sufficient condition on the transformations these networks can compute, and we showed that the discrete Fourier transform (DFT) satisfies it. In this paper, we take a first step toward the formalization of analog computing with microwaves in Lean, a programming language and proof assistant that is increasingly adopted in mathematics. We formalize the considered components, their series and parallel connections, and the class of networks they can implement. Then, we formally prove the necessary and sufficient condition characterizing these networks, as well as the implementability of the DFT of any size power of two. All proofs are checked by the Lean kernel, and the code is openly available at: https://github.com/matteonerini/formalizing-analog-computing.