Processing-in-DRAM reduces carbon footprint in transformer inference servers
Sustainable Processing-in-DRAM Integration and Replacement for Transformer Inference Servers
Hardware ArchitectureEmerging Technologies
Summary
Transformer models used in AI rely on powerful computers that use a lot of energy. The authors studied a special kind of computer part called Processing-in-DRAM (PID) that can do computing right where the data is stored, which saves energy and cuts carbon emissions. They created a way to measure the total carbon impact of these parts, including manufacturing and running costs. Their findings show that using PID can reduce overall carbon emissions in certain AI tasks, but this depends on the type of GPU it is paired with and how long the parts are used before replacement.
What this means in practice
- •For data center operators: Optimize deployment and replacement schedules of PID accelerators in AI servers to minimize carbon emissions over their lifespan.
- •For hardware architects: Design AI inference hardware integrating PID modules with GPUs to balance performance and carbon footprint effectively.
Authors
Bipin Thapa Magar, Ishan Thakkar
Abstract
Processing-in-DRAM (PID) accelerators are emerging as a sustainable alternative to GPUs for transformer model inference. They bring compute to data, mitigate the memory bandwidth limitation, and reduce the overall energy footprint. Yet, carbon analysis of PID accelerators is covered for only a limited set of hardware and workloads. In this work, we present a carbon modeling framework for PID-enabled inference servers that captures embodied carbon along with idle and active operational carbon. Our evaluation spans three GPU baselines, six PID architectures, and eight transformer workloads. Additionally, we propose an aging-aware model that finds the carbon-optimal replacement period for PID accelerators over a 10-year horizon. Across evaluated configurations, our analysis shows that including PIDs in AI inference servers increases the total embodied carbon by at most 2.5\%, while the total carbon is reduced by up to 1.14$\times$ for encoder and 1.04$\times$ for decoder transformer workloads. The results also depend on GPU selection for pairing, where introducing PID modules in high-end HGX-H100 servers incurs 5.4--5.9$\times$ higher total carbon compared to A100 and H100 servers, due to their larger idle and embodied carbon. Introducing PID modules in higher-end GPU servers, compared to the A100-only baseline, increases (decreases) the total carbon-delay product (CDP) for encoder workloads (decoder workloads).