Ising machine chip cuts energy for context selection in AI models

p-DCIM: A 28nm Probabilistic SRAM DCIM based Ising Machine for Near-Memory Context Selection in LLM-RAG Pipeline

Hardware Architecture

Summary

Large language models that use retrieved information need to pick the most useful pieces of context before answering questions. This picking process usually takes a lot of energy because it involves running the language model multiple times. The authors created a tiny computer chip that solves this selection problem more efficiently by using a special memory circuit that performs calculations directly inside its own cells. Their chip uses way less energy—over a million times less for selection—and speeds up the process without losing accuracy, which could help devices run smarter AI with less power.

What this means in practice

  • For ai hardware engineers: Integrate the low-power Ising solver macro into GPU packages to accelerate context selection for retrieval-augmented LLMs near memory.
  • For gpu system architects: Design hybrid memory stacks with embedded Ising solver logic to reduce energy consumption in large language model inference pipelines.

Authors

Madhav Vadlamani, Rahul Singh, Yuyao Kong, Haomei Liu, Zheng Zhang, Shimeng Yu

Abstract

We present a fabricated SRAM DCIM Ising machine that performs set-level context selection for large language model (LLM) with retrieval-augmented generation (RAG). In RAG, the evidence-selection stage is increasingly delegated to repeated language-model reranking, which adds substantial energy to the inference pipeline. We reformulate evidence selection as a quadratic unconstrained binary optimization (QUBO) problem and solve it on a dual-mode digital compute-in-memory (DCIM) macro: the same multiply-and-accumulate datapath computes the embedding-similarity scores that populate the QUBO coefficients and evaluates the per-spin Hamiltonian differences that drive the anneal. The annealing temperature is set by a digital plane-count control along with the SRAM pseudo-read random bit flips, giving graded cooling entirely from on-chip digital configuration. The macro is taped-out in a 28 nm process [0.16 mm2, 0.41 mW] and is proposed to be integrated as part of the logic base-die in hybrid high-bandwidth memory (HBM) and high-bandwidth flash (HBF) stacks in a GPU package, where only the selected passage identifiers return to the host. Replacing the language-model selection pass with the near-memory Ising solver yields a macro normalized selection-energy estimate more than six orders of magnitude lower than the LLM-based baseline and reduces the evaluated total energy per query by 67%, while maintaining comparable token-level F1 on the evaluated examples.