Latent space evaluation cuts AI judgment time by six times

Judging in Latent Space: Efficient Generative Reward Modeling via Semantics-Preserving Compression

Computation and Language

Summary

Judging complex tasks often needs checking many things one by one, which can take a lot of computer time. The authors came up with LatentGRM, a method that turns these step-by-step checks into shorter, compact summaries in a special continuous form. This lets computers make decisions faster without writing out long explanations each time. Their tests show it keeps quality close to traditional methods but works much quicker. They also found that their summaries still keep important info for later review.

What this means in practice

  • For ai system engineers: Deploy evaluation modules that quickly judge AI outputs with less computational cost by using compressed latent summaries instead of verbose explanations.
  • For customer support automation teams: Speed up automated evaluation of chat responses for quality and relevance by replacing text-based scoring with compact latent judgment representations.

Authors

Mingqing Yuan, Xiaobo Liang, Junwei Yang, Ziwei Chen, Zeren Zhang, Hejin Wang, Yubin Wang, Juntao Li

Abstract

Reward modeling often requires jointly representing and reasoning over multiple evaluation criteria, yet verbalizing this process token by token can incur substantial inference cost. Recent work on latent reasoning suggests that continuous states may support this computation more compactly. We introduce LatentGRM, a latent evaluation framework built on semantic chunking, compression, and reconstruction. By using the structure of rubric-guided evaluations to guide compression, LatentGRM learns compact continuous trajectories that support autonomous pairwise judgments without generating textual assessments. A separate interpreter reconstructs evaluation text from these trajectories, providing an offline view of the information retained under compression. Under matched training data and backbones, LatentGRM achieves competitive aggregate preference accuracy relative to explicit Supervised Fine-Tuning (SFT) judges at both 4B and 8B scales. Across four benchmark domains, LatentGRM-8B compresses evaluation trajectories by 8.9--9.2x and reduces total judge inference time by 6.1--7.0x at vote@5. Controlled rubric interventions show that criterion-dependent preference information is carried through the latent sequence. Together, these results demonstrate that continuous latent evaluation can substantially reduce inference cost while preserving competitive judgment quality.