Multi dimensional scale improves fast personalized subjective judgments

Multi-Dimensional Comparative Scale Construction for Efficient Personalized Subjective Judgment in High-Traffic Applications

Artificial Intelligence

Summary

Measuring how people feel about things is hard because everyone experiences intensity differently. The authors created a method where people compare pairs of items along several dimensions to build a personalized scale showing relative strengths and preferences. They made the method efficient so it can handle lots of judgments without slowing down, and built a fast model to score new cases quickly. Tests show their approach works well and is faster than some existing large language models.

What this means in practice

  • For content moderation teams: Provide quick, personalized judgments of subjective content intensity across multiple criteria to handle high volumes efficiently.
  • For recommender system engineers: Build multi dimensional preference scales from sparse comparisons to refine personalized recommendations at scale.

Authors

Xianglong Shi, Shifeng Liu, Sirui Zhao, Shengming Yuan, Enhong Chen

Abstract

Subjective judgments are central to many high-traffic applications, but subjective intensity is difficult to quantify and perceptions vary substantially across individuals. To address these challenges, we propose a pairwise comparative framework for multi-dimensional scale construction. By comparing case-person pairs along case and profile dimensions, the framework constructs relative scales that capture both fine-grained intensity and individual variation. To support practical high-traffic deployment, we optimize both offline scale construction and online inference. For scale construction, we combine sparse Elo comparisons with multi-judge voting, cutting the comparison cost from $O(N^2)$ to $O(NK)$ for $N$ objects and a budget of $K$ opponents per object, while limiting reliance on any single judge. For inference, we propose SubJudge, a System One model for personalized scoring with Batchwise Preference Optimization (BPO). Using Bradley-Terry comparisons, BPO trains the model to learn relative orderings, and SubJudge reads a continuous score from digit-token probabilities at the first response position, requiring only one forward pass per criterion and reducing the inference complexity to $O(1)$. Experiments on PluriHarms and iNews show that our 9B models match or surpass the evaluated frontier LLMs on multiple metrics. On the H100 GPU, SubJudge achieves an approximately $1.29\times$ to $261\times$ speedup in mean inference latency over Qwen3.5-9B with different thinking budgets. The code is available at https://github.com/Longchentong/SubJudge.