Multimodal models improve human image captions with better spatial accuracy
Human-Centric Image Captioning with Subject-Centered Spatial Understanding
Computer Vision and Pattern Recognition
Summary
Current AI models that describe images do well overall but often mix up details about people’s positions and body parts. This can cause mistakes like confusing left and right or mismatching body parts with objects. To measure and fix these mistakes, the authors created a special test called SPACE that focuses on understanding people’s precise positions. They also built a new training method that teaches AI to pay closer attention to these spatial details, making captions more accurate for human-centric images. Their tests show this approach helps AI create better, more exact descriptions of people in images.
multimodal large language modelsimage captioningspatial groundinghuman-centric scenariosegocentric lateralitybody-part localizationcaption rewritingpolicy optimizationbenchmark evaluationstructural hallucination
Authors
Bozhou Li, Jiahang Zhang, Yue Ding, Yushuo Guan, Bohan Zeng, Yiyan Ji, Xinlong Chen, Yang Shi, Yifan Dai, Yuran Wang, Chengzhuo Tong, Pengfei Wan, Yuanxing Zhang, Wentao Zhang
Abstract
While multimodal large language models (MLLMs) achieve remarkable performance on generic image captioning, they frequently suffer from structural hallucinations in human-centric scenarios. Accurately modeling human subjects is foundational for critical downstream applications, such as accurate avatar/video/image generation and fine-grained human action understanding. However, these tasks require highly precise subject-centered spatial grounding, such as distinguishing egocentric left/right laterality and maintaining correct anatomical-object bindings. Although catastrophic for structural integrity, these localized spatial inversions are often overshadowed by overall descriptive metrics in existing benchmarks. To systematically expose and quantify this bottleneck, we introduce SPACE (Subject-centric Poses, Appearance, and Characteristics Evaluation), a benchmark designed to evaluate subject-centered spatial understanding. On SPACE, we reveal that despite strong generic perception, current MLLMs consistently fail to ground descriptions in the subject's intrinsic frame of reference. To bridge this gap, we propose a specialized data construction and alignment pipeline. We first extract structured spatial hints from fine-grained body-part localization to guide a two-stage caption rewriting process, yielding highly spatially-faithful training data. Furthermore, we design a rubric-based reward for Group Relative Policy Optimization (GRPO) that explicitly penalizes structurally critical spatial errors during alignment. Extensive experiments on SPACE demonstrate our framework significantly improves human-centric caption quality, particularly in subject-centered spatial reasoning, achieving performance competitive with strong closed-source models. Our benchmark and code are available at https://github.com/JHang2020/SPACE-Eval.