Benchmark reveals gap between realistic images and exact identity matching
LEVI: Benchmarking Visual Identity Fidelity with Verifiable Low-Entropy Constructs
Computer Vision and Pattern Recognition
Summary
Generating images that look real is easier than making them match a specific identity exactly. The authors created a way to test if generated images faithfully follow the requested identity rules by using low-entropy identity concepts. They built a language and benchmark to check how well models stick to identity constraints, finding that even the best models fail to fully meet the requirements. This shows that creating images that both look good and are exactly right is still challenging.
What this means in practice
- •For image generation developers: Evaluate and improve generative models’ ability to produce images that strictly follow defined identity rules using the LEVI benchmark and visual grammar.
- •For digital content verification teams: Verify if automatically generated images meet exact identity specifications for reliable content creation or moderation.
Authors
Xiao Ma, Zhonghao Zhan, Hamed Haddadi, Qi Chen
Abstract
Modern generative models can produce visually plausible images while violating exact identity constraints. Evaluating such failures requires distinguishing violations of the requested identity from permissible variation in appearance. We introduce Low-Entropy Visual Identity (LEVI), which characterizes visual identities determined by compact facts and explicit rules. The low-entropy requirement concerns identity uncertainty, while appearance may remain diverse. We formulate identity fidelity in terms of the probability that a conditional image distribution assigns to the identities permitted by the prompt. Visual Identity Spaces (VIS) formalize this distinction by specifying admissible identity values and which different realizations express the same identity. Building on this formulation, we develop Verifiable Visual Grammar (VVG), a declarative language whose typed primitives and compositional constraints describe identity-bearing structures. Using this grammar, we build a benchmark for image generation. It assesses each output against the task's identity requirements, drawing evidence from the final pixels. The curated benchmark contains 1,375 tasks spanning 120 base identity spaces across six domains. Among ten evaluated model configurations, the highest whole-task pass rate is 58.69%. These results reveal a persistent gap between visually plausible images and faithful realization of their specified identities. Project page (forthcoming): https://maxiao0234.github.io/LEVI/.