Two-layer evaluation improves assessment of text-to-SVG faithfulness

SVGFaith: A Two-Layer Meta-Evaluation of Faithfulness Metrics for Text-to-SVG Generation

Computer Vision and Pattern RecognitionComputation and Language

Summary

It is hard to check if pictures made from text match the text well, especially when pictures are complex SVG images. The authors made SVGFaith, a new way to compare and score how well captions and images match by using carefully made test pairs and human judgments. They created a diagnostic dataset that tests specific abilities like recognizing objects and spatial relations, showing existing scores work well on objects but struggle with details. They also built SVGFaith-Score, which breaks down prompts into small parts and judges each part, giving a clearer and more accurate faithfulness score.

What this means in practice

  • For graphic design teams: Evaluate and improve automatic SVG graphics generated from text prompts with more precise feedback on visual details.
  • For software developers in ai tools: Incorporate SVGFaith-Score to better measure how closely AI-generated SVG images adhere to their input descriptions during development.

Authors

Guotao Liang, Haitao Zhou, Biao Liu, Huanxuan Liao, Lingfeng Qiao, Qian-Wen Zhang, Qian Yu, Di Yin, Xing Sun

Abstract

Text-to-SVG systems are commonly evaluated by applying raster-image metrics to rasterised outputs, yet whether these scores capture fine-grained prompt faithfulness remains unclear. We introduce SVGFaith, a two-layer meta-evaluation framework in which the more faithful candidate of each pair is determined by construction or by human judgment. SVGFaith-Diagnostic contains 1,842 controlled contrastive pairs across eight capability axes, split evenly between two directions: two renderings compared against one prompt, and two descriptions compared against one rendering. SVGFaith-Arena contains 1,327 human-annotated comparisons of real generator outputs. On SVGFaith-Diagnostic, CLIP and three other image--text embedding metrics reach at least 0.789 pairwise accuracy on object identity but at most 0.672 on spatial relations and attribute binding, where chance is 0.5. We further introduce SVGFaith-Score, an interpretable pointwise metric. A domain-tuned decomposer turns each prompt into a dependency graph of atomic visual requirements, a visual question answering model scores whether each requirement is satisfied, and a requirement's score is discounted when its prerequisites score low. In addition to the overall score, the per-requirement scores show which requirements are judged unsatisfied and which inherit discounts from their prerequisites. SVGFaith-Score significantly outperforms all nine baselines, including a Qwen3.5-122B-A10B judge, on Arena and in both Diagnostic directions. Code and data: https://github.com/Yukinonooo/SVGFaith.