StyleJudge rates artistic style transfer more like human judges

StyleJudge: A Human-Validated, In-Context-Learning Vision-Language Evaluator for Artistic Style Transfer

Computer Vision and Pattern Recognition

Summary

Automatically judging how well one image’s style has been changed to another has been hard because existing methods don’t consider examples of the intended style and don’t match human opinions well. The authors created StyleJudge, a small AI model that first looks at examples of the desired style changes before deciding how well another image matches that style. This model correlates better with human judgments than older methods and keeps most of the performance of a much larger AI used during training. They also built a new dataset and tools to help train and test these kinds of style evaluations.

What this means in practice

  • For digital artists: Evaluate and rank style transfer results to better match human preferences before publishing or showcasing.
  • For photo editing software developers: Incorporate an AI evaluator that judges style transfer quality aligned with human taste to improve automated style filters.$Commercial implications: Enables enhanced style filter features in photo apps marketed to consumers valuing artistic edits.

Authors

Lokesh, Pramukto Mandal, Somanshu Juneja, Jashnoor Singh, Amit Kumar, Pranav Adlinge, Richa Thakur, Kunal Swami, Aditya Nigam

Abstract

Automatic evaluation of artistic style transfer remains an open problem: existing metrics score an image pair in isolation, without seeing an example of the intended style transformation, and correlate poorly with human judgment. We introduce StyleJudge, a compact vision-language evaluator conditioned on a small set of reference source-to-stylized pairs before scoring a query pair, mirroring how a human rater calibrates to a style before judging a candidate. On a human-annotated benchmark of 878 rated pairs across 12 artistic styles (728 from 10 training/validation styles, 150 from 2 unseen styles), our efficient 0.8B-parameter StyleJudge reaches a Pearson correlation of 0.291 (Spearman 0.325) with human ratings on seen styles and 0.381 on unseen ones. On seen styles, it achieves the highest positive correlation among tested metrics: Gram-matrix style loss and CLIP similarity correlate weakly ($r=0.041$, $r=0.168$), while LPIPS (scored higher when closer to the source) correlates negatively ($r=-0.404$). We achieve this by distilling evaluation knowledge from a 72B-parameter vision-language teacher used only during training, which reaches $r=0.327$, confirming that most of its human alignment survives compact distillation. For training, we build a dataset of source-to-stylized pairs across the 12 styles via LoRA-adapted diffusion, densified with graded intermediate examples from a high-resolution image-morphing pipeline so the evaluator recognizes partial and complete stylization. On two held-out validation styles, StyleJudge reaches 28.2% exact-match agreement with its teacher's structured scores versus 1.1% for the untuned base model, with 67.8% shorter reasoning traces, and scores a pair in 39 s (median) using 2.5 GB of GPU memory on one NVIDIA T4. We release the human-rating benchmark, annotation guideline, SDXL-adapted FreeMorph code, and model weights.