New scoring method improves style transfer evaluation aligned with human preference
Perceptually Aligned Evaluation of Style Transfer
Computer Vision and Pattern Recognition
Summary
Style transfer is a way to change a picture’s look but keep the original content, and judging how good these style changes are is tricky because people don’t always agree. The authors created ASTRA, which includes a special set of images, user votes about style transfer quality, and a smart scoring system called ASTRA-Score. This system learns from human opinions to automatically rate new style transfer images better than older methods. It helps compare style transfer results more reliably and matches what people actually prefer.
What this means in practice
- •For computer vision developers: Automatically evaluate and rank style transfer outputs to improve algorithm development efficiency.
- •For digital artists and content creators: Use better quality scores to select style transfer tools that produce visually preferred images for creative projects.
Authors
Yang Deng, Eleftherios Ioannou, David Mould, Steve Maddock, Paul L. Rosin, Yu-Kun Lai
Abstract
Style transfer lacks a reliable evaluation standard: ground truth is inherently ill-defined, and existing automatic metrics often fail to reflect human preference. This paper introduces ASTRA (Assessment of Style TRansfer Algorithms), an approach for automatic evaluation of style transfer algorithms; it contains two components, ASTRA-Data and ASTRA-Score. ASTRA-Data consists of a benchmark image set of content and style references, a collection of style transfer results generated on the benchmark set, and user study data capturing human judgements through a two-stage pairwise comparison protocol. From these annotations, we derive ranking-based ground truth for content preservation, style fidelity, and overall preference. Based on ASTRA-Data, we construct ASTRA-Score, a learnt evaluator that predicts preference-aligned scores from content-style-stylization image triplets, enabling automatic and scalable evaluation of new models applied to the benchmark set. Experimental results demonstrate that ASTRA-Score achieves substantially higher correlation with human rankings compared to prior metrics. Overall, ASTRA establishes a robust mechanism for standardised evaluation of style transfer methods.