Benchmark for image harmonization mixes distinct test image groups
Only Means Only: what a harmonization benchmark score averages over
Computer Vision and Pattern Recognition
Summary
Image harmonization aims to make objects pasted into photos look natural by adjusting color and lighting. The authors discovered that one popular benchmark test includes many images derived from the same original photos, even though it was supposed to keep training and test photos separate. Because of this overlap, some methods appear to perform much better on this subset, skewing the overall average benchmark score. To address this, the authors provide a new test set that clearly separates training and test images to give a more accurate measure of method performance.
What this means in practice
- •For computer vision developers: Evaluate image harmonization methods more reliably by using a test set that avoids overlap with training images.
- •For photo editing software teams: Improve software evaluation by integrating benchmark tests that fairly assess harmonization quality without data leakage.
Authors
Imad Buljić, Delaida Muminović
Abstract
Image harmonization methods adjust the colour and lighting of an object pasted into a photograph so that it looks native to its new scene, and they are ranked by average error on one benchmark whose test images are real photographs with one object recoloured. One photograph can yield several such test images, and the benchmark promises that no photograph appears in both the training and the test split. We find that 1,399 photographs do, and the composites built on them are a quarter of the test set. Duplicate checks over file names or bytes miss this, because every composite is a different file. On that quarter the three methods we score end to end read two to five times lower error than on the rest, but most of the gap is composition rather than the sharing: the shared composites come almost entirely from one of the benchmark's four source collections, and inside that collection the gap falls to about one and a half times. An official score is therefore an average over two populations, mixed in a proportion the benchmark does not report. We release a test list of 5,581 composites whose construction identities do not appear in training, with four confirmed near-duplicate pairs disclosed rather than removed, adoptable without retraining at the price of one rescore per checkpoint. We leave open whether the sharing itself changes what a model learns.