ChatGPT Images 2.5 shows mixed results on image forgery tasks

From Advertised Improvements to Measured Capabilities: Evaluating ChatGPT Images 2.5 on Forgery Tasks

Computer Vision and Pattern Recognition

Summary

This paper tests whether ChatGPT Images 2.5 really improves at editing pictures to cheat or fake receipts and product labels. The authors found it makes fewer accidental changes around the edited parts but does not necessarily get the main edits more correct. Some product details looked clearer, but this might be because they were made bigger rather than more accurate. Also, systems that try to detect fakes still catch most of these altered images. The authors suggest that specific, clear testing is important to understand AI image editing capabilities and limits.

What this means in practice

Authors

Ankit Raj, Yuxin Zhang, Kidus Zewde, Tommy Duong, Jiaqi Gan, Xingyu Shen, Yuchen Zhou, Huaiyu Guo, Siyu Zhang, Simiao Ren

Abstract

We evaluate whether the improvements advertised for ChatGPT Images 2.5 translate into better performance on forgery tasks with predetermined answers. We compare its Flare and Sunburst API models with GPT-Image-2 re-run in the same week, using receipt-field edits, repeated editing, product placement and fine-print rendering. After image registration, Flare and Sunburst show fewer OCR-detected changes to surrounding receipt text (31.7% and 31.2% versus 44.2% for both GPT-Image-2 baselines), mainly on CORD receipts, without a detectable improvement in target-field correctness. Flare retains fewer earlier edits on CORD receipts, while photo-edit sequences provide little separation between models. Product codes are more often legible with Images 2.5, alongside larger product placement; the analyses do not establish a fidelity gain independent of size. Fine-print improvements remain unresolved below the OCR reliability limit. Refusals are rare and localisation is weak in both generations. At a fixed detection threshold, Community Forensics flags 68.6% of controlled Images 2.5 images averaged across cells, versus 35.9% of self-reported images posted online. These results motivate task-specific evaluation of advertised capabilities and defences, with explicit limits on what automatic checks can establish.