ChatGPT Images 2.5 shows mixed results on image forgery tasks
From Advertised Improvements to Measured Capabilities: Evaluating ChatGPT Images 2.5 on Forgery Tasks
Computer Vision and Pattern Recognition
Summary
This paper tests whether ChatGPT Images 2.5 really improves at editing pictures to cheat or fake receipts and product labels. The authors found it makes fewer accidental changes around the edited parts but does not necessarily get the main edits more correct. Some product details looked clearer, but this might be because they were made bigger rather than more accurate. Also, systems that try to detect fakes still catch most of these altered images. The authors suggest that specific, clear testing is important to understand AI image editing capabilities and limits.
What this means in practice
- •For content authenticity teams: Use detailed forgery evaluation methods to better understand AI image editing limits when verifying altered images.
- •For digital forensics analysts: Improve detection protocols by considering how image generation models differ in leaving detectable traces in edits.
Authors
Ankit Raj, Yuxin Zhang, Kidus Zewde, Tommy Duong, Jiaqi Gan, Xingyu Shen, Yuchen Zhou, Huaiyu Guo, Siyu Zhang, Simiao Ren
Abstract
We evaluate whether the improvements advertised for ChatGPT Images 2.5 translate into better performance on forgery tasks with predetermined answers. We compare its Flare and Sunburst API models with GPT-Image-2 re-run in the same week, using receipt-field edits, repeated editing, product placement and fine-print rendering. After image registration, Flare and Sunburst show fewer OCR-detected changes to surrounding receipt text (31.7% and 31.2% versus 44.2% for both GPT-Image-2 baselines), mainly on CORD receipts, without a detectable improvement in target-field correctness. Flare retains fewer earlier edits on CORD receipts, while photo-edit sequences provide little separation between models. Product codes are more often legible with Images 2.5, alongside larger product placement; the analyses do not establish a fidelity gain independent of size. Fine-print improvements remain unresolved below the OCR reliability limit. Refusals are rare and localisation is weak in both generations. At a fixed detection threshold, Community Forensics flags 68.6% of controlled Images 2.5 images averaged across cells, versus 35.9% of self-reported images posted online. These results motivate task-specific evaluation of advertised capabilities and defences, with explicit limits on what automatic checks can establish.