Residual subspace experts improve image forgery detection generalization
RiSE: Residual Subspace Expert for Generalizable Text-Centric Image Forgery Localization
Computer Vision and Pattern Recognition
Summary
Detecting fake parts in images with text is important to keep financial and legal records trustworthy. Current methods often fail to spot new types of fakes because they overfit known examples and mix subtle fake traces. The authors propose a new model named RiSE that freezes main learned features and adapts only leftover parts, training separate experts for different forgery types. This approach improves detecting unseen forgeries and selects the best expert quickly. Testing shows that RiSE beats previous methods by a large margin while requiring less training.
What this means in practice
- •For digital forensics teams: Precisely locate text-related forgery traces in images to verify financial and legal documents against unexplored fake types.
- •For automated document verification providers: Improve detection of image manipulations in scanned documents by integrating specialized subspace experts focusing on various known forgery types.$Commercial implications: Enables more reliable forgery detection software for sectors like banking and law by enhancing cross-domain image verification accuracy.
Authors
Kahim Wong, Kemou Li, Yiming Chen, Haiwei Wu, Jiantao Zhou
Abstract
As AI-assisted image editing becomes increasingly prevalent, Text-Centric Image Forgery Localization (TFL) is essential for protecting trust in financial and legal records. However, we observe that existing TFL detectors predominantly rely on Full-Parameter Fine-Tuning (FPFT) and fail to generalize to unseen forgery types because FPFT can induce low-rank feature collapse and overfit training artifacts. Meanwhile, prior detectors capture diverse forgery traces with a single unified representation, making subtle traces entangled and difficult to learn. To address these challenges, we propose RiSE, a Residual Subspace Expert model that preserves pre-trained priors by freezing principal SVD components and adapting only the residual subspaces with a ViT-only segmentor to avoid feature collapse. RiSE introduces forgery-aware residual experts, where each expert is trained independently on a training subset with a particular forgery type, and we show that cross-domain localization performance improves consistently as more experts are added. At inference, we introduce Latent Perturbation Confidence (LPC), which selects the most confident expert by analytically computing the latent-perturbed output. LPC enables efficient confidence estimation with a single forward pass, avoiding repeated forward passes on perturbed samples and improving expert selection. By training on synthetic forgery data, RiSE with 1 expert outperforms prior methods on real-world forgery images by 20.4% while reducing training steps by 10 times. Scaling the number of experts to 9 yields a 35.8% gain. RiSE remains effective with DCT feature, even when the majority of the parameters is frozen with RGB-only pretraining. Code: https://github.com/KahimWong/RiSE.