ERCLIP improves long caption image text retrieval with small data
ERCLIP: Small-Data Adaptation of CLIP for Free-Length Image-Text Retrieval
Information Retrieval
Summary
Standard CLIP models can only read short parts of image descriptions, missing details from longer captions. The authors introduce ERCLIP, a way to train CLIP using small datasets of long captions by splitting captions into sentences during training only. This method improves matching images with longer text descriptions without changing how the model works during use. They also add a technique to keep the original model's ability to recognize images without hurting this improvement.
What this means in practice
- •For multimedia search teams: Enable better retrieval of images using long descriptive captions with limited training data, improving search accuracy for detailed content.
- •For digital content managers: Improve matching between detailed user-generated image descriptions and images in databases without increasing inference costs or altering existing systems.
Authors
Hoang-Bao Le, Allie Tran, Binh T. Nguyen, Liting Zhou, Cathal Gurrin
Abstract
CLIP's 77-token text encoder truncates detailed image descriptions. We present ERCLIP, a small-data adaptation of CLIP for long-caption image--text retrieval. During training, a long caption is split into sentences that the CLIP text encoder encodes independently; the sentence embeddings are fused by mean pooling or a gated residual Transformer scorer and receive their own contrastive loss. At inference ERCLIP is an unmodified CLIP dual encoder: test captions are still read through the 77-token window, and only training sees the full caption. We find that this fine-tuning erodes CLIP's zero-shot classification, most when the fusion output is not a combination of CLIP embeddings, and add a frozen-CLIP preservation loss on the image and short-caption embeddings. With 20k pairs and one epoch, ERCLIP improves long-caption R@1 over CLIP by 6.5--11.2 points, closing 62--77% of the gap to LongCLIP with 2% of its long captions. Preservation recovers 1.5--2.1 ImageNet zero-shot points (30--42% of the loss) and adds up to 2.4 points of COCO image-to-text R@1, at no inference cost.