Papers for

multimedia search teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

ERCLIP improves long caption image text retrieval with small data

ERCLIP: Small-Data Adaptation of CLIP for Free-Length Image-Text Retrieval

Abstract: CLIP's 77-token text encoder truncates detailed image descriptions. We present ERCLIP, a small-data adaptation of CLIP for long-caption image--text retrieval. During training, a long caption is split into sentences that the CLIP text encoder encodes independently; the sentence embeddings are fused by mean pooling or a gated residual Transformer scorer and receive their own contrastive loss. At inference ERCLIP is an unmodified CLIP dual encoder: test captions are still read through the 77-token window, and only training sees the full caption. We find that this fine-tuning erodes CLIP's zero-shot classification, most when the fusion output is not a combination of CLIP embeddings, and add a frozen-CLIP preservation loss on the image and short-caption embeddings. With 20k pairs and one epoch, ERCLIP improves long-caption R@1 over CLIP by 6.5--11.2 points, closing 62--77% of the gap to LongCLIP with 2% of its long captions. Preservation recovers 1.5--2.1 ImageNet zero-shot points (30--42% of the loss) and adds up to 2.4 points of COCO image-to-text R@1, at no inference cost.

Wed 21 OctInformation Retrieval
The gist
Standard CLIP models can only read short parts of image descriptions, missing details from longer captions. The authors introduce ERCLIP, a way to train CLIP using small datasets of long captions by splitting captions into sentences during training only. This method improves matching images with longer text descriptions without changing how the model works during use. They also add a technique to keep the original model's ability to recognize images without hurting this improvement.
Open → 2610.29939v1