Semantic knowledge adds little improvement for rare image classes
Where Does the Semantic Gain Come From? A Reproduction and Extension of Semantic Knowledge-driven Contrastive Learning for Long-Tailed Recognition
Computer Vision and Pattern RecognitionMachine Learning
Summary
The paper looks at a method that uses language information to help computers recognize less common image categories better. The authors of this work found that adding this language-based knowledge to an existing image learning method only improves accuracy a little. Much of the improvement claimed in past work comes from longer training or other factors, not the language knowledge itself. They also tested tweaks to the approach but saw only minor benefits, showing that the language-based help may not add much on certain datasets.
What this means in practice
- •For computer vision engineers: Optimize training approaches for image classification with rare categories by focusing on data augmentation and training length instead of semantic language knowledge.
- •For machine learning platform developers: Refine benchmarking and model evaluation by including semantic graph effects and training duration analysis when comparing long-tailed image classifiers.
Authors
Sushrut Ghimire
Abstract
Semantic Knowledge-driven Contrastive Learning (SKCL) uses a language model to decide which classes are related, and pulls each image towards the prototypes of its semantic neighbours. On CIFAR-100-LT (beta = 100) it reports 54.02% top-1 accuracy, 2.01 points above Balanced Contrastive Learning (BCL), the method it builds on. The code and the class descriptions are not public. I reimplement SKCL, BCL and ConCutMix in one framework, check it against the public baseline code, and run every configuration with three seeds. The two baselines reproduce within 1.5 points, but SKCL built on BCL, as the paper describes it, ends up 0.56 points below BCL. To find out why, I add SKCL to the authors' own ConCutMix code. Trained for the paper's 300 epochs, it reaches 53.69, only 0.33 below the published number. At the same budget, however, the semantic graph adds just 0.23 points over ConCutMix, while training ConCutMix for 100 more epochs adds 1.04. Together with ConCutMix's published lead over BCL (1.15), this explains the claimed gain. A BCL model that never sees the graph already shares 41.2% of the graph's top-2 neighbours with its own most-confused classes (2.0% by chance), which shows why the graph adds so little on these benchmarks. I also test several changes to SKCL. Combining it with the CutMix branch improves it by 1.06 points, and an adaptive version of the graph improves it slightly (+0.34 and +0.28 in two codebases), although these gains are within seed noise.