Summary
Training models that connect images and text can involve uneven updates between the visual and language parts, which people often try to fix. The authors studied whether balancing the sizes of these updates always improves performance across different image types like underwater or aerial photos. They found that how much imbalance there is and whether fixing it helps changes depending on the type of images used. Simple steps like lowering learning rates or gradually freezing parts affect imbalance differently, and forcing exact balance does not always result in better captions. They also found a common setup for fine-tuning was accidentally leaving some visual parts unchanged, and correcting this helped performance on all tested datasets.
What this means in practice
- •For ai model trainers: Improve image captioning fine-tuning across specialties like underwater or radiological images by adjusting learning rates instead of enforcing equal gradient sizes.
- •For ml system developers: Avoid unintended parameter freezing in low-rank adaptation methods to enhance multi-domain vision-language model performance consistently.
Abstract
Imbalanced gradient magnitudes between the visual and language pathways of a vision-language model are often treated as a defect to be corrected. We test that premise for one family of correction, deliberately excluding adaptive, signal-driven schemes (e.g. BalGrad, OGM, PMR, CGGM), which are a mechanistically distinct class outside this study's scope. Measuring the language-to-visual gradient-norm ratio, reported in parameter-normalised form, across nine fine-tuning conditions, three seeds, and three captioning datasets spanning distinct physical domain shifts -- underwater, aerial, radiological -- we find imbalance magnitude varies markedly across domains with no predictable ordering. A plain learning-rate reduction cuts imbalance substantially and lands within a few BLEU points of the best method on every dataset. Staged freezing reduces the ratio on every domain yet never ranks first; a schedule-only control isolates freezing as the cause on one dataset but not the other two. Forcing the two gradient groups to equal magnitude drives per-parameter imbalance close to zero on every domain, yet is both the best result in the study and the worst placement among full fine-tuning methods, on different datasets, with identical settings. Reductions in gradient-norm ratio do not consistently predict captioning performance across domains, and how a given level of balance is reached matters as much as the level itself. As a secondary finding, a commonly reused LoRA configuration applied to BLIP silently adapts zero visual parameters; correcting it improves BLEU-4 on all three datasets.