Where Do Multilingual Vision-Language Encoders Fail on Low-Resource Languages?
2026-08-31 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied why vision-language models work worse with low-resource languages (like Swahili) compared to high-resource ones (like English). They found that the usual explanation—a simple language bias in the output—is not the main cause. Instead, the key problem is that the model’s internal language representations diverge deeper inside the network, especially in the end-of-sequence features. By adjusting these features to be more similar to English earlier in the model, they significantly improved performance for low-resource languages without hurting high-resource ones.
multilingual vision-language modelslow-resource languageslanguage alignmentend-of-sequence hidden stateencoder depthlinear language classifierINLP (Iterative Nullspace Projection)cross-lingual retrievalmodel representationlanguage divergence
Authors
Donghoon Han, SungHyun Moon, Aidyn Zhakatayev, Junghun Cha, SeungJae Lee
Abstract
Recent multilingual vision--language encoders cover hundreds of languages in a single model, yet on two state-of-the-art instances retrieval on low-resource languages (LRL; e.g. Swahili) trails high-resource ones (HRL; e.g. English) by $30^+$\,pp. We ask where in the trained encoder this gap is located. Prior modality-gap and cross-lingual subspace work suggests a linear language direction at the output crowds out alignment-relevant geometry. We falsify this: LEACE drives the linear language classifier from $>99\%$ to near chance and iterated INLP to $37$--$50\%$ while LRL retrieval moves within $\pm 1.5$\,pp and all tier means within $2.2$\,pp, tracking random controls. The linear bias is a \emph{symptom}, not the cause. Instead, the alignment-causal factor lies along the encoder's forward path: the EOS (end-of-sequence) hidden state's per-language trajectory diverges with depth. Substituting the EOS with its parallel English value three blocks before the projector lifts Swahili from $22.1\%$ to $69.1\%$ on one encoder (and reproduces on the other); three controls rule out pooled-position tautology and English specificity. A front-layer trunk that pulls each language's projection toward the parallel-content centroid corroborates the diagnosis at training time, recovering $+9.6$ / $+17.1$\,pp on LRL XM3600 retrieval (1{,}000-image subset), with consistent gains across three further benchmarks while preserving HRL performance.