Span level supervision improves multilingual token and sentence representations
Improving Cross-Lingual Token Representations by Adding a Pinch of SALT
Computation and LanguageArtificial Intelligence
Summary
Many language processing tools are trained to understand whole sentences in many languages but are later used to analyze smaller parts like individual words or phrases, which can be tricky. The authors show a simple way called SALT that adds extra training focused on chunks of text within sentences to make the tools better at these smaller pieces. This leads to improved performance not only for word-level tasks but also for whole sentence tasks across multiple languages. Their approach works better than other fine-tuning methods on different language benchmarks.
What this means in practice
- •For machine translation teams: Improve word-level alignment and detection of translation errors using SALT-enhanced multilingual encoders.
- •For multilingual chatbot developers: Enhance token-level understanding and response accuracy in chatbots supporting multiple languages by applying SALT.
Authors
Guillem Ramírez
Abstract
Cross-lingual sentence encoders enable scalable transfer across hundreds of languages, powering applications such as translation mining and zero-shot learning in low-resource settings. Although trained for sentence-level alignment, they are increasingly also applied to token-level tasks such as hallucination detection and sequence tagging, exposing a mismatch between training and usage. We propose SALT, a lightweight post-training method that improves token representations by injecting span-level supervision into existing sentence encoders. Across five multilingual token-level benchmarks, SALT achieves the best overall results on four of them, outperforming alternative fine-tuning strategies and competitive encoders. It also improves sentence-level performance on cross-lingual retrieval and classification tasks. These results demonstrate that span-level supervision is an effective signal for improving both token and sentence representations.