TransBERT improves specialized French life sciences NLP with synthetic text
TransBERT: A Framework for Synthetic Translation in Domain-Specific Language Modeling
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
There is often not enough non-English data in specialized areas like life sciences to build good language understanding tools. This paper presents TransBERT, which trains language models using only automatically translated text. The authors create a large French life sciences text collection by translating from English and show that this synthetic data can produce strong results on different language tasks. They provide open-source tools and models to help build similar resources for other languages and domains. This approach makes it easier to develop language technology in low-data settings.
What this means in practice
- •For medical data teams: Create French-language NLP tools for life sciences by training models on synthetic translated data when native text is scarce.
- •For multilingual chatbot developers: Build domain-specific dialogue systems in under-resourced languages using synthetically translated corpora as training data.
Authors
Julien Knafou, Luc Mottin, Anaïs Mottaz, Alexandre Flament, Patrick Ruch
Abstract
The scarcity of non-English language data in specialized domains significantly limits the development of effective Natural Language Processing (NLP) tools. We present TransBERT, a novel framework for pre-training language models using exclusively synthetically translated text, and introduce TransCorpus, a scalable translation toolkit. Focusing on the life sciences domain in French, our approach demonstrates that state-of-the-art performance on various downstream tasks can be achieved solely by leveraging synthetically translated data. We release the TransCorpus toolkit, the TransCorpus-bio-fr corpus (36.4GB of French life sciences text), TransBERT-bio-fr, its associated pre-trained language model and reproducible code for both pre-training and fine-tuning. Our results highlight the viability of synthetic translation in a high-resource translation direction for building high-quality NLP resources in low-resource language/domain pairs.