Papers for

multilingual nlp engineers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Small language models adapt well to Romanized Nepali with fine tuning

Adapting Small Language Models to Romanized Nepali: Tokenizer Fertility, Reference Likelihood, and Generation Metrics

Abstract: Romanized Nepali, Nepali written in Latin script, is common in digital communication but has no standard spelling and few resources, and it is unclear how reliably small open language models can be adapted to it. We adapt three 2-3B base models from different families, Llama-3.2-3B, Qwen2.5-3B, and Gemma-2-2B, with QLoRA on 9,000 instruction examples produced by deterministic transliteration of Nepali instruction data. Each model is trained with one shared recipe and three seeds and evaluated on 1,000 held-out examples with reference-overlap metrics, byte-normalized reference likelihood (bits per byte, BPB), and an output audit, against zero-shot baselines under two prompts. Fine-tuning raises chrF++ from 10-14 to 29.0-30.6 and lowers BPB from 2.23-2.41 to 0.78-0.84; most zero-shot outputs (54-92%) are English-like. Likelihood and overlap disagree on how far apart the models are. Llama-3.2-3B has the lowest BPB in every seed, 0.042 below Qwen2.5-3B (95% CI 0.039-0.046), yet the two cannot be separated on chrF++ (30.51 and 30.61) or BLEU, even with paired item-level tests. Gemma-2-2B, whose tokenizer splits Romanized Nepali words least (2.72 tokens per word), scores worst on every measure. Which model gains most over its zero-shot baseline depends on the zero-shot prompt. In a supplementary comparison with an identical tokenizer, Llama-3.1-8B lowers BPB by 0.053 (0.050-0.056) relative to Llama-3.2-3B, while chrF++ and BLEU rise by only 0.68 and 0.58 points. Lower fertility and higher reference likelihood therefore did not identify the model whose outputs were closest to the references.

Mon 9 NovComputation and LanguageArtificial Intelligence
The gist
Romanized Nepali is Nepali language written using Latin letters, which varies a lot in spelling and has few language tools. The researchers trained three small language models on examples transliterated from Nepali to Romanized Nepali to see how well they learn the language style. They found fine-tuning greatly improved the models' ability to generate text matching reference examples, but common metrics like overlap scores and likelihood disagreed on which model was best. Models with fewer tokens per word scored worse, showing tokenization matters for this task.
Open → 2611.12628v1

Large language models improve hard named entity recognition in Nepali tweets

Using an LLM to Tackle Difficult Cases in NER for Nepali Tweets

Abstract: Named entity recognition (NER) is challenging for social media data in a low-resource language. LLMs often lag behind pretrained language models (PLMs) in NER performance, but they show promise in handling cases that PLMs struggle with. Using a dataset of Nepali tweets, we split the test set into "easy" and "hard" cases using token probabilities from a fine-tuned pretrained language model (PLM). We then evaluate an LLM on the hard test cases using 16 different prompt configurations. We find that selecting examples based on semantic similarity and increasing the number of examples improves F1 score by almost 10 points from the baseline configuration. Although the LLM underperforms the PLM on the full and easy test sets, it surpasses the PLM on the hard test set, resulting in an improved score when the two models are combined.

Fri 6 NovComputation and Language
The gist
Named entity recognition (NER) tries to find names of people, places, and things in text. It is hard to do well when the text is messy social media posts and the language is less commonly studied, like Nepali. The authors split tweet examples into easy and hard groups using a special kind of language model. They then tested a large language model (LLM) on the hard examples and found it did better than regular models on those cases. By combining both models, they achieved better overall performance.
Open → 2611.08967v1

Tajik text embedder improves finding answers in documents significantly

Saidzoda-Tajik-E5-Large: Fine-Tuning a 560M Embedder for Tajik Retrieval

Abstract: Finding the passage that answers a question is the first step of most practical systems built on documents, and for Tajik we found no openly described model for it. We present Saidzoda-Tajik-E5-Large, a Tajik text embedder built from multilingual-e5-large-instruct (560M parameters). We asked our own Tajik language model to write questions for passages from Tajik laws, tax documents, Wikipedia and web text, and trained on 235,140 question and passage pairs for one epoch, with hard negatives chosen by another embedder and English and Russian pairs mixed in so the model would not forget those languages. On laws the model had never seen, the right passage is among the top ten results for 93.6% of questions, up from 80.2%; on tax documents, 94.5% against 82.6%. On Belebele, the one test we have that was written by people, the gain is smaller and not confirmed (92.2% to 95.2%, with touching intervals), and Russian and English do not change. On law and tax questions written in the same style as its training questions, it is tied with Giga-Embeddings-3B, a model five times its size; on Belebele the 3B and 8B models score slightly higher, and only after we fine-tuned them on our data are they clearly ahead. We also report what failed: training in bfloat16 broke the model, and replacing Tajik letters with similar Russian ones did not help. Our law, tax and Wikipedia test questions come from the same model that wrote the training questions, so we treat those scores as an upper limit until questions written by Tajik speakers are available.

Sat 31 OctComputation and Language
The gist
Finding the correct piece of text to answer questions is important, but there was no openly available tool for the Tajik language. The authors adapted a large multilingual language model to better find relevant passages in Tajik laws, tax documents, and Wikipedia by training it on specially created question-and-answer pairs. Their model finds the right passage in the top ten results more often than previous tools, especially on legal and tax texts. While the model performs very well on test questions generated by another model, its advantage is less clear on questions written by people, showing that more human data is needed.
Open → 2611.00592v1

Large language model judge choice matters more than prompt style

Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness

Abstract: LLM-as-a-Judge evaluation is increasingly applied to multilingual tasks, yet prompting strategies such as Chain-of-Thought and Role Conditioning are recommended based solely on English benchmarks. Whether their effectiveness transfers across judge models and languages remains unknown: prompt-sensitivity studies are largely English-centric, while existing multilingual judge studies fix the prompt template. In this work, we address this gap by measuring this interaction through a benchmark of 9 prompting strategies, 4 open-weight judges, 27 languages, and 4 tasks across 13 datasets, yielding over 1M pointwise scores on gold-reference outputs, validated against human judgments. We find that prompting strategies are not transferable: the choice of the judge model dominates the choice of prompting strategy, strategy effects reverse direction across judges, and stability degrades in low-resource languages, where these effects are concealed by aggregate reporting. These findings show that prompting effects in multilingual evaluation are conditioned on the judge, task, and language. Consequently, prompting strategies should be validated for the intended evaluation setting, and cross-study comparisons should account for both judge and prompt configuration.

Thu 22 OctComputation and LanguageArtificial IntelligenceMachine Learning
The gist
Evaluating language tasks with large language models (LLMs) involves giving the model specific instructions or prompts. This study finds that which LLM is used as a judge influences evaluation results more than the style of prompting itself. The effectiveness of prompting strategies varies a lot across different languages, models, and tasks. It shows that evaluation methods that work well for English may not transfer well to other languages and judges. The authors suggest testing prompting strategies specifically for each evaluation setup instead of assuming a one-size-fits-all approach.
Open → 2610.31172v1

Language model improvements give better confidence in factual claims

Truth in Contrast: Calibrated Factual Evidence from Language Model Likelihoods

Abstract: Large Language Models (LLMs) encode substantial factual knowledge. Yet, estimating confidence in factual claims reliably remains challenging. We study likelihood-based factual confidence from the complementary perspectives of discrimination and calibration across multilingual factual statements. Specifically, we instantiate contrastive generative likelihoods as Contrastive Factual Evidence (CFE), which compares the likelihood of the same claim under competing factual and non-factual conditions, and derive its probabilistic counterpart, the Contrastive Factual Posterior (CFP). We show analytically that CFP calibration depends on the differential error between the two surrogate conditionals, allowing errors shared across them to cancel and yield tighter calibration bounds. Empirically, CFP substantially improves calibration while maintaining factual discrimination comparable to its strongest competitor. Nonetheless, discrimination is uniformly weak across methods in lower-resource languages, which simultaneously undermines the utility of the measured calibration. To this end, we interpret a likelihood geometry, within which an evidence-bearing direction is induced, along with an orthogonal evidence-invariant direction. In our investigation, we selectively align target-language claims with stronger English evidence only along the factual-evidence direction, improving discrimination and open-ended factual generation without target-language labels.

Fri 16 OctComputation and LanguageArtificial Intelligence
The gist
It is hard for language models to know how sure they are when making factual statements. The authors created a way to compare how likely a language model thinks a fact is true versus false, called Contrastive Factual Evidence and its probabilistic version Contrastive Factual Posterior. Their method improves how well the model’s confidence matches reality, especially in high-resource languages. They also found a way to use strong English-language evidence to help models in other languages make better factual predictions without needing extra labels.
Open → 2610.23527v1

Neural pos tagging improves saraiki language processing accuracy

Neural POS Tagging for the Low-Resource Saraiki

Abstract: Part-of-speech tagging is a fundamental sequence labeling task and an important component for many downstream natural language processing applications. However, Saraiki, written from right to left in a Perso-Arabic script, remains an under-resourced Indo-Aryan language, despite being spoken as a mother tongue by 28,849,579 people (approximately 28.9 million) according to Pakistan 7th Population and Housing Census 2023. This paper presents a manually annotated Saraiki Universal POS tagging dataset developed from plain-text. We annotated the text with 17 POS categories. We evaluate three pretrained transformer encoders under the same fine-tuning setup: XLM-RoBERTa base, Urdu RoBERTa small, and multilingual DistilBERT. Experimental results over five random seeds show that XLM-RoBERTa base achieves the best performance, with 87.29% +/- 0.21% development accuracy and 88.60% +/- 0.21% test accuracy. Urdu RoBERTa small and multilingual DistilBERT also provide competitive results, suggesting that multilingual and related-script pretrained encoders are effective for Saraiki UPOS tagging. The dataset, evaluation setup, and baseline results introduced in this work provide a benchmark for future research on Saraiki NLP and low-resource sequence labeling.

Wed 14 OctComputation and Language
The gist
Saraiki is a language spoken by nearly 29 million people but lacks enough data for computers to understand it well. The authors created a new dataset where they labeled words with their parts of speech, like nouns or verbs, to help computers learn Saraiki better. They tested three advanced language models to see which tagged words the best and found the XLM-RoBERTa model worked most accurately. This work provides a useful starting point for others wanting to build language tools for Saraiki.
Open → 2610.20279v1

Calibrated metrics match judges in multilingual style text evaluation

Fantastic Metrics and How to Find Them: Meta-Evaluating Metrics for Multi-lingual Style-Personalized Generation

Abstract: Meta-evaluation of text generation metrics and LLM-judges is primarily conducted using metric-human correlations. However, prior work has shown that correlation is not an appropriate estimate of a metric's evaluation capability and can be unstable across evaluation settings, such as different datasets or languages. Therefore, we propose a regression-based meta-evaluation of metrics that learns on calibrated metric outputs to predict human ratings. We ground this measurement relative to a trivial constant-mean baseline and human agreements to provide a comparable measure of metric capability across evaluation settings. To test our meta-evaluation framework, we developed a large-scale multi-lingual style-personalized text generation (SPTG) benchmark, collecting multiple annotations across 10 topologically diverse languages and four evaluation dimensions, each with varying degrees of subjectivity. We meta-evaluated 32 metrics from three style evaluation paradigms (string-based, embedding-based, and LLM-judges) and their ensembles over these 40 evaluation settings. Our analysis revealed that even simple string- and embedding-based measurements can match strong LLM-judges when calibrated to human ratings. We also identified characteristics of human annotations that determine metric performance, as well as conditions under which a metric ensemble outperforms its individual constituents.

Tue 13 OctComputation and Language
The gist
Measuring how well computer programs judge text quality can be tricky, especially across different languages and styles. The authors created a way to better evaluate these judging tools by teaching them to predict human scores more accurately. They tested this method on a large, multilingual set of texts with many human ratings, showing that even simple measurements can perform as well as more complex AI judges when properly calibrated. They also learned what factors affect how well these judges work and when combining different judges is beneficial.
Open → 2610.19166v1

Language models contain neurons linked to storytelling quality

Language Models Have Interpretable Narrative Mechanisms

Abstract: We investigate whether language models contain internal components that contribute to representing narrativity, the degree to which a text exhibits narrative properties. Across GPT-2-Small, Gemma-2-2B, and Llama-3.1-8B, we identify attention heads, MLP neurons, and sparse autoencoder features whose activations correlate with narrativity. Ablating selected neuron and feature sets increases next-token prediction loss more for high-narrativity than for low-narrativity texts, and reduces the narrativity of open-ended generations. On downstream benchmarks, the largest effects occur in story completion and narrative question answering, although they vary across models and languages; results on confabulation and hallucination benchmarks are mixed. These findings provide causal evidence that selected sets of narrative-associated components contribute to model behaviour, while their internal organisation, relation to narrative theory, and broader functions remain open questions.

Fri 9 OctComputation and Language
The gist
The paper shows that some parts inside popular language models help the models understand and produce stories. The researchers found specific neurons and attention mechanisms that react more when the text is like a story. When these parts are blocked, the models' ability to predict story-like text and create narratives gets worse. This suggests certain internal components are important for storytelling in AI models, but how exactly they work and connect to narrative theory is still unknown.
Open → 2610.12989v1

Language specific thresholds improve text classification accuracy

One Threshold Does Not Fit All Languages: Language-Conditional Deferral for Reliable and Efficient Low-Resource Text Classification

Abstract: In the Global South, the lower-income countries of Africa, Asia, and Latin America where most of the world's languages are spoken, a deployed text classifier usually runs on ordinary CPUs, serves many languages with a single model, has few labeled examples in any of them, and relies on people to catch its mistakes. Such a system is only useful if it can promise how often it will be wrong: at most a fixed fraction of the labels it assigns on its own may be incorrect, and everything else must go to a person. Split conformal prediction delivers this promise through a single confidence threshold, normally estimated on validation data pooled across languages. We ask whether the promise reaches every language, and it does not. On MasakhaNEWS (16 African languages) and AfriSenti (12 languages plus two never seen in training), a pooled threshold meets the 90% target on average but covers Somali at 77.5%, Tigrinya at 83.7%, and the two unseen languages at 77.5% and 81.2%. Estimating one threshold per language brings every language to between 89.1% and 91.0% without retraining, and it shows how unequal the cost of the promise is: keeping it means sending 43% of Somali news and over 80% of Amharic and Xitsonga tweets to a person, against under 8% of Nigerian Pidgin news. One or two hundred labels per language are enough and the models train in minutes on one CPU core, so the fix is affordable: calibrate, report, and budget human review one language at a time.

Tue 29 SeptComputation and LanguageMachine Learning
The gist
Many language models are used to classify text in multiple low-resource languages but often fail to meet accuracy promises equally across all languages. The authors found that using one set confidence threshold for many languages underperforms some, especially those with less data. By setting thresholds separately for each language, the system becomes more reliable, though it means sending many more texts for human review in some languages. This approach requires only a small number of labeled examples per language and does not need retraining.
Open → 2609.37861v1

Emoji rating bias hides true differences among language models

Two Emojis of Difference: What Multilingual Affective Generation Benchmarks Actually Measure

Abstract: We audit a multilingual affective generation benchmark eight instruction-tuned LLMs producing emoji summaries for 17,100 Bangla, English and Hindi sentences, with 6,960 human judgements and find its headline conclusions to be artefacts of the measurement instrument rather than properties of the systems. Treating annotators as a random rather than a fixed factor, no system differs significantly from any other ($F(7,14)=0.59$, $p=0.76$), although the conventional analysis declares 19 of 28 pairwise differences significant. Annotator identity explains far more rating variance than system identity, and the winning system changes whenever any single annotator is removed. The ordering that does emerge tracks output length: mean emoji count explains 78.7\% of between-system variance, and a within-item length-matched comparison over 2,599 pairs reverses the leaderboard. We further show that cross-provider anisotropy differences vanish under mean-centring, that per-language token costs change sign with the normalising unit, and that multi-view row-wise splits inflate macro-F1 by $3.1$ points and change the top-ranked system. In place of preference scoring we propose **emoji-affect decodability**, a reference-based probe whose rankings are stable to $\pm0.003$ macro-F1 across seeds.

Thu 24 SeptComputation and Language
The gist
Measuring how well language models generate emotional emoji summaries in multiple languages is tricky because human ratings vary a lot depending on the reviewer. The authors show that differences between language models mostly disappear when factoring in individual annotator preferences. They also find the number of emojis used affects scores more than quality. Instead of current rating methods, the authors propose a more stable way to measure emoji-based emotion generation.
Open → 2609.29445v1

Structural similarity improves cross language scientific sentence classification

Improving Cross-Lingual Transfer for Sequential Sentence Classification in Research Papers via Structural Similarity

Abstract: Sequential sentence classification (SSC) is an essential task for structuring scientific publications, and extending SSC research to languages other than English can improve accessibility to scientific knowledge in multilingual digital libraries. Cross-lingual transfer is a promising approach to address the scarcity of training data in non-English languages. Prior work on other natural language processing tasks has shown the benefits of capturing linguistic similarity between source and target languages. However, SSC inherently depends on patterns at the discourse level, such as label sequences and positional regularities, which appear consistently across languages regardless of linguistic differences. To examine the factors that determine transfer success in SSC, we constructed a multilingual SSC dataset covering 13 non-English languages collected from five academic databases. Our cross-lingual transfer experiments, using both encoder-based and generative models, show that linguistic proximity has no consistent predictive power for transfer performance, whereas structural similarity in rhetorical organization shows a weak but consistent positive correlation across models. After controlling for source-language performance, the similarity of label distributions is the most consistent predictor. Building on this finding, we propose a set of three methods that explicitly leverage structural information using generative models. In the in-domain evaluation, the best combination reaches parity with the strongest encoder baselines, and in transfer to languages unseen during training, it outperforms the strongest encoder baseline.

Thu 17 SeptComputation and LanguageDigital Libraries
The gist
Scientific papers often organize sentences into sections like introduction, methods, and results, which follow similar patterns across languages. This paper shows that using the structural similarities of these patterns helps computers better understand and classify sentences in scientific papers written in different languages. The authors created a multilingual dataset and found that language closeness matters less than how sentences are organized. They developed methods that use this structure information and improved classification accuracy when transferring models to languages without training data.
Open → 2609.19650v1