Papers for

localization teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Language models wrongly prefer english grammar in other languages

I play of the cello: Evaluating Constructional Bleeding in Multilingual LLMs with CALQUE

Abstract: Languages use different morphosyntactic strategies to express the same meaning. Speakers of multiple languages are able to apply strategies from one language to another, either by mistake or intentionally, to produce a literal translation, or calque. Considering that large language models (LLMs) have been shown to generalize crosslingually, despite being predominantly trained on English, we investigate whether they overgeneralize constructional strategies from English to other languages. To test this, we create a dataset of 5583 sentence pairs across seven languages, matching literal translations of strategies from English into other languages, and literal translations of the other languages' strategies into English. We then quantify constructional bleeding by measuring how often strategies from English are assigned a higher probability than native strategies. We find that models often fail to assign higher probabilities to native strategies and tend to prefer English-derived strategies even when evaluating sentences in other languages. Our findings highlight the limitations of the current English-dominant training paradigm.

Fri 6 NovComputation and Language
The gist
Languages express the same ideas in different ways, and people who speak multiple languages sometimes translate phrases literally from one language to another, creating calques. The authors studied whether large language models, mostly trained on English, incorrectly apply English grammatical patterns when working in other languages. They created a dataset comparing literal translations between English and six other languages and found that models often favor English-style constructions even in non-English languages. This shows a limitation in current language models caused by their heavy English training.
Open → 2611.10266v1

Fine-tuning Meta models improves German English translation quality

Dataset Collection and Fine-Tuning of NLLB for German--English Translation

Abstract: This paper describes the curation of a high-quality parallel corpus from OPUS Books and the fine-tuning of Meta's NLLB-200 models for German--English translation. We detail a dataset extraction pipeline that filters 45k raw OPUS sentence pairs to 25k high-quality examples through encoding validation, length filtering, language identification, deduplication, and quality scoring. We present a reproducible fine-tuning methodology optimized for consumer GPUs using mixed precision, gradient accumulation, and gradient checkpointing. Our fine-tuned NLLB-600M model achieves statistically significant improvements (+3.4 BLEU, +2.6 chrF++, -3.2 TER) over the base model on literary and technical German-English translation. We demonstrate practical deployment by integrating the fine-tuned model into LinguaHaru, an existing open-source document translation system, achieving format-preserving translation across PDF and Office documents. All dataset preparation scripts, training code, and evaluation tools are provided for reproducibility. The fine-tuned model is available at HuggingFace.

Thu 5 NovComputation and Language
The gist
Translating between German and English can be tricky, especially for books and technical documents. The authors gathered a carefully cleaned set of German-English sentence pairs from a public collection and used it to improve an existing Meta AI translation model. Their updated model translates texts better in both style and accuracy. They also integrated this improved tool into an open translation system that works well with popular document formats like PDFs and Office files. All the steps and code are shared to let others reproduce and use their work.
Open → 2611.07797v1

Machine translation evaluation improved with expert scoring method

Refined Assessment for Translation Evaluation: Rethinking Machine Translation Evaluation in the Era of Human-Level Systems

Abstract: As machine translation systems approach human-level quality, traditional evaluation methodologies struggle to detect subtle translation errors. We critically examine limitations in current gold-standard approaches (MQM and ESA), including inconsistencies from variable annotator expertise, excessive categorization complexity, coarse severity granularity, accuracy bias over fluency, and time constraints. To address this issue, we introduce a high-quality dataset consisting of human evaluations for English-Russian translations from WMT24, created by professional linguists. We show that expert assessments without time pressure yield substantially different results from standard evaluations. To enable consistent and rich annotation by these experts, we developed the RATE (Refined Assessment for Translation Evaluation) protocol. RATE provides a streamlined error taxonomy, expanded severity ratings, and multidimensional scoring balancing accuracy and fluency, facilitating deeper analysis of MT outputs. Our analysis, powered by this expert dataset, reveals that state-of-the-art MT systems may have surpassed human translations in accuracy while still lagging in fluency - a critical distinction obscured by existing accuracy-biased metrics. Our findings highlight that advancing MT evaluation requires not only better protocols but crucially, high-quality annotations from skilled linguists.

Sat 24 OctComputation and Language
The gist
Machine translation systems are getting so good that it’s hard to tell their mistakes using old evaluation methods. The authors found that usual tests miss small errors because of inconsistent judgments and limited focus on accuracy over the flow of language. They created a new dataset scored by professional linguists who took their time, along with a simpler and clearer scoring system called RATE. Their results show that some machines now translate more accurately than humans but still don’t sound as natural, a difference older tests couldn’t reveal.
Open → 2610.33749v1

Controllable machine translation improves low resource language accuracy

GLOSS-MT: Controllable Low-Resource Machine Translation with an Editable Dictionary

Abstract: Adapting a pretrained machine translation (MT) model to new terminology usually requires new parallel data and another round of fine-tuning, which is costly for low-resource languages. We ask whether controllability can be added to MT, and the fine-tuning burden reduced, without degrading the pretrained model. GLOSS-MT looks up the source's noun phrases in an offline English-Twi dictionary, replaces every phrase found by its Twi translation in place, and fine-tunes a translation model once to translate these mixed inputs. We apply the same data and pipeline to NLLB-200 (600M, 1.3B, 3.3B), OPUS-MT en-tw and TranslateGemma 4B. On a document-level English-to-Asante Twi benchmark (84 documents, four domains), every fine-tune scores above its pretrained base in chrF2 (+2.85 to +5.48), and the three NLLB fine-tunes are the best open systems (50.12-50.80). A direct controllability test on human-translated Bible passages shows that the three fine-tunes tested write 79-97% of the dictionary terms, and that replacing every dictionary entry by a different term changes the output accordingly (70-97% of the new terms appear) with no retraining. We release the models, data and code.

Sat 17 OctComputation and LanguageArtificial Intelligence
The gist
Translating languages that have little data available is hard, especially when you want to use specific words correctly. This paper shows a way to use a dictionary to replace important words in a sentence before translating, so the translation keeps those words right without much extra work. The authors tested this on English to Asante Twi translation and found it worked better than before, accurately using the dictionary words. Their method lets people change dictionary entries to control translations without retraining the model.
Open → 2610.24730v1

Chain of thought faithfulness increases in low resource languages for knowledge tasks

The Reliance Hypothesis: Chain-of-Thought Faithfulness Rises as Language Resource Level Falls

Abstract: Chain-of-Thought (CoT) faithfulness matters for monitoring, explaining and safely deploying large language models, yet it has been studied mostly in English. Languages differ in level, the amount of text available to learn from, so models know less in lower-resource languages, which may change how much they rely on their reasoning. To our knowledge, we are the first to relate CoT faithfulness to both resource level and task type (knowledge- or reasoning-intensive). On parallel multiple-choice questions from Multilingual Massive Multitask Language Understanding (MMMLU) in eight languages, faithfulness rises as resource level falls on knowledge-intensive tasks but not on reasoning-intensive ones, a gap that points the same way across four models and several faithfulness measures. We propose, and find support for, the Reliance Hypothesis: models rely more on their reasoning when they cannot recall the answer, as in lower-resource languages on knowledge-intensive tasks. Model confidence partly explains the effect of resource level, but not the knowledge-reasoning gap. Languages in which models are less confident rely more on reasoning, yet less confident answers within a language do not, a Simpson's paradox. In lower-resource languages, models rely on their reasoning for more answers, not more strongly for each. CoT monitoring may therefore be most informative in lower-resource languages, where it is least often validated.

Fri 16 OctComputation and LanguageArtificial Intelligence
The gist
Large language models explain their answers by showing their reasoning, but this is mostly studied in English. The authors found that in languages with less training data, models rely more on reasoning when they don’t know the answer directly, especially for questions needing factual knowledge. This pattern doesn’t hold for questions requiring deeper reasoning. They call this the Reliance Hypothesis and show it across different models and languages.
Open → 2610.22841v1

Back-translation improves low-resource language translation with large models

Back-Translation Still Matters: Leveraging Monolingual Data for Low-Resource Machine Translation with LLMs

Abstract: Pre-trained Large Language Model (LLMs) are effective on many tasks in zero-shot and few-shot settings. For machine translation of low-resource languages, the obvious options are few-shot prompting and fine-tuning on a small parallel set. We ask: what if moderate amounts of low-resource monolingual data are also available? We argue that, rather than continued pretraining, a more effective use of this data is the pre-LLM era technique of back-translation (BT). The current fine-tuned LLM translates monolingual sentences in each language, and the resulting pairs train it in the reverse direction. Updates on the parallel set are interleaved for stable learning. With 1K parallel pairs on Odia and Swahili, BT beats continued pretraining into the low-resource language by 7.7 to 16.4 XCOMET-XXL points in domain. With beam-decoded intermediate translations, it also beats a model with 10 time more parallel data. In-context examples retrieved from the parallel data further improve BT. We also analyze the source, quality and amount of monolingual text. Across five low-resource languages, with Hindi as a high-resource control, and two 3B backbones, BT improves translation into the low-resource language by 16 to 173% XCOMET-XXL over parallel-only fine-tuning. Blind human evaluation on Odia and Sanskrit agrees with the automatic metrics.

Tue 13 OctComputation and LanguageArtificial Intelligence
The gist
Translating languages with very little existing data is hard for computers. This paper shows that using a method called back-translation—translating sentences back and forth between languages—helps large language models learn better translations when only small amounts of paired examples exist. The authors found that this old technique still outperforms simply fine-tuning the models with more monolingual data. They tested this on several rare languages and confirmed the improvements with both automatic scores and human reviewers.
Open → 2610.18818v1

Patmt2 improves multilingual patent translation with two-stage adaptation

PATMT2: Two-stage domain adaptation for multilingual patent translation

Abstract: We present PatMT2 (Patent Machine Translation, v2), a two-stage LoRA pipeline that adapts the Hy-MT2-7B dedicated translation model to four-language (Chinese/English/Japanese/Korean) patent translation. On six translation directions, the Stage-2 checkpoint reaches a macro-average BLEU of 43.39 (BF16) to 43.57 (AWQ INT4), improving over the Hy-MT2-7B base by up to +8.8 BLEU and also exceeding Qwen3-32B and several external systems on the same average. We then measure the cost of adaptation: probing 38 languages absent from the fine-tuning data, the model still produces correct, fluent output on 34, while four low-resource languages collapse into distinct, reproducible failure modes. We treat these as diagnostic cases rather than retention rates, and report their mechanisms without attributing them to a single cause. Paired bootstrap tests confirm the translation gains over all baselines (p < 0.001).

Mon 12 OctComputation and LanguageArtificial Intelligence
The gist
Translating patent documents between languages like Chinese, English, Japanese, and Korean can be tricky because of technical terms and formal language. The authors developed PatMT2, a system that improves an existing translation model by training it in two stages specifically for patents. Their improved model translates better, scoring higher on tests than previous versions and some big alternative systems. They also tested it on many other languages to understand how well the model keeps working outside its training set.
Open → 2610.16466v1

Bengali English translations reveal bias in honorific use by language models

Who Receives Respect? A Counterfactual Benchmark for Honorific Allocation Bias in Bengali

Abstract: Bengali has no grammatical gender, but honorificity is obligatory: every third-person reference must choose ordinary or respectful register. English has no such signal, so the system must choose a level of deference that the source never specified. We use this property to measure how large language models (LLMs) allocate respect in English--Bengali translation. Our evaluation set contains 2{,}016 sentences across 72 occupations. Each appears in seven sentence frames under four pronoun conditions. Masculine and feminine versions differ by one word, forming counterfactual pairs. We report a single Honorific Bias Score (HBS) for each model on a 0--100 scale, covering both the gender and occupation axes. A lower score means that respect is allocated more evenly. For the occupation analysis, we compute each occupation's honorific rate across the seven no-pronoun sentence frames. The standard deviation of these 72 occupation-level rates ranges from 17.2 to 35.5 percentage points across models, showing substantial variation in how honorifics are allocated across occupations. Occupation is associated with 3.2\% to 51.5\% of the variance in register choice, compared with at most 1.2\% for gender. Gender effects are much smaller overall: only two of the nine models show significant directional gender gaps, and they favor opposite genders. HBS ranges from 19.9 to 70.1 across the nine models, showing substantial variation in how evenly the models allocate respect.

Sat 10 OctComputation and Language
The gist
Some languages like Bengali always use polite or respectful words when talking about others, while English doesn’t. The authors studied how computer programs that translate English to Bengali decide when to be respectful. They discovered that these programs treat different jobs with very different respect levels, but show much smaller differences based on gender. They created a score to measure how fairly respect is given, and found big differences between different translation models.
Open → 2610.14769v1

Bangla text to speech model achieves near-natural voice cloning with less data

BanglaBox: A Phonetically-Balanced Corpus and Data-Efficient Foundation-Model Adaptation for Bangla Text-to-Speech with Zero-Shot Voice Cloning

Abstract: We present a recipe for adapting English-pretrained autoregressive TTS foundation models to underrepresented languages, demonstrated on Bangladeshi Bangla. Existing Bangla TTS corpora are small and single-speaker, and to our knowledge no open zero-shot voice-cloning system is available for the Bangladeshi register. We contribute a phonetically- and gender-balanced two-tier Bangladeshi Bangla corpus balanced via a tiered Jensen-Shannon divergence objective over conjunct clusters (juktakkhor), together with three fine-tuning changes: a merge-consistent tokenizer extension, Bangla text normalization, and a prompt-masked dual-loss objective. These changes preserve zero-shot cloning across the language switch. Our BanglaEval protocol applies Wilcoxon signed-rank tests with Bonferroni correction over native-speaker ratings. BanglaBox attains near-natural Naturalness, outperforms commercial and open-source baselines on Naturalness, Speaker Similarity, and Clarity, and reaches speaker similarity comparable to prior few-shot results while using approximately 7x less Bangla fine-tuning audio. We further validate the recipe beyond our own test split using a seven-category stress set of difficult real-world text, the public BnTTS evaluation benchmarks, and naturally occurring Bangla that no language model wrote. All artifacts, including the corpus, weights, code, and complete evaluation materials, are released publicly and unconditionally.

Tue 6 OctSound
The gist
Making speech computer programs for languages with few recordings is hard. The authors created a special set of Bangla speech samples and smart training steps to teach a model trained on English to speak Bangla with many different voices. Their approach uses much less Bangla recording data than before but still sounds natural and clear. They tested it carefully with native speakers and compared it to other systems, finding it worked better. All parts of their work are openly shared so others can use or improve them.
Open → 2610.09211v1

Language models reliably recognize and follow script instructions

Are Language Models Script-Aware?

Abstract: Language models frequently generate outputs in unintended languages or scripts, a phenomenon known as off-target generation. While existing research has focused on language selection, the dimension of script knowledge remains understudied: before any linguistic understanding can occur, users must recognize the graphic symbols in a model's response. We investigate whether Small and Large Language Models (SLMs and LLMs) possess script knowledge by testing them on multi-scriptic languages. Through two complementary experiments, we evaluate whether models (1) adapt their output script to match the input, and (2) follow explicit instructions to generate text in a specified script. The models we tested demonstrate substantial script knowledge: they all achieve a near-perfect Latin script fidelity (more than 98%) and follow script instructions with high frequency. Nevertheless, we notice differences between LLMs and SLMs, with higher scores for LLMs including for non-standard script combinations.

Tue 6 OctComputation and Language
The gist
Language models sometimes produce text in the wrong writing system, making it hard to read. The paper explores whether these models can identify and use different scripts correctly, especially for languages that use multiple scripts. The authors found that both small and large language models can match the script of the input text and follow instructions to write in specific scripts, with larger models performing better. This ability helps ensure that computer-generated text looks as expected and is easier for people to understand.
Open → 2610.08037v1

ChindaMT improves Thai-English translation following detailed instructions

Reference-Grounded Data Curation for Instruction-Following Thai-English Machine Translation

Abstract: Instruction-following machine translation (IF-MT) requires respecting prompt-level rules on terminology, formatting, and register. Rule compliance typically trades off against translation quality, a tension that general-purpose IF data augmentation methods do not address. We propose Reference-Grounded Data Curation, a two-phase pipeline that extracts every supervised constraint from a reference translation that already satisfies it, ensuring feasibility by construction. Phase 1 applies Instruction-Following Difficulty (IFD) scoring to retain the hardest-but-learnable instances from an English-Thai parallel pool. Phase 2 extracts constraints from each reference target and keeps only generations satisfying every constraint, yielding the 1.97M-record Grounded dataset. We fine-tune open-weight bases on Grounded to produce ChindaMT, a Thai-English translation family at 4B, 2B, and 0.8B parameters. Under length-controlled pairwise judging, ChindaMT outperforms or matches every same-size baseline at every tier on both plain translation and under explicit rules, reaching up to a 68.4% win rate against the strongest baseline. The recipe transfers cleanly across Qwen generations. We release model weights, the Grounded dataset, and evaluation suites.

Mon 28 SeptComputation and Language
The gist
Translating between Thai and English accurately and following specific instructions about word choice and style is hard. The authors created a new method that picks the most useful training examples and makes sure the translated text meets all the given rules from examples that already do. Their resulting translation models, called ChindaMT, produce better or equally good translations compared to existing models, especially when strict rules need to be followed. They also share their trained models, data, and tools for others to use.
Open → 2609.34770v1

Cross-language study reveals how players judge video games differently

Deciphering the Babel of Play: A Human-AI Collaborative Approach for Large-Scale Cross-Language Analysis of Game Reviews

Abstract: We present a large-scale cross-language analysis of game reviews using a human-AI collaborative framework that combines quantitative screening with multilingual large language models (LLMs). Starting from 17 million Steam reviews across 30 languages and 2,000 top-selling titles, we select 28 games with notable cross-language rating patterns. We then apply LLM-assisted content analysis to 442,162 reviews spanning 17 languages, with human researchers guiding codebook development and interpreting the results. Our findings reveal differences in both the aspects language communities prioritize and how they evaluate them, highlighting the roles of narrative expectations, game mechanics and stability, localization quality, cultural proximity, and perceptions of developers and publishers. We also identify rare cases of cross-language consensus. This work offers empirical insights into cross-cultural game evaluation and a scalable methodological approach to multilingual content analysis that preserves human interpretation.

Sat 19 SeptHuman-Computer Interaction
The gist
People from different language backgrounds often rate video games differently because they focus on different things like story, game mechanics, or how well the game is translated. The authors looked at hundreds of thousands of game reviews in many languages using a mix of human help and AI language tools. They found both differences and some rare areas where players agree regardless of language. This helps us understand how culture affects opinions about games.
Open → 2609.23104v1

Gender bias affects machine translation scores across jobs and languages

Benchmarking Gender Bias in Machine Translation Evaluation Metrics across Occupations

Abstract: Gender bias remains a persistent concern in machine translation (MT), affecting both generated translations and their automatic evaluation. When a source text leaves a person's gender unspecified, translations may realize that person using masculine or feminine forms, and both MT systems and evaluation metrics may exhibit systematic preferences between these alternatives despite the source providing no basis for such a distinction. We study this behavior in the WMT 2026 Automated Translation Quality Evaluation Systems Shared Task using an occupation-balanced subset of GAMBIT+. We consider seven English-source language pairs, six from the original dataset, targeting Arabic, Czech, Greek, Icelandic, Russian, and Ukrainian, and extend the original resource with German. The subset contains 1,308 masculine/feminine translation pairs per target language, with three examples for each of the 436 ISCO-08 occupational groups. We evaluate shared-task submissions and baselines for score prediction and error annotation, examining the direction, magnitude, and frequency of gender-related differences. We find an overall tendency for masculine translations to receive higher scores, as well as differences per occupation following stereotypical gender representations, although the strength and consistency of this preference vary considerably across evaluators and languages. Our results show that gender bias remains present in MT evaluation, but that capturing its extent requires looking beyond a single aggregate measure to complementary dimensions of evaluator behavior.

Fri 18 SeptComputation and Language
The gist
When translating texts where a person's gender is unknown, machine translation systems and their evaluation tools sometimes prefer masculine forms over feminine ones. The authors studied how this bias appears in translations involving different occupations across several languages. They found that masculine translations often get higher scores, and these preferences sometimes reflect job gender stereotypes. However, this bias varies widely depending on the language and the evaluation method used.
Open → 2609.21490v1

TeochewBench benchmark evaluates Teochew Hanzi translation quality

TeochewBench: A Human-Reviewed Benchmark for Teochew Hanzi Translation

Abstract: Teochew has a substantial speaker community and exhibits distinctive lexical, syntactic, and pragmatic features, yet textual resources for evaluating large language models remain limited. We present TeochewBench, a human-reviewed benchmark comprising 300 Teochew Hanzi expressions for evaluating translation from Teochew Hanzi into Mandarin Chinese and English. The dataset covers five categories: basic vocabulary; everyday sentences; Teochew-specific expressions; tone, politeness, and context; and idiomatic, ambiguous, and culturally specific expressions. A primary Teochew-speaking reviewer examined all entries individually and revised them as needed, while two additional Teochew speakers verified selected items. Our main evaluation covers 11 official general-purpose post-trained models on the reviewed dataset in both translation directions, yielding 6,600 predictions. Two official base checkpoints provide 1,200 predictions for supplementary diagnostics, bringing the total to 13 models and 7,800 predictions. We additionally include a Hanzi-copy control, which returns the source input unchanged, to assess how shared Hanzi affect automatic scores for translation into Mandarin Chinese. Qwen3.5-27B achieved the highest overall chrF-style score among the evaluated checkpoints, at 60.63, followed by Qwen2.5-72B-Instruct at 56.61, Gemma-3-27B-IT at 56.36, and GLM-4-32B-0414 at 55.82. Across the 11 main-evaluation models, the mean chrF-style score decreased from 69.25 for low-specificity items to 27.52 for high-specificity items. High-specificity expressions received lower scores and exhibited smaller cross-model differences, suggesting that they constitute a shared low-scoring region across the model families evaluated here. The Hanzi-copy control further indicates that surface overlap in low-specificity items can substantially affect automatic scores for translation into Mandarin Chinese.

Wed 16 SeptComputation and Language
The gist
Teochew is a distinct Chinese dialect with unique words and expressions, but there are few resources to test how well computers translate its written form. The authors created TeochewBench, a collection of 300 Teochew written phrases checked by native speakers, covering everyday and culture-specific expressions. They tested 13 language models to see how accurately they translate these phrases into Mandarin and English. Results showed models struggle more with specific or culturally unique phrases, highlighting translation challenges where meanings are subtle or tied to context.
Open → 2609.18156v1

Measuring translation quality loss from mixing similar Mozambican language varieties

Measuring the Cost of Variety Conflation in Multilingual MT Evaluation: Adding Mozambican Xichangana, Nyanja and Sena to FLORES+

Abstract: In this paper, we extend FLORES+ with Portuguese-source evaluation sets for three Mozambican Bantu varieties: Xichangana, Mozambican Nyanja, and Sena. We compare Xichangana with the existing Tsonga reference and Mozambican Nyanja with Chichewa, and evaluate NLLB-200, Google Translate, GPT, and a variant-aware NLLB model. Holding system output fixed reveals substantial reference sensitivity. On \textit{devtest}, changing only the reference from Tsonga to Xichangana reduces spBLEU by 13.10 points for NLLB-200 and 15.30 for Google. On matched Nyanja subsets, replacing Chichewa with Mozambican Nyanja produces smaller but consistent reductions of 3.03 and 6.10 spBLEU, respectively. Variant-aware fine-tuning reverses this pattern on the intended targets: relative to NLLB-200, it improves Xichangana by 7.04 spBLEU and Mozambican Nyanja by 5.33 on \textit{devtest}, while losing performance on the sibling references. GPT is competitive on Tsonga and Chichewa but substantially weaker on the Mozambican varieties. For Sena, the finetuned model reaches 12.64 spBLEU and 36.21 chrF++ on \textit{devtest}. These findings motivate variety-aware language identifiers, references, and reporting for cross-border languages or language dialects/variants. The data is publicly available on Hugging Face at https://huggingface.co/datasets/MOZNLP/FLORES_MOZ

Sat 12 SeptComputation and Language
The gist
When automatic translation tools try to translate languages with many similar varieties or dialects, mixing them up can cause big drops in quality. The authors added new test data sets for three Mozambican languages to an existing benchmark to study this problem. They found that using the wrong variety as a reference for evaluation can reduce measured translation accuracy by up to 15 points. Training models to recognize specific language varieties improved results for the intended targets but harmed accuracy on related varieties. This work highlights the need for language-aware tools and more precise evaluation when dealing with closely related languages.
Open → 2609.13847v1

Multilingual large language models struggle with Urdu stories

Multilingual in Name Only? Cultural and Linguistic Weaknesses of LLMs in Urdu

Abstract: Multilingual large language models (LLMs) are increasingly used for open-ended text generation, yet their behaviour in low-resource languages remains poorly understood. In this work, we question how correct and reliable is the generation of multilingual LLMs when used for the task of story generation. We consider Urdu language as a representative low-resource language. We generate Urdu-Stories, a corpus of 93 stories generated using three contemporary LLMs (GPT-5.1, Qwen-3-Max, DeepSeek-3.1). We manually annotate the errors present in them under a nine-label linguistic, semantic, and cultural taxonomy. Our notable findings suggest that LLMs often make basic errors of grammar and semantics. The stories lack coherence, have unnatural repetition and show pervasive cultural shallowness. We further show using few-shot prompting that the cultural and context errors largely remain unresolved. Our findings highlight the limitations of current LLMs as a reliable source of content generation and information retrieval for low-resource languages.

Wed 9 SeptComputation and LanguageArtificial IntelligenceMachine Learning
The gist
Large language models (LLMs) can write stories in many languages, but their ability in less common languages like Urdu is not well understood. The authors studied three popular LLMs that generated Urdu stories and found many problems. These stories often had basic grammar mistakes, confusing meanings, repeated phrases, and lacked cultural depth. Even when given extra examples to improve, the issues remained. This shows that current LLMs are not yet reliable for creating or understanding content in low-resource languages like Urdu.
Open → 2609.10758v1

Machine translation accuracy improves by accounting for term variation

Improving Term Evaluation in Machine Translation: Variation Matters

Abstract: Terminology evaluation in machine translation (MT) usually assumes a single correct target form per source term. However, human translators routinely introduce variation that current metrics penalize as inconsistency. We examine how to account for this variation in document-level MT evaluation of English-French scientific translation, combining glossary-based accuracy, translation consistency, and a new cross-term variation (CTV) diagnostic measure that tests whether variation relationships are preserved across languages. Based on analyses of two parallel corpora, translated by four MT systems, we find that (1) MT systems generate less target-side variation than human translators; (2) transfer patterns strongly depend on the variation type; (3) consistency rankings vary with the choice of metric; and (4) constraining MT with a glossary improves accuracy and consistency but degrades CTV by suppressing valid variation. We argue for variation-aware evaluation that conditions consistency penalties on whether target-side variation mirrors source-side variation.

Tue 8 SeptComputation and Language
The gist
Machine translation systems usually treat each word or term as having only one correct translation, but human translators often use different valid versions of the same term. The authors studied English-to-French scientific translations and found that current evaluation methods wrongly penalize translations for these natural variations. They introduced a new way to measure whether variations in the translation reflect variations in the source text. Their work shows that recognizing valid variations can make evaluation of translation quality more accurate and fair.
Open → 2609.08779v1