Large language models improve satire detection in Romanian code-mixed sentences
Benchmarking Large Language Models for Detecting Satirical Code-Mixed Sentences
Computation and Language
Summary
Satire is tricky to spot when people mix languages in a sentence, especially in languages with fewer resources like Romanian. The authors created a large set of Romanian sentences mixed with English, German, French, and Italian to help test if big language models can find satire in these tricky examples. They tested several methods and found that newer AI models can detect satire much better than older ones. Training the models specifically on these mixed-language examples made them even more accurate.
What this means in practice
- •For multilingual ai developers: Develop and fine-tune AI systems to detect satire in mixed-language Romanian sentences more accurately using the new large synthetic dataset.
- •For social media content moderators: Use improved language models to automatically identify satirical posts that combine Romanian with other languages, aiding content review.
Authors
Răzvan-Alexandru Smădu, Ioana Buhnila, Daniela Tanase, Dumitru-Clementin Cercel, Florin Pop, Mihaela-Claudia Cercel
Abstract
Code-switching is a natural linguistic phenomenon, yet its impact on high-level reasoning tasks remains largely unexplored in under-resourced languages like Romanian. Natural language processing tasks, such as satire detection, rely heavily on nuanced cultural and syntactic cues that large language models struggle to capture. In this paper, we investigate four methods (i.e., random replacement, phrase translation, dependency parsing, and large language models) for generating synthetic code-mixed Romanian sentences with English, German, French, and Italian. We start with an existing dataset for satire detection, compiled from Romanian news articles and annotated at the sentence level by humans, and augment it with over 600k samples, namely CS-SeLeRoSa. We provide an in-depth analysis of the dataset generation process. Then, we evaluate several multilingual large language models across zero-shot, chain-of-thought, and fine-tuning regimes on both the synthetic corpus and a human-adjudicated gold benchmark, namely CS-SeLeRoSa-Gold. While older open models suffer substantial degradation (dropping to 37%-55% macro-F1), newer models like Gemma 4 31B achieve 68.1% zero-shot, rivaling frontier models like GPT-5 mini (69.5%). Furthermore, fine-tuning significantly reduces formatting errors and improves the results, increasing Gemma 4 31B to 77.5% on CS-SeLeRoSa and 72.6% on the gold standard, outperforming frontier zero-shot baselines.