Can Large Language Models Handle Discourse Particles? A Case Study of Colloquial Malay
2026-05-27 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied how large language models (LLMs) understand small words like "well" and "kind of" that add emotion and intention to speech, focusing on the Malay language. They created a new test called MalayPrag to check how well these models handle these words in casual Malay. They also developed five categories to explain the different meanings of these words. Testing ten existing LLMs, they found that the models struggle to grasp these meanings accurately. The authors noticed that using their five categories helps the models perform better, showing that more organized help is needed for LLMs to understand language nuances.
discourse particleslarge language modelspragmaticsMalay languagelanguage benchmarkpragmatic functionsnatural language processingcolloquial languagemodel evaluationlanguage understanding
Authors
Mariah Al Giptiah Binte Yusoff, Jakin Tan, Bocheng Chen, Guangliang Liu, Xi Chen
Abstract
Discourse particles, such as \textit{well} and \textit{kind of}, are crucial components that enable LLMs to ``speak'' more like humans. They are used to convey emotions, intentions, and interpersonal meanings. However, existing studies have not yet built a comprehensive understanding of LLMs' capabilities in handling discourse particles. Moreover, the limited number of studies focuses primarily on high-resource languages such as English, with little attention paid to Southeast Asian languages. In this paper, we (1) propose \textsc{MalayPrag}, a benchmark designed to systematically evaluate and analyze LLMs' capabilities in handling discourse particles in colloquial Malay; and (2) introduce five attributes that provide a linguistically grounded, unified framework for interpreting the pragmatic functions of discourse particles. Applying these two contributions, we prompt ten off-the-shelf LLMs to perform three prediction tasks. The experimental results reveal substantial challenges for current LLMs in accurately connecting discourse particles with their pragmatic functions in Malay. The provision of the five attributes designed in this study is found to significantly improve these connections, highlighting the need for structured scaffolding for models' pragmatic competence.