Cantonese language model improves reasoning with limited data
CantoneseLLM v2: Reasoning in a Low-Resource Language
Computation and Language
Summary
Cantonese is widely spoken but lacks large amounts of written data, especially examples of logical reasoning for teaching computers. The authors created CantoneseLLM v2, a set of language models trained on many Cantonese and Hong Kong-related texts, using special training steps to improve understanding and reasoning in Cantonese. They found that some training methods helped the models follow instructions but weakened reasoning ability, while others restored reasoning and improved performance. The final model closely matches previous versions in test scores but better understands and reasons in Cantonese.
What this means in practice
- •For chatbot developers: Create Cantonese-speaking chatbots that can follow instructions and reason using limited native data.$Commercial implications: Enables development of Cantonese chatbots with better reasoning for customer support or virtual assistants, filling a market gap.
- •For natural language processing engineers: Build Cantonese NLP tools that leverage large training datasets and advanced fine-tuning to improve language understanding in low-resource settings.
Authors
Tsz Chung Cheng, Chung Shing Cheng, Chaak Ming Lau, Cheuk Hei Chong
Abstract
Cantonese is widely spoken but remains low-resource in written data, with no large corpus of native Cantonese reasoning traces available for model training. We develop and release CantoneseLLM v2, comprising models based on Qwen3 8B and 30B-A3B. The models are trained through CPT on 784 million Cantonese and Hong Kong-related tokens, chat-vector merging, SFT, DPO, and RLVR. Evaluation across the training stages shows that chat-vector merging transfers instruction following but preserves the donor model's reasoning language, while SFT with limited Cantonese reasoning data substantially shortens or removes reasoning traces and reduces benchmark performance. DPO restores the reasoning-block format, particularly for the 8B model, but recovers only part of the lost performance. The RLVR training with Cantonese language and Traditional Chinese scripts as multiplicative constraints introduced Cantonese language alignment and restored the lost performance. The 30B-A3B model reaches 73.16 on HKCanto-Eval, within 1.20 points of its merged checkpoint, while retaining the Cantonese reasoning behaviour absent from that checkpoint. We release the model checkpoints, the training environments, and a thirteen-year Traditional Chinese Common Crawl dataset. The models can be accessed at https://huggingface.co/collections/hon9kon9ize/cantonesellm-v20