GoiBERT delivers smaller faster Vietnamese language models with strong accuracy
GoiBERT: Compact Vietnamese Language Models via Two-Stage Knowledge Distillation
Computation and Language
Summary
Vietnamese language models that understand text well tend to be very large or rely on extra tools that slow them down. The authors created GoiBERT, a smaller model that reads raw Vietnamese text directly and learns from two larger, expert models. Their compact model performs nearly as well as bigger ones on many language tasks but needs less memory and runs faster. It works best for sentiment analysis and text classification, while some tasks like entity recognition still lose accuracy when the model is smaller.
What this means in practice
- •For mobile app developers: Build efficient Vietnamese language features that run faster and use less memory on phones without needing word segmentation.
- •For customer support teams: Implement lightweight sentiment analysis models for Vietnamese messages to monitor customer satisfaction in real time.
Authors
Quang Anh Kiet Tran, Nam Anh Tran, Manh Hung Trinh, Thanh Danh Le, Khang Nhat Hoang Vo, Huynh Dai Nhan Tran
Abstract
Vietnamese pre-trained encoders have improved rapidly, but the strongest ones are large, multilingual-vocabulary models or depend on external word segmentation, which makes them costly to deploy. We present GoiBERT, a family of two compact Vietnamese encoders with 26M and 49M parameters that operate directly on raw text. GoiBERT learns from two complementary teachers. A same-tokenizer Vietnamese encoder first teaches general-purpose representations. A much larger teacher then provides task-level guidance through soft labels that are generated without ever labeling its own training data. Across eight benchmarks covering text classification, aspect-based sentiment analysis, natural language inference, and named entity recognition, GoiBERT-large achieves the highest average score among non-teacher models. It does so with fewer than half the parameters of the strongest compact baseline and twice its throughput. GoiBERT-base retains 99% of that baseline's performance at a quarter of its size. Our analysis shows that compression is nearly lossless for sentiment and classification tasks but remains costly for inference and entity recognition, which points to clear directions for future work. Our code and models are available at: https://github.com/Nhantran65/GoiBERT.git