Syntactic language models lose edge when scaled to billion parameters
A Scaled-Up Empirical Study of Syntactic Language Models
Computation and LanguageArtificial Intelligence
Summary
Syntactic language models try to generate sentences along with their grammatical structures, which can help computers understand language better. This study tested bigger and more powerful versions of these models to see if they perform better on language tasks. The researchers found that simpler models with standard attention work just as well or better than those with complex syntax-focused methods. Also, when the models get very large, the syntax-based models no longer outperform simpler models with the same amount of computing. This suggests that scaling up model size may reduce the benefits of explicitly modeling syntax.
What this means in practice
- •For natural language processing engineers: Design language models that avoid complex syntax-specific architectures but still perform well on diverse language tasks at large scales.
- •For machine learning infrastructure teams: Implement efficient attention mechanisms compatible with popular acceleration tools like FlashAttention in large-scale language models.
Authors
Pengcheng Wang, Zhexuan Zhang, Chao Lou, Kewei Tu
Abstract
Syntactic language models (SLMs) that generate sentences along with their syntactic tree structures have shown promise in both syntactic generalization and downstream tasks. Prior studies, however, have mostly evaluated smaller models trained on less pretraining data and tested on narrower task suites. In addition, the design space of tree linearization and attention masks remains underexplored. We conduct a large-scale empirical study of 12 controlled configurations---one terminal baseline and 11 SLM variants spanning different linearizations and masks---at scales up to 1B parameters and 100B training tokens, evaluated across 11 downstream tasks and two syntactic benchmarks. The experimental results show that SLMs with standard causal attention match or exceed structural-mask variants across downstream and syntactic benchmarks. They require no architectural modifications and remain compatible with FlashAttention. We also observe that when scaling up to the 1B-parameter scale, the top-performing SLM variant no longer shows a clear advantage over a compute-matched non-structural baseline, suggesting that SLMs may become less competitive with scale. Our code is available at https://github.com/qlwpc/TG-Interpolation.