Grammarrl improves language model output with grammar and reinforcement learning
GrammarRL: Effective Grammar-Constrained Decoding via Reinforcement Learning
Artificial IntelligenceComputation and LanguageMachine Learning
Summary
Sometimes language models are told to only produce sentences that follow certain grammar rules, but this can make their answers less meaningful or accurate. The authors developed GrammarRL, a method that teaches a language model to better follow grammar rules without needing extra example data. It uses special rewards based on how well the model predicts the input and output together. Tests on different language tasks showed GrammarRL works better than traditional methods, giving more accurate and grammatically correct outputs without extra computing cost.
What this means in practice
- •For machine translation developers: Generate grammatically valid and semantically accurate translations efficiently with reduced decoding cost for structured languages like sign language.
- •For natural language processing teams: Improve entity tagging and hierarchical text classification outputs by adapting language models to grammar constraints without labeled data.
Authors
Gabriele Tuccio, Antonino Furnari, Aldo Gangemi, Misael Mongiovì
Abstract
Grammar-constrained generation guarantees syntactic validity, but can substantially degrade semantic quality when the model's preferred outputs are poorly aligned with the imposed grammar. This trade-off is particularly severe when the prompt is underspecified or the model has limited instruction-following ability. Beam search can partially mitigate these failures by exploring multiple valid sequences, but its computational cost grows with beam width, while sequence-level probability is only an imperfect proxy for semantic quality. We introduce GrammarRL, a label-free reinforcement learning method that adapts language models to grammar constraints without requiring annotated data. GrammarRL optimizes the model using two complementary self-supervised rewards derived from its own likelihoods: a direct reward, measuring how likely the constrained output is given the input, and a reverse reward, measuring how well the input can be reconstructed from the generated output. We optimize these rewards with a Reinforce Leave-One-Out (RLOO) objective over groups of grammar-constrained rollouts, augmented with the top-1 beam-search hypothesis and regularized towards a frozen base model. We evaluate GrammarRL on sign language gloss translation, hierarchical text classification, and named entity recognition using Llama models ranging from 1B to 8B parameters. GrammarRL consistently outperforms constrained greedy decoding, with an average improvement of 9.8 points and gains of up to 22.8 BLEU. It matches or outperforms beam search on two of the three tasks while preserving greedy-decoding inference cost. Ablations further show that the two rewards are complementary: either reward alone can underperform the untrained baseline, whereas their combination consistently improves upon it.