Summary
Language models can be trained to give answers that follow certain formatting rules using reinforcement learning with verifiable rewards (RLVR). However, the authors found that when models learn to prefer one way of reporting answers, they may ignore new instructions asking for a different format, even if the answer itself is still correct. They showed this by training models on different reporting styles and testing if they could switch formats when asked. Their results reveal that simply measuring whether answers match an expected format is not enough to understand how well models follow current instructions. This suggests we should also check how well models stick to the specific requests given each time they produce an answer.
What this means in practice
- •For language model developers: Detect when reinforcement-learned preferences reduce adherence to new answer formatting requests to improve model responsiveness.
- •For ai compliance teams: Monitor language model outputs for alignment with task instructions beyond surface-level format accuracy for regulatory or policy adherence.
Abstract
Reinforcement learning with verifiable rewards (RLVR) has become a prominent approach for improving language-model performance on reasoning tasks using automatically checked answers. Yet convention-matched evaluation cannot reveal whether reinforcing one reporting convention reduces adherence to a different request that the initial policy already follows. To test this, we train matched policies under two reporting conventions and evaluate each policy under both current requests, using the same initial policy as a shared reference. We complement this crossed design with controlled interventions and independent human calibration. On GSM8K, boxed-format RLVR reduces the fraction of Qwen2.5-7B responses containing the requested hash-format payload by 35.33--74.37 percentage points relative to a 95.45% initial baseline in four of five training seeds; the fifth improves by 2.50 points. In the four deteriorating runs, almost every response that omits the requested payload instead retains the trained boxed convention, and the same four seeds deteriorate under two fixed paraphrases. Changing only the final-answer marker in supervised targets reverses which reporting convention the model prefers across three seeds, providing controlled evidence that this preference is learnable. Across three settings with independent human calibration, gains under a convention-sensitive scorer exceed the corresponding gains in committed-answer correctness, i.e., the correctness of the answer the model actually commits to. Together, these results separate three distinct post-training outcomes: learned reporting preference, current-request adherence, and committed-answer correctness. They show that convention-matched accuracy alone does not fully characterize post-training behavior and motivate evaluating current-request adherence alongside convention-matched task accuracy.