TIAR: Trajectory-Informed Advantage Reweighting for LLM Abstention Learning
2026-05-25 • Computation and Language
Computation and LanguageArtificial IntelligenceMachine Learning
AI summaryⓘ
The authors study how large language models can decide when to say 'I don't know' (abstain) to avoid giving wrong answers. They improve on previous methods by adjusting the reward for abstaining based on how confident the model is, using information from multiple answer attempts during training. This new method helps the model better identify when to abstain without losing accuracy. They tested their approach on a special benchmark called AbstentionBench and found it worked better than older methods in most cases.
large language modelsabstention learningreward reweightingGroup Relative Policy Optimization (GRPO)hallucination reductionTrajectory-Informed advantage reweighting (TIAR)confidence estimationbenchmark evaluationF1 score
Authors
Muyu Pan, Shu Zhao, Nan Zhang, Philip Shin, Varun Parekh, Vijaykrishnan Narayanan, Rui Zhang
Abstract
This paper investigates large language model (LLM) abstention learning, specifically using ternary reward, which incentivize truthfulness in large language models. This paper extends that idea by moving from a ternary reward to a Trajectory-Informed advantage reweighting, dynamically re-weights the abstention reward during Group Relative Policy Optimization (GRPO) training. The objective of this work focuses on abstention learning instead of improving truthfulness, serving as an exploration into hallucination reduction. The novelty of this paper lies in methodological innovation, advantage re-weighting, and benchmark selection. Leveraging GRPO's multiple trajectories as a natural abstention signal, this method uses a reward signal to explore knowledge boundaries and encourage consistency. By demonstrating that trajectories can be used as a confidence indicator of the policy relative to the query, they are then used to dynamically calculate the abstention advantage. AbstentionBench is used as the evaluation benchmark, as this work aims to contribute to the field of abstention learning. All datasets on the benchmark were tested against this method and various baselines. Empirical results demonstrate that TIAR achieves state-of-the-art abstention F1 scores across five of six evaluation categories, outperforming the static ternary baseline on 17 of 31 benchmark datasets while fully preserving baseline accuracy.