Uncertainty estimation improves trust in sentence meaning parsers
How Reliable are AMR Parser Outputs? Uncertainty Estimation for AMR Parsing
Computation and LanguageArtificial Intelligence
Summary
Parsing sentences to understand their meaning is challenging, and computer programs called AMR parsers do this automatically. However, it is unclear when these parsers' outputs can be trusted. This paper studies how to estimate uncertainty in AMR parsers by looking at multiple possible interpretations they generate. The authors found that this uncertainty estimate helps to tell when parser outputs are reliable, adding an important new way to evaluate these tools.
What this means in practice
- •For natural language processing engineers: Use uncertainty scores to decide when to trust automatic sentence meaning parses for better downstream language tasks.
- •For machine translation developers: Incorporate parser uncertainty signals to avoid propagating unreliable semantic parses into translation pipelines.
Authors
Anuj Kumar, Shintaro Ozaki, Yamuna Prasad, Katsuhiko Hayashi, Hidetaka Kamigaito, Taro Watanabe
Abstract
Recent advances have significantly improved the performance of Abstract Meaning Representation (AMR) parsers, yet determining when their outputs can be trusted remains largely unexplored. This work presents the first study of uncertainty estimation for AMR parsers in a black-box setting using top-k beam-search candidates. Uncertainty is estimated from pairwise candidate similarities and evaluated using the Prediction Rejection Ratio. This work characterises the relationship between parsing performance and uncertainty, evaluates the reliability of beam candidate distributions, analyses the effect of evaluation metrics on uncertainty assessment, and identifies parsers that provide the strongest balance between performance and reliability. Experimental results show the value of uncertainty estimation as a complementary dimension for evaluating AMR parsers.