Self reported limitations in nlp research reveal common challenges and trends
What Limits Us? Analyzing Self-Reported Limitations in NLP Research
Computation and Language
Summary
Many Natural Language Processing (NLP) research papers now include sections where authors share the limits of their work. The authors of this study analyzed thousands of these sections from major conferences between 2020 and 2025. They used a mix of human and AI methods to identify common problems researchers admit to and patterns in how these are written. Their work helps us understand what challenges are most frequently faced and how researchers communicate about their own work’s limits.
What this means in practice
- •For conference organizers: Use insights on self-reported challenges to guide review criteria and encourage transparency in future NLP paper submissions.
- •For academic journal editors: Develop structured guidance for authors on reporting limitations based on common themes found across NLP research papers.
A survey. It maps existing work.
Authors
Tawan Thaepprasit, Peeranuth Kehasukcharoen, Ding Wang, Remi Denton, Peerapon Vateekul, Piyawat Lertvittayakumjorn
Abstract
Since late 2022, a Limitations section has become mandatory at many top-tier NLP conferences. The growing number of accepted papers at these venues has resulted in a vast corpus of self-reported limitations that cannot all be manually reviewed, yet remains systematically unanalyzed. Therefore, in this paper, we conduct a large-scale analysis of the Limitations sections from ACL and EMNLP papers published between 2020 and 2025 to understand what researchers disclose about their own work. To do so, we implement a novel human-AI framework for iterative hybrid qualitative coding. This framework enables us to investigate trends in self-reported limitations over time, their correlations with specific paper attributes, and the writing patterns that recur around these disclosures. Our findings offer a critical reflection on the diverse reported challenges as well as the self-reporting practices of researchers in the NLP community.