Large models improve spotting research gaps but struggle to create them
Where Research Ideation Begins: Forward-Citation Supervision for Actionable Limitation Generation
Artificial Intelligence
Summary
Coming up with new research ideas often starts by finding a problem or gap in previous studies. The authors found that bigger language models get better at noticing which problems later research solved but don’t get better at inventing new problems to explore. They created a special dataset to teach the models how to generate useful research limitations. Their training method helped the models produce better and more helpful suggestions for new research directions.
What this means in practice
- •For ai research teams: Produce better starting points for new research projects by generating actionable limitations from prior work reports.
- •For product development teams: Identify overlooked issues or gaps in existing technologies to guide innovation and enhancements.
Authors
Liyang Zhao, Yong Cao, Haibin Ling
Abstract
Automated research ideation has advanced rapidly, with multi-agent workflows that generate whole research ideas end to end. These frameworks leave largely unexamined the step that seeds a good idea: identifying an actionable limitation from literature. In this paper, we focus on this first step in the research ideation pipeline. Starting from a pilot study, we find that as large language models (LLMs) scale from 0.6B to 32B parameters, they become far better at recognizing which limitation later work addressed, but no better at generating one, suggesting that the bottleneck lies in the supervision signal rather than model capacity. We therefore construct ActLim-Bench from forward citations, yielding 18,279 evidence-linked supervision triples with human verification. On this signal, we propose a three-stage post-training pipeline: supervised fine-tuning, verifier-filtered rejection-sampling fine-tuning, and reinforcement learning with the verifier as an online dense reward. Experimental results show that our method achieves the highest recall score compared to existing methods among all frontier models. We further show that our generated limitations are preferred in 64.9% of non-tied human comparisons and improve downstream ideation systems. We will release our data and code upon publication.