Large language models improve relation extraction from academic texts
Enhancing Knowledge Entity Relation Extraction in Abstracts and Full-text Content of Academic Articles Using Large Language Models
Computation and LanguageDigital LibrariesInformation Retrieval
Summary
Finding connections between key ideas in academic papers can be tricky, especially across abstracts and full texts. The authors tested several ways to use large language models (LLMs) to spot these relationships more accurately. They created special datasets and showed that some adapted models perform better than others. They also looked closely at the types of mistakes made and suggested ways to improve future models. Combining different methods led to even better results.
What this means in practice
- •For academic digital libraries: Improve automated extraction of relationships between concepts in scholarly articles to enhance search and discovery features.
- •For legal document analysis teams: Extract relations between entities in complex legal texts to support case preparation and information retrieval.
Authors
Yingyi Zhang, Chengzhi Zhang
Abstract
This study investigates the effectiveness of large pre-trained language models (LLMs) for knowledge entity relation extraction (KERE) from academic papers. Existing studies have examined individual types of LLM-based strategies and evaluated model performance using metrics such as precision and recall. To provide a broader evaluation, this study investigates multiple strategies, including closed-source LLM-based CoT prompting, LoRA-adapted open-source LLMs, LLM-enhanced small pre-trained language models (SLMs), and SLM-enhanced closed-source LLMs. In addition, this study introduces two strategies, namely SLM-enhanced open-source LLMs and closed-source LLM-enhanced open-source LLMs. These strategies are evaluated from multiple perspectives, including metrics, their ability to mitigate errors observed in SLMs, and potential directions for model optimization. This study manually constructs two annotated datasets in the NLP domain, covering Abstract and Full-Text datasets. Experimental results show that the LoRA-adapted T5 model achieves the highest macro F1 scores of 0.7408 and 0.7013 on the Abstract and Full-Text datasets, respectively. SLMs also benefit from LLM-generated entity descriptions under the LLM-enhanced SLM strategy. Error analysis further reveals that the effectiveness of LLM-based strategies in mitigating SLM errors varies across error types. Based on the identified limitations and error patterns of these strategies, several optimization directions are proposed. In addition, combining strategies with different error patterns further improves KERE performance.