Astronomical Knowledge Entity Extraction in Astrophysics Journal Articles via Large Language Models

Abstract

Astronomical knowledge entities, such as celestial object identifiers, are crucial for literature retrieval and knowledge graph construction, and other research and applications in the field of astronomy. Traditional methods of extracting knowledge entities from texts face numerous challenging obstacles that are difficult to overcome. Consequently, there is a pressing need for improved methods to efficiently extract them. This study explores the potential of pre-trained Large Language Models (LLMs) to perform astronomical knowledge entity extraction (KEE) task from astrophysical journal articles using prompts. We propose a prompting strategy called Prompt-KEE, which includes five prompt elements, and design eight combination prompts based on them. We select four representative LLMs (Llama-2-70B, GPT-3.5, GPT-4, and Claude 2) and attempt to extract the most typical astronomical knowledge entities, celestial object identifiers and telescope names, from astronomical journal articles using these eight combination prompts. To accommodate their token limitations, we construct two data sets: the full texts and paragraph collections of 30 articles. Leveraging the eight prompts, we test on full texts with GPT-4 and Claude 2, on paragraph collections with all LLMs. The experimental results demonstrate that pre-trained LLMs show significant potential in performing KEE tasks, but their performance varies on the two data sets. Furthermore, we analyze some important factors that influence the performance of LLMs in entity extraction and provide insights for future KEE tasks in astrophysical articles using LLMs. Finally, compared to other methods of KEE, LLMs exhibit strong competitiveness in multiple aspects.