Generating SPARQL Queries over CIDOC-CRM using a Two-Stage Ontology Path Patterns Method in LLM Prompts

Abstract

In this paper, we focus on the task of exploiting the capabilities of Large Language Models (LLMs) to generate SPARQL Queries for answering natural questions over cultural Knowledge Graphs (KGs) expressed according to the ISO standard ontology CIDOC-CRM. Since CIDOC-CRM is an event-based model, usually we have to follow long paths for answering a question, thereby, the challenge is how to construct the prompt for aiding the LLM to produce the right SPARQL query. We propose and comparatively evaluate methods based on the creation of ontology path patterns of a configurable path radius (or length). Then, we construct a new dedicated benchmark that includes 100 natural questions and the corresponding SPARQL queries over two real KGs from the cultural domain describing artworks. Finally, we present comparative results about the effectiveness and efficiency over the benchmark by using ChatGPT-3.5. The most effective method follows a two-stage process that predicts and uses the most appropriate path patterns of \(r\leq 4\) . This method achieves 3.5 \(\times\) higher accuracy than the baseline method (0.66 versus 0.19), that includes in the prompt only the list of properties and classes of the KG. Benchmark: https://github.com/mountanton/CIDOC-QA-using-LLMs