Exploratory Inference Chain: Exploratorily Chaining Multi-hop Inferences with Large Language Models for Question-Answering

Abstract

Successful few-shot question-answering with large language models (LLMs) has been reported for a variety of tasks. In the usual approach, an answer is generated by a single call to an LLM, but it has been pointed out that the performance of multi-hop inference by LLMs is not sufficient. Thus, an LLM is unable to perform the complex processing necessary to get an answer, which leads to poor performance. Moreover, the inference process is opaque. Against this, approaches that call an LLM multiple times have been proposed, but many of these approaches can only be used for a limited number of effective tasks, and LLMs essentially require complex processing.To address these problems, we propose the Exploratory Inference Chain (EIC) framework that combines the implicit processing of LLMs with explicit inference chains, and this is based on the dual process theory of human cognitive processes. The EIC framework first generates the information needed to answer a multi-hop question as keywords and then performs 1-hop inference for each keyword. If the inference is not sufficient, additional inferences are performed. This process is repeated, and when sufficient inferences are obtained, they are aggregated, and the final answer is generated. This makes the information per inference by LLM simplified, and logical inference is achieved through an explicit inference chain.We conducted experiments on two multi-hop QA datasets and confirmed through a quantitative evaluation that our EIC framework performed better than existing approaches. Moreover, a qualitative evaluation confirmed that our approach can effectively perform inference so as to get closer to the answer in question-answering tasks that require knowledge. In addition, compared with existing approaches, the EIC framework improves the interpretability of the output.