From “what” to “how”: Extracting the Procedural Scientific Information Toward the Metric-optimization in AI

Abstract

In response to the exponential growth of the volume of scientific publications, researchers have proposed a multitude of information extraction methods for extracting entities and relations, such as task, dataset, metric, and method entities. However, the existing methods cannot directly provide readers with procedural scientific information that demonstrates the path to the problem's solution. From the perspective of applied science, we propose a novel schema for the applied AI community, namely a metric-driven mechanism schema (Operation, Effect, Direction). Our schema depicts the procedural scientific information concerning “How to optimize the quantitative metrics for a specific task?” In this paper, we choose papers in the domain of NLP for our study, which is a representative branch of Artificial Intelligence (AI). Specifically, we first construct a dataset that covers the metric-driven mechanisms in single and multiple sentences. Then we propose a framework for extracting metric-driven mechanisms, which includes three sub-models: 1) a mechanism detection model, 2) a query-guided seq2seq mechanism extraction model, and 3) a task recognition model. Finally, a metric-driven mechanism knowledge graph, named MKGNLP, is constructed. Our MKGNLP has over 43K n-ary mechanism relations in the form of (Operation, Effect, Direction, Task). The human evaluation shows that the extracted metric-driven mechanisms in MKGNLP achieve 81.4% accuracy. Our model also shows the potential for creating applications to assist applied AI scientists to solve specific problems.