BERT-based ensemble methods with data augmentation for legal textual entailment in COLIEE statute law task

Abstract

The Competition on Legal Information Extraction/Entailment (COLIEE) statute law legal textual entailment task (task 4) is a task to make a system judge whether a given question statement is true or not by provided articles. In the last COLIEE 2020, the best performance system used bidirectional encoder representations from transformers (BERT), a deep-learning-based natural language processing tool for handling word semantics by considering their context. However, there are problems related to the small amount of training data and the variability of the questions. In this paper, we propose a BERT-based ensemble method with data augmentation to solve this problem. For the data augmentation, we propose a systematic method to make training data for understanding the syntactic structure of the questions and articles for entailment. In addition, due to the nature of the non-deterministic characteristics of BERT fine-tuning and the variability of the questions, we propose a method to construct multiple BERT fine-tuning models and select an appropriate set of models for ensemble. The accuracy of our proposed method for task 4 was 0.7037, which was the best performance among all submissions.