Abstract
We describe the techniques applied by the University of Alberta (UA) team in the most recent Competition on Legal Information Extraction and Entailment (COLIEE 2021). We participated in retrieval and entailment tasks for both case law and statute law; we applied a transformer-based approach for the case law entailment task, an information retrieval technique based on BM25 for legal information retrieval, and a natural language inference mechanism using semantic knowledge applied to statute law texts. This competition included 25 teams from 14 countries; our case law entailment approach was ranked no. 4 in Task 2, the BM25 technique for legal information retrieval was ranked no. 3 in Task 3, and the natural language inference technique incorporating semantic information was ranked no. 4 in Task 4. The combination of the latter two techniques on Task 5 was ranked no. 2. We also performed error analysis of our system in Task 4, which provides some insight into current state-of-the-art and research priorities for future directions.
Highlights
To help build a legal research community, the Competition on Legal Information Extraction and Entailment (COLIEE) was created, to develop a research community that focuses on four specific challenge problems in the legal domain: case law retrieval, case law entailment, statute law retrieval and statute law entailment
Our method for the case law entailment task is based on adapting our methods from the past editions [1, 2], with an increased focus on transformer methods and a heuristic post-processing technique based on a priori probabilities
This approach faces two main problems: the lack of sufficient training data to make the models converge and generalize, and the computational cost of training, which increases exponentially on the size of the dataset. They proposed two association rule models: (1) the basic association rule model, which considers only the similarity between the source document and the target document, and (2) the co-occurrence association rule model, which uses a relevance dictionary in addition to the basic model. Another technique [20] worth mentioning approached the task as a binary classification problem, and built feature vectors comprised of the measures of similarity between the candidate paragraph and (1) the entailed fragment of the base case, (2) the base case summary and (3) the base case paragraphs
Summary
Tools to help legal professionals manage the increasing volume of legal documents are essential. The current state-of-the-art, especially for problems which have access to enough labeled data, relies on deep learning-based approaches (more notably those based on transformer methods), which have shown very good results in a wide range of textual processing benchmarks, including benchmarks specific to entailment tasks. Our method for the case law entailment task is based on adapting our methods from the past editions [1, 2], with an increased focus on transformer methods and a heuristic post-processing technique based on a priori probabilities. In this year, we decided to drop similarity calculations, as our previous results have shown they did not significantly contribute to improved performance.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
