BERT-TSR: A BERT-based two-stage ranking model for legal case retrieval

The growing volume of digital documents in Brazilian courts has intensified challenges related to judicial inefficiency, including the slow identification of established precedents and the risk of inconsistent rulings in similar cases. To address this, this study presents a systematic empirical study on the effectiveness of different classes of BERT-based embedding models for the semantic retrieval of legal documents, focusing on identifying relevant precedents for new complaints. A pipeline employing document chunking to handle long legal texts and using Elasticsearch for large-scale dense vector retrieval was implemented and evaluated. Using a corpus of legal complaints from the Maranhão State Court of Justice, three model categories were compared: (i) a general-purpose Portuguese model, (ii) three domain-specific models trained on Brazilian legal corpora, and (iii) a task-specific Sentence-BERT model fine-tuned for similarity tasks. Performance was assessed using a proxy-based protocol with precedent class labels as the ground truth, measured by Precision@k, MRR@15, and MAP@15. The results indicate that the task-specific SBERT-pt model achieved higher performance within the evaluated setting, consistently outperforming other tested models across the selected metrics. This model showed improvements in both immediate relevance (Precision@1 of 0.787) and overall ranking quality (MAP of 0.806). Although domain adaptation provided marginal benefits over the general-purpose baseline, task-specific finetuning for similarity appeared to be the most influential factor for retrieval quality in this scenario. These findings suggest that optimizing models for retrieval tasks can offer a promising direction for enhancing semantic searches in legal contexts. However, further studies using larger and more diverse datasets are needed to confirm the generalizability of these results and assess their impact on real-world judicial workflows.

Medical procedure entity normalization is an important task to realize medical information sharing at the semantic level; it faces main challenges such as variety and similarity in real-world practice. Although deep learning-based methods have been successfully applied to biomedical entity normalization, they often depend on traditional context-independent word embeddings, and there is minimal research on medical entity recognition in Chinese Regarding the entity normalization task as a sentence pair classification task, we applied a three-step framework to normalize Chinese medical procedure terms, and it consists of dataset construction, candidate concept generation and candidate concept ranking. For dataset construction, external knowledge base and easy data augmentation skills were used to increase the diversity of training samples. For candidate concept generation, we implemented the BM25 retrieval method based on integrating synonym knowledge of SNOMED CT and train data. For candidate concept ranking, we designed a stacking-BERT model, including the original BERT-based and Siamese-BERT ranking models, to capture the semantic information and choose the optimal mapping pairs by the stacking mechanism. In the training process, we also added the tricks of adversarial training to improve the learning ability of the model on small-scale training data. Based on the clinical entity normalization task dataset of the 5th China Health Information Processing Conference, our stacking-BERT model achieved an accuracy of 93.1%, which outperformed the single BERT models and other traditional deep learning models. In conclusion, this paper presents an effective method for Chinese medical procedure entity normalization and validation of different BERT-based models. In addition, we found that the tricks of adversarial training and data augmentation can effectively improve the effect of the deep learning model for small samples, which might provide some useful ideas for future research.

Recent developments of machine learning models, and in particular deep neural networks, have yielded significant improvements on several computer vision, natural language processing, and speech recognition tasks. Progress with information retrieval (IR) tasks has been slower, however, due to the lack of large-scale training data as well as neural network models specifically designed for effective information retrieval [9]. In this dissertation, we address these two issues by introducing task-specific neural network architectures for a set of IR tasks and proposing novel unsupervised or weakly supervised solutions for training the models. The proposed learning solutions do not require labeled training data. Instead, in our weak supervision approach, neural models are trained on a large set of noisy and biased training data obtained from external resources, existing models, or heuristics. We first introduce relevance-based embedding models [3] that learn distributed representations for words and queries. We show that the learned representations can be effectively employed for a set of IR tasks, including query expansion, pseudo-relevance feedback, and query classification [1, 2]. We further propose a standalone learning to rank model based on deep neural networks [5, 8]. Our model learns a sparse representation for queries and documents. This enables us to perform efficient retrieval by constructing an inverted index in the learned semantic space. Our model outperforms state-of-the-art retrieval models, while performing as efficiently as term matching retrieval models. We additionally propose a neural network framework for predicting the performance of a retrieval model for a given query [7]. Inspired by existing query performance prediction models, our framework integrates several information sources, such as retrieval score distribution and term distribution in the top retrieved documents. This leads to state-of-the-art results for the performance prediction task on various standard collections. We finally bridge the gap between retrieval and recommendation models, as the two key components in most information systems. Search and recommendation often share the same goal: helping people get the information they need at the right time. Therefore, joint modeling and optimization of search engines and recommender systems could potentially benefit both systems [4]. In more detail, we introduce a retrieval model that is trained using user-item interaction (e.g., recommendation data), with no need to query-document relevance information for training [6]. Our solutions and findings in this dissertation smooth the path towards learning efficient and effective models for various information retrieval and related tasks, especially when large-scale training data is not available.

In this paper, we develop a knowledge representation model for the innovative intelligent retrieval of legal cases, which provides effective legal case management. Examples are taken from the domain of accident compensation. A new set of sub-elements for legal case representation (sub-issues, pro-claimant, pro-respondent and contextual features) has been developed to extend the traditional representation elements of issues and factors. In our representation model, an issue may need to be further decomposed into sub-issues; factors are categorised into pro-claimant and pro-respondent factors; and contextual features are also introduced to help retrieval. These extensions can effectively reveal the factual relevance between legal cases. Based on the knowledge representation model, we propose the IPF scheme for intelligent legal case retrieval. Experiment and statistical analysis have been conducted to demonstrate the effectiveness of the proposed representation model and retrieval scheme.

Vocabulary mismatch is a significant issue when it comes to query-based document retrieval in the medical field. Since the documents are typically authored by professionals, they may contain many specialized terms that are not widely understood or used. Traditional information retrieval (IR) models like vector space and best match-based models fail in this regard. Neural Learning to Rank (NLtR) and transformer models have attracted significant research attention in the field of IR. Recent works in the medical field utilize medical knowledge bases (KB) that map words to concepts and aid in connecting several words to the same concept. In this paper, we present various Siamese-structured transformer and knowledge-based retrieval models designed to address the retrieval issues in the medical domain. The experimental evaluation highlighted the superior performance of the proposed retrieval model, and the best one, based on the UMLSBert ENG transformer, achieved best-in-class performance with respect to all evaluation metrics.

Recently, the performance of machine-reading and comprehension (MRC) systems has been significantly enhanced. However, MRC systems require high-performance text retrieval models because text passages containing answer phrases should be prepared in advance. To improve the performance of text retrieval models underlying MRC systems, we propose a re-ranking model, based on artificial neural networks, that is composed of a query encoder, a passage encoder, a phrase modeling layer, an attention layer, and a similarity network. The proposed model learns degrees of associations between queries and text passages through dot products between phrases that constitute questions and passages. In experiments with the MS-MARCO dataset, the proposed model demonstrated higher mean reciprocal ranks (MRRs), 0.8%p–13.2%p, than most of the previous models, except for the models based on BERT (a pre-trained language model). Although the proposed model demonstrated lower MRRs than the BERT-based models, it was approximately 8 times lighter and 3.7 times faster than the BERT-based models.

In any ranking system, the retrieval model outputs a single score for a\ndocument based on its belief on how relevant it is to a given search query.\nWhile retrieval models have continued to improve with the introduction of\nincreasingly complex architectures, few works have investigated a retrieval\nmodel's belief in the score beyond the scope of a single value. We argue that\ncapturing the model's uncertainty with respect to its own scoring of a document\nis a critical aspect of retrieval that allows for greater use of current models\nacross new document distributions, collections, or even improving effectiveness\nfor down-stream tasks. In this paper, we address this problem via an efficient\nBayesian framework for retrieval models which captures the model's belief in\nthe relevance score through a stochastic process while adding only negligible\ncomputational overhead. We evaluate this belief via a ranking based calibration\nmetric showing that our approximate Bayesian framework significantly improves a\nretrieval model's ranking effectiveness through a risk aware reranking as well\nas its confidence calibration. Lastly, we demonstrate that this additional\nuncertainty information is actionable and reliable on down-stream tasks\nrepresented via cutoff prediction.\n

BERT-LBIA: A BERT-Based Late Bidirectional Interaction Attention Model for Legal Case Retrieval

Dense passage retrieval (DPR) models show great effectiveness gains in first stage retrieval for the web domain. However in the web domain we are in a setting with large amounts of training data and a query-to-passage or a query-to-document retrieval task. We investigate in this paper dense document-to-document retrieval with limited labelled target data for training, in particular legal case retrieval. In order to use DPR models for document-to-document retrieval, we propose a Paragraph Aggregation Retrieval Model (PARM) which liberates DPR models from their limited input length. PARM retrieves documents on the paragraph-level: for each query paragraph, relevant documents are retrieved based on their paragraphs. Then the relevant results per query paragraph are aggregated into one ranked list for the whole query document. For the aggregation we propose vector-based aggregation with reciprocal rank fusion (VRRF) weighting, which combines the advantages of rank-based aggregation and topical aggregation based on the dense embeddings. Experimental results show that VRRF outperforms rank-based aggregation strategies for dense document-to-document retrieval with PARM. We compare PARM to document-level retrieval and demonstrate higher retrieval effectiveness of PARM for lexical and dense first-stage retrieval on two different legal case retrieval collections. We investigate how to train the dense retrieval model for PARM on limited target data with labels on the paragraph or the document-level. In addition, we analyze the differences of the retrieved results of lexical and dense retrieval with PARM.

We study the utility of the lexical translation model (IBM Model 1) for English text retrieval, in particular, its neural variants that are trained end-to-end. We use the neural Model1 as an aggregator layer applied to context-free or contextualized query/document embeddings. This new approach to design a neural ranking system has benefits for effectiveness, efficiency, and interpretability. Specifically, we show that adding an interpretable neural Model 1 layer on top of BERT-based contextualized embeddings (1) does not decrease accuracy and/or efficiency; and (2) may overcome the limitation on the maximum sequence length of existing BERT models. The context-free neural Model 1 is less effective than a BERT-based ranking model, but it can run efficiently on a CPU (without expensive index-time precomputation or query-time operations on large tensors). Using Model 1 we produced best neural and non-neural runs on the MS MARCO document ranking leaderboard in late 2020.

Information Retrieval is a domain of transforming unstructured data to structured data by various strategies. These strategies are heuristic in nature. It is an implementation of data mining and warehousing. Probabilistic and Statistical Language Models already exist in the field of IR. This paper describes a new blend of retrieval model to the past concepts of information retrieval as the vector space information retrieval model, probabilistic retrieval model, and last one as statistical model. Our novel framework is centric around the Sampling Distribution of statistics. We are emphasizing on Statistical Model and Probabilistic Model that is basically Statistical cum Probabilistic Model. This work shows the presence of effective novel retrieval algorithms that use the common terms of statistical computation using expectation, mean, and variance. By using the concept of Sampling Distribution, we categorize our novel algorithm as ad hoc and filtering approaches. This paper deals with summarized statistical calculations and suggests a new model for relevancy and ranking of documents.

Due to high annotation costs making the best use of existing human-created\ntraining data is an important research direction. We, therefore, carry out a\nsystematic evaluation of transferability of BERT-based neural ranking models\nacross five English datasets. Previous studies focused primarily on zero-shot\nand few-shot transfer from a large dataset to a dataset with a small number of\nqueries. In contrast, each of our collections has a substantial number of\nqueries, which enables a full-shot evaluation mode and improves reliability of\nour results. Furthermore, since source datasets licences often prohibit\ncommercial use, we compare transfer learning to training on pseudo-labels\ngenerated by a BM25 scorer. We find that training on pseudo-labels -- possibly\nwith subsequent fine-tuning using a modest number of annotated queries -- can\nproduce a competitive or better model compared to transfer learning. Yet, it is\nnecessary to improve the stability and/or effectiveness of the few-shot\ntraining, which, sometimes, can degrade performance of a pretrained model.\n

Over the past two years, large pretrained language models such as BERT have been applied to text ranking problems and showed superior performance on multiple public benchmark data sets. Prior work demonstrated that an ensemble of multiple BERT-based ranking models can not only boost the performance, but also reduce the performance variance. However, an ensemble of models is more costly because it needs computing resource and/or inference time proportional to the number of models. In this paper, we study how to retain the performance of an ensemble of models at the inference cost of a single model by distilling the ensemble into a single BERT-based student ranking model. Specifically, we study different designs of teacher labels, various distillation strategies, as well as multiple distillation losses tailored for ranking problems. We conduct experiments on the MS MARCO passage ranking and the TREC-COVID data set. Our results show that even with these simple distillation techniques, the distilled model can effectively retain the performance gain of the ensemble of multiple models. More interestingly, the performances of distilled models are also more stable than models fine-tuned on original labeled data. The results reveal a promising direction to capitalize on the gains achieved by an ensemble of BERT-based ranking models.

Recently, neural networks have been successfully employed to improve upon\nstate-of-the-art performance in ad-hoc retrieval tasks via machine-learned\nranking functions. While neural retrieval models grow in complexity and impact,\nlittle is understood about their correspondence with well-studied IR\nprinciples. Recent work on interpretability in machine learning has provided\ntools and techniques to understand neural models in general, yet there has been\nlittle progress towards explaining ranking models.\n We investigate whether one can explain the behavior of neural ranking models\nin terms of their congruence with well understood principles of document\nranking by using established theories from axiomatic IR. Axiomatic analysis of\ninformation retrieval models has formalized a set of constraints on ranking\ndecisions that reasonable retrieval models should fulfill. We operationalize\nthis axiomatic thinking to reproduce rankings based on combinations of\nelementary constraints. This allows us to investigate to what extent the\nranking decisions of neural rankers can be explained in terms of retrieval\naxioms, and which axioms apply in which situations. Our experimental study\nconsiders a comprehensive set of axioms over several representative neural\nrankers. While the existing axioms can already explain the particularly\nconfident ranking decisions rather well, future work should extend the axiom\nset to also cover the other still "unexplainable" neural IR rank decisions.\n

Question answering (QA) systems for large document collections typically use pipelines that (i) retrieve possibly relevant documents, (ii) re-rank them, (iii) rank paragraphs or other snippets of the top-ranked documents, and (iv) select spans of the top-ranked snippets as exact answers. Pipelines are conceptually simple, but errors propagate from one component to the next, without later components being able to revise earlier decisions. We present an architecture for joint document and snippet ranking, the two middle stages, which leverages the intuition that relevant documents have good snippets and good snippets come from relevant documents. The architecture is general and can be used with any neural text relevance ranker. We experiment with two main instantiations of the architecture, based on POSIT-DRMM (PDRMM) and a BERT-based ranker. Experiments on biomedical data from BIOASQ show that our joint models vastly outperform the pipelines in snippet retrieval, the main goal for QA, with fewer trainable parameters, also remaining competitive in document retrieval. Furthermore, our joint PDRMM-based model is competitive with BERT-based models, despite using orders of magnitude fewer parameters. These claims are also supported by human evaluation on two test batches of BIOASQ. To test our key findings on another dataset, we modified the Natural Questions dataset so that it can also be used for document and snippet retrieval. Our joint PDRMM-based model again outperforms the corresponding pipeline in snippet retrieval on the modified Natural Questions dataset, even though it performs worse than the pipeline in document retrieval. We make our code and the modified Natural Questions dataset publicly available.