Natural Language Processing Research Research Articles

Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AI

While many Natural Language Processing (NLP) techniques have been proposed for fact-checking, both academic research and fact-checking organizations report limited adoption of such NLP work due to poor alignment with fact-checker practices, values, and needs. To address this, we investigate a co-design method, Matchmaking for AI, to enable fact-checkers, designers, and NLP researchers to collaboratively identify what fact-checker needs should be addressed by technology, and to brainstorm ideas for potential solutions. Co-design sessions we conducted with 22 professional fact-checkers yielded a set of 11 design ideas that offer a "north star'', integrating fact-checker criteria into novel NLP design concepts. These concepts range from pre-bunking misinformation, efficient and personalized monitoring misinformation, proactively reducing fact-checker potential biases, and collaborative writing fact-check reports. Our work provides new insights into both human-centered fact-checking research and practice and AI co-design research.

Automatic authorship attribution in Albanian texts.

Automatic authorship identification is a challenging task that has been the focus of extensive research in natural language processing. Regardless of the progress made in attributing authorship, the need for corpora in under-resourced languages impedes advancing and examining present methods. To address this gap, we investigate the problem of authorship attribution in Albanian. We introduce a newly compiled corpus of Albanian newsroom columns and literary works and analyze machine-learning methods for detecting authorship. We create a set of hand-crafted features targeting various categories (lexical, morphological, and structural) relevant to Albanian and experiment with multiple classifiers using two different multiclass classification strategies. Furthermore, we compare our results to those obtained using deep learning models. Our investigation focuses on identifying the best combination of features and classification methods. The results reveal that lexical features are the most effective set of linguistic features, significantly improving the performance of various algorithms in the authorship attribution task. Among the machine learning algorithms evaluated, XGBoost demonstrated the best overall performance, achieving an F1 score of 0.982 on literary works and 0.905 on newsroom columns. Additionally, deep learning models such as fastText and BERT-multilingual showed promising results, highlighting their potential applicability in specific scenarios in Albanian writings. These findings contribute to the understanding of effective methods for authorship attribution in low-resource languages and provide a robust framework for future research in this area. The careful analysis of the different scenarios and the conclusions drawn from the results provide valuable insights into the potential and limitations of the methods and highlight the challenges in detecting authorship in Albanian. Promising results are reported, with implications for improving the methods used in Albanian authorship attribution. This study provides a valuable resource for future research and a reference for researchers in this domain.

A knowledge-Aware NLP-Driven conversational model to detect deceptive contents on social media posts

The widespread dissemination of deceptive content on social media presents a substantial challenge to preserving authenticity and trust. The epidemic growth of false news is due to the greater use of social media to transmit news, rather than conventional mass media such as newspapers, magazines, radio, and television. Humans' incapacity to differentiate among true and false facts exposes fake news as a threat to logical truth, democracy, journalism, and government credibility. Using combination of advanced methodologies, Deep learning (DL) methods, and Natural Language Processing (NLP) approaches, researchers and technology developers attempt to make robust systems proficient in discerning the subtle nuances that betray deceptive intent. Analysing conversational linguistic patterns of misleading data, these techniques’ purpose to progress the resilience of social platforms against the spread of deceptive content, eventually contributing to an additional informed and trustworthy online platform. This paper proposed a Knowledge-Aware NLP-Driven AlBiruni Earth Radius Optimization Algorithm with Deep Learning Tool for Enhanced Deceptive Content Detection (BER-DLEDCD) algorithm on Social Media. The purpose of the BER-DLEDCD system is to identify and classify the existence of deceptive content utilizing NLP with optimal DL model. In the BER-DLEDCD technique, data pre-processing takes place to change the input data into compatible format. Furthermore, the BER-DLEDCD approach applies hybrid DL technique encompassing Convolutional Neural Network with Long Short-Term Memory (CNN-LSTM) methodology for deceptive content detection. Moreover, the BER approach has been deployed to boost hyperparameter choice of the CNN-LSTM technique which leads to enhanced detection performance. The simulation outcome of the BER-DLEDCD system has been examined employing benchmark database. The extensive outcomes stated the BER-DLEDCD system achieved excellent performance with the accuracy of 94 %, 94.83 % precision, 94.30 % F-score with other recent approaches.

Rule-based fine-grained vocabulary for enhanced Chinese pre-training

Abstract Chinese pre-training is an essential direction in Chinese natural language processing, and vocabulary constitutes the foundation of pre-trained models. Existing methods for constructing vocabularies for Chinese pre-trained models typically treat each Chinese character as an indivisible token, overlooking the additional information embedded in the intrinsic structure of Chinese characters and its impact on model performance. To leverage this information, we propose a method of training with rule-based fine-grained vocabularies to directly learn the sequence of intrinsic structures of Chinese characters in Chinese pre-training. Specifically, we first construct a mapping rule-based fine-grained vocabulary based on the glyph and radical splitting mapping relationship of Chinese characters. Subsequently, we employ a whole char masking strategy to train Chinese pre-trained models based on this new vocabulary. Experimental results demonstrate that compared to the BERT baseline model, our model achieves promising performance on multiple downstream tasks, with fewer model parameters and stronger robustness. The proposed Chinese pre-training method exploits the intrinsic structural information of Chinese characters, providing a novel method for subsequent research in Chinese natural language processing.

Joint Entity and Relation Extraction Using Machine Reading Comprehension for Urdu

Joint Entity and Relation Extraction (JERE) plays an important role in natural language processing (NLP) by identifying names, locations, and the relationships among them from unstructured text. Despite extensive research in languages like English, JERE poses significant challenges in low-resource languages, particularly Urdu, due to limited annotated da-ta and inherent linguistic complexities. In this paper, we propose a novel Machine Reading Comprehension (MRC)-based approach that effectively addresses the JERE task for Urdu, integrating a text encoder and a question-answering module that work synergistically to enhance entity and relationship extraction. We introduce an annotated Urdu JERE dataset and demonstrate how our methodology will significantly contribute to multilingual NLP efforts. We propose an innovative Machine Reading Comprehension (MRC)-based method to tackle JERE in Urdu. This method has two main components: a text encoder and a question answering (QA) module. The text encoder converts Urdu text into a compact vector form, which is then fed into the QA module. The QA module generates answers to queries regarding the desired entities and relationships, producing a sequence of tokens that represent these entities and their interactions. The model is trained to minimize the difference between its predicted answers and the correct ones. Our approach, along with the introduction of an annotated Urdu JERE dataset, significantly advances multilingual NLP and information ex-traction research. The insights gained can be applied to other low-resource languages, aiding in the development of NLP tools and applications for a broader array of languages.

Extracting Features from Text Flows based on Semantic Similarity for Text Classification: an Approach Inspired by Audio Analysis

Text classification is a mainly investigated challenge in Natural Language Processing (NLP) research. The higher performance of a classification model depends on a representation that can extract valuable information about the texts. Aiming not to lose crucial local text information, a way to represent texts is through flows, sequences of information collected from texts. This paper proposes an approach that combines various techniques to represent texts: the representation by flows, the benefit of the word embeddings text representation associated with lexicon information via semantic similarity distances, and the extraction of features inspired by well-established audio analysis features.In order to perform text classification, this approach splits the text into sentences and calculates a semantic similarity metric to a lexicon on an embedding vector space. The sequence of semantic similarity metrics composes the text flow. Then, the method performs the extraction of twenty-five features inspired by audio analysis (named Audio-Like Features). The features adaptation from audio analysis comes from a similitude between a text flow and a digital signal, in addition to the existing relationship between text, speech, and audio. We evaluated the method in three NLP classification tasks: Fake News Detection in English, Fake News Detection in Portuguese, and Newspaper Columns versus News Classification. The approach efficacy is compared to baselines that embed semantics in text representation: the Paragraph Vector and the BERT. The objective of the experiments was to investigate if the proposed approach could compete with the baselines methods improve their efficacy when associated with them. The experimental evaluation demonstrates that the association between the proposed and the baseline methods can enhance the baseline classification efficacy in all three scenarios. In the Fake News Detection in Portuguese task, our approach surpassed the baselines and obtained the best effectiveness (PR-AUC = 0.98).

Graphic association learning: Multimodal feature extraction and fusion of image and text using artificial intelligence techniques

With the advancement of technology in recent years, the application of artificial intelligence in real life has become more extensive. Graphic recognition is a hot spot in the current research of related technologies. It involves machines extracting key information from pictures and combining it with natural language processing for in-depth understanding. Existing methods still have obvious deficiencies in fine-grained recognition and deep understanding of contextual context. Addressing these issues to achieve high-quality image-text recognition is crucial for various application scenarios, such as accessibility technologies, content creation, and virtual assistants. To tackle this challenge, a novel approach is proposed that combines the Mask R-CNN, DCGAN, and ALBERT models. Specifically, the Mask R-CNN specializes in high-precision image recognition and segmentation, the DCGAN captures and generates nuanced features from images, and the ALBERT model is responsible for deep natural language processing and semantic understanding of this visual information. Experimental results clearly validate the superiority of this method. Compared to traditional image-text recognition techniques, the recognition accuracy is improved from 85.3% to 92.5%, and performance in contextual and situational understanding is enhanced. The advancement of this technology has far-reaching implications for research in machine vision and natural language processing and open new possibilities for practical applications.

From admission to discharge: a systematic review of clinical natural language processing along the patient journey

BackgroundMedical text, as part of an electronic health record, is an essential information source in healthcare. Although natural language processing (NLP) techniques for medical text are developing fast, successful transfer into clinical practice has been rare. Especially the hospital domain offers great potential while facing several challenges including many documents per patient, multiple departments and complex interrelated processes.MethodsIn this work, we survey relevant literature to identify and classify approaches which exploit NLP in the clinical context. Our contribution involves a systematic mapping of related research onto a prototypical patient journey in the hospital, along which medical documents are created, processed and consumed by hospital staff and patients themselves. Specifically, we reviewed which dataset types, dataset languages, model architectures and tasks are researched in current clinical NLP research. Additionally, we extract and analyze major obstacles during development and implementation. We discuss options to address them and argue for a focus on bias mitigation and model explainability.ResultsWhile a patient’s hospital journey produces a significant amount of structured and unstructured documents, certain steps and documents receive more research attention than others. Diagnosis, Admission and Discharge are clinical patient steps that are researched often across the surveyed paper. In contrast, our findings reveal significant under-researched areas such as Treatment, Billing, After Care, and Smart Home. Leveraging NLP in these stages can greatly enhance clinical decision-making and patient outcomes. Additionally, clinical NLP models are mostly based on radiology reports, discharge letters and admission notes, even though we have shown that many other documents are produced throughout the patient journey. There is a significant opportunity in analyzing a wider range of medical documents produced throughout the patient journey to improve the applicability and impact of NLP in healthcare.ConclusionsOur findings suggest that there is a significant opportunity to leverage NLP approaches to advance clinical decision-making systems, as there remains a considerable understudied potential for the analysis of patient journey data.

Text Summarization in the Shona Language using Natural Language Processing

The rise of digital information in many languages, including Shona, highlights the significance of developing effective text summarizing techniques to promote information accessibility and usability. This work fills a major gap in the natural language processing (NLP) for the Shona language, which is widely spoken in Zimbabwe and its surrounding areas but has received little attention. The lack of pre-trained language models specifically designed for Shona, the intricacy of Shona's morphology, and the scarcity of annotated datasets provide the main obstacles to Shona text summarization.[1] The goal of this research is to create and modify contemporary machine learning methods for efficient Shona text summarizing in order to address these issues. By gathering and analyzing texts from a variety of sources, such as news stories, scholarly papers, and social media, we produced large annotated corpora. These datasets were utilized to fine-tune existing NLP models, such as Transformer-based architectures, ensuring they account for Shona’s specific language traits. To address the morphological and syntactic complexities of Shona, our solution combines statistical and rule-based techniques. When compared to baseline methods, the results show a significant improvement in the relevancy and accuracy of Shona text summaries. In addition to serving as a starting point for further NLP research in underrepresented languages, the generated models help Shona-speaking people in the areas of business, education, and media. By encouraging inclusivity and linguistic variety, showcasing the possibility for cross- lingual breakthroughs, and emphasizing the ethical development of technology, this research adds to the larger area of NLP.

Text Classification Based on Natural Language Processing and Machine Learning in Multi-Label Corpus

The rapid development of the Internet has led to a geometric expansion of text information resources online. Among them, corpus, as the basic data source of natural language processing based on statistical language model, its construction and application have become a hot issue in current language processing research. After consulting a large number of relevant literature and materials, it was found that many researchers have provided new ideas for multi label corpus text classification methods. However, this article adds its own understanding and takes this as the direction and basis. In the introduction, the research significance of text classification was introduced, and then academic research and analysis were carried out on the two key sentences of corpus text classification and natural language processing in multi-tag corpus text classification. This article then utilizes an algorithm model to provide a theoretical basis for the study of multi-label corpus text classification methods; At the end of this article, a simulation comparative experiment is conducted, and the experiment is summarized and discussed; In the Enterprise L corpus, the difference in recall rates before and after the use of Entrance 1 was 5.5%, the difference in recall rates before and after the use of Entrance 2 was 7.8%, the difference in recall rates before and after the use of Entrance 3 was 3.3%, and the difference in recall rates before and after the use of Entrance 4 was 4.5%. At the same time, with the continuous research of natural language processing and machine learning, the research on text classification methods of multi tag corpus is also facing new opportunities and challenges.

Exploring Thematic Diversity in Classical Chinese Poetry: A Novel Dataset and a BERT-enhanced Ensemble Learning Approach

Classical Chinese poetry, as an essential aspect of cultural heritage, exhibits rich theme diversity often overlooked in natural language processing research. To address this gap, we aim to explore the classification of thematic categories within this literary domain. We curate a dataset of 2,918 annotated poems spanning seven common themes and propose a BERT-based ensemble learning approach for effective classification. Although this method integrates existing models, it achieves an accuracy and F1 score of over 72% in the 7-class task, surpassing established baselines, and providing a baseline for future research. The experimental findings reveal the effectiveness of ensemble strategies in improving individual base model performance and highlight the potential of the MLP-based ensemble technique. The study contributes to a deeper understanding of thematic categories and textual features in classical Chinese poetry, and offers an automated classification system for classical Chinese poems.

Natural language processing for detecting adverse drug events: A systematic review protocol

Background Detecting Adverse Drug Events (ADEs) is an emerging research area, attracting great interest in the research community. Better anticipatory management of predisposing factors has considerable potential to improve outcomes. Automatic extraction of ADEs using Natural Language Processing (NLP) has a great potential to significantly facilitate efficient and effective distillation of such knowledge, to better understand and predict risk of adverse events. Methods This systematic review follows the six-stage including the literature from 6 databases (Embase, Medline, Web Of Science Core Collection, ACM Guide to Computing Literature, IEEE Digital Library and Scopus). Following the title, abstract and full-text screenings, characteristics and main findings of the included studies and resources will be tabulated and summarized. The risk of bias and reporting quality was assessed using the PROBAST tool. Results We developed our search strategy and collected all relevant publications. As of July 2024, we have completed all the stages of the systematic review. We identified 178 studies for inclusion through the academic literature search (where data was extracted from all of the papers). Right now, we are writing up the systematic review paper where we are synthesising the different findings. Further refinement of the eligibility criteria and data extraction has been ongoing since August 2022. Conclusion In this systematic review, we will identify and consolidate information and evidence related to the use and effectiveness of existing NLP approaches and tools for automatically detecting ADEs from free text (discharge summaries, General Practitioner notes, social media, etc.). Our findings will improve the understanding of the current landscape of the use of NLP for extracting ADEs. It will lead to better anticipatory management of predisposing factors with the potential to improve outcomes considerably. Our results will also be valuable both to NLP researchers developing methods to extract ADEs and to translational/clinical researchers who use NLP for this purpose and in healthcare in general. For example, from our initial analysis of the studies, we can conclude that the majority of the proposed works are about the detection (extraction) of ADEs from text. An important portion of studies also focus on the binary classification of text (for highlighting if it includes or not ADEs). Different challenges related to the unbalanced dataset, abbreviations and acronyms but also to the lower results with rare ADEs were also mentioned by the studied papers.

Reversal of the Word Sense Disambiguation Task Using a Deep Learning Model

Word sense disambiguation (WSD) remains a persistent challenge in the natural language processing (NLP) community. While various NLP packages exist, the Lesk algorithm in the NLTK library demonstrates suboptimal accuracy. In this research article, we propose an innovative methodology and an open-source framework that effectively addresses the challenges of WSD by optimizing memory usage without compromising accuracy. Our system seamlessly integrates WSD into NLP tasks, offering functionality similar to that provided by the NLTK library. However, we go beyond the existing approaches by introducing a novel idea related to WSD. Specifically, we leverage deep neural networks and consider the language patterns learned by these models as the new gold standard. This approach suggests modifying existing semantic dictionaries, such as WordNet, to align with these patterns. Empirical validation through a series of experiments confirmed the effectiveness of our proposed method, achieving state-of-the-art performance across multiple WSD datasets. Notably, our system does not require the installation of additional software beyond the well-known Python libraries. The classification model is saved in a readily usable text format, and the entire framework (model and data) is publicly available on GitHub for the NLP research community.

The Role of Typological Feature Prediction in NLP and Linguistics

Abstract Computational typology has gained traction in the field of Natural Language Processing (NLP) in recent years, as evidenced by the increasing number of papers on the topic and the establishment of a Special Interest Group on the topic (SIGTYP), including the organization of successful workshops and shared tasks. A considerable amount of work in this sub-field is concerned with prediction of typological features, for example, for databases such as the World Atlas of Language Structures (WALS) or Grambank. Prediction is argued to be useful either because (1) it allows for obtaining feature values for relatively undocumented languages, alleviating the sparseness in WALS, in turn argued to be useful for both NLP and linguistics; and (2) it allows us to probe models to see whether or not these typological features are encapsulated in, for example, language representations. In this article, we present a critical stance concerning prediction of typological features, investigating to what extent this line of research is aligned with purported needs—both from the perspective of NLP practitioners, and perhaps more importantly, from the perspective of linguists specialized in typology and language documentation. We provide evidence that this line of research in its current state suffers from a lack of interdisciplinary alignment. Based on an extensive survey of the linguistic typology community, we present concrete recommendations for future research in order to improve this alignment between linguists and NLP researchers, beyond the scope of typological feature prediction.

Research on the application of deep learning-based BERT model in sentiment analysis

With the rapid expansion of the Internet and social media, there has been an explosion of text data, containing valuable emotional insights. Sentiment analysis, aimed at discerning and processing people's emotional inclinations, opinions, and perspectives, has emerged as a pivotal area within natural language processing (NLP). This research explores the application of Deep Learning-based BERT models, particularly DistilBERT, in sentiment analysis. It highlights the importance of sentiment analysis in understanding emotional insights from vast text data and introduces BERT models as revolutionary methodologies for enhancing accuracy and efficiency. The study conducts experiments using the SST2 dataset, showcasing the effectiveness of DistilBERT in sentiment classification tasks. Overall, it underscores the transformative potential of BERT models in revolutionizing sentiment analysis methodologies and driving advancements in natural language processing research.

Research on the application of deep learning-based BERT model in sentiment analysis

With the rapid expansion of the Internet and social media, there has been an explosion of text data, containing valuable emotional insights. Sentiment analysis, aimed at discerning and processing people's emotional inclinations, opinions, and perspectives, has emerged as a pivotal area within natural language processing (NLP). This research explores the application of Deep Learning-based BERT models, particularly DistilBERT, in sentiment analysis. It highlights the importance of sentiment analysis in understanding emotional insights from vast text data and introduces BERT models as revolutionary methodologies for enhancing accuracy and efficiency. The study conducts experiments using the SST2 dataset, showcasing the effectiveness of DistilBERT in sentiment classification tasks. Overall, it underscores the transformative potential of BERT models in revolutionizing sentiment analysis methodologies and driving advancements in natural language processing research.

Verbal behavior and the future of social science.

Natural language processing (NLP)-previously the domain of a select few language and computer scientists-is undergoing an unprecedented surge in popularity across disciplines. The ubiquity of language data, alongside extremely rapid methodological innovations, has magnetized the field, attracting researchers with the promise of measuring, forecasting, and understanding the most central questions in business, psychology, biology, sociology, the humanities, and beyond. The power of language analysis to reveal insights into human thought, feeling, and behavior has become a core interest emerging from recent technological advances, which are being probed to unearth deeply embedded truths about the human condition. However, NLP research has reached a critical juncture, sitting at the cusp of societal transformation in many aspects of daily life. The details of how NLP research develops over the next 3-5 years will define this transformation. In this emerging, near-infinite space of NLP-driven research, we provide a critical frame of reference for how, when, and why these technologies should evolve in a particularly transdisciplinary manner. Specifically, we discuss (a) the urgency of pairing existing and emerging NLP research with existing scientific knowledge, theory, and principles from the behavioral sciences; (b) the coevolution of NLP technologies; and (c) the practical implications and ethical consequences of expanding language analysis using broader psychosocial theories of the human condition. While our discussion focuses principally on using language as a window in the individual mind, this topic holds substantial implications for other disciplines and lines of inquiry, including the dynamics of social interaction and beyond. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Online anti-immigrant discourse in Germany: ethnographically backed analysis of user comments

This article investigates discourse- and language-specific features of online anti-immigrant extreme speech in Germany. We analyze a context rich dataset collected and annotated through a collaborative effort involving fact-checkers, ethnographers and natural language processing (NLP) researchers. Using a bottom-up annotation scheme, we capture the nuances of the discourse and develop a typology of lexical innovations. The analysis combines thematic and critical discourse analysis with a linguistic perspective, revealing that direct forms of racism intertwine with argumentative forms of antagonism and playful word games within anti-immigrant discourses, in ways that center around narratives of victimhood and perceived threats from migrants. We further show how the specific conditions in Germany, including strict legal regulations of speech, have shaped the emergence of a non-standard language variety in and through anti-immigrant discourse, which helps the discourse community maintain group identity. The ethnographically backed analysis provides a granular understanding of the phenomenon of online extreme speech in the German context, contributing to the broader field of discourse studies and offering insights into the varieties within anti-immigrant discourse.

Enhancing Arabic Dialect Detection on Social Media: A Hybrid Model with an Attention Mechanism

Recently, the widespread use of social media and easy access to the Internet have brought about a significant transformation in the type of textual data available on the Web. This change is particularly evident in Arabic language usage, as the growing number of users from diverse domains has led to a considerable influx of Arabic text in various dialects, each characterized by differences in morphology, syntax, vocabulary, and pronunciation. Consequently, researchers in language recognition and natural language processing have become increasingly interested in identifying Arabic dialects. Numerous methods have been proposed to recognize this informal data, owing to its crucial implications for several applications, such as sentiment analysis, topic modeling, text summarization, and machine translation. However, Arabic dialect identification is a significant challenge due to the vast diversity of the Arabic language in its dialects. This study introduces a novel hybrid machine and deep learning model, incorporating an attention mechanism for detecting and classifying Arabic dialects. Several experiments were conducted using a novel dataset that collected information from user-generated comments from Twitter of Arabic dialects, namely, Egyptian, Gulf, Jordanian, and Yemeni, to evaluate the effectiveness of the proposed model. The dataset comprises 34,905 rows extracted from Twitter, representing an unbalanced data distribution. The data annotation was performed by native speakers proficient in each dialect. The results demonstrate that the proposed model outperforms the performance of long short-term memory, bidirectional long short-term memory, and logistic regression models in dialect classification using different word representations as follows: term frequency-inverse document frequency, Word2Vec, and global vector for word representation.

An Extensible Evaluation Framework Applied to Clinical Text Deidentification Natural Language Processing Tools: Multisystem and Multicorpus Study.

Clinical natural language processing (NLP) researchers need access to directly comparable evaluation results for applications such as text deidentification across a range of corpus types and the means to easily test new systems or corpora within the same framework. Current systems, reported metrics, and the personally identifiable information (PII) categories evaluated are not easily comparable. This study presents an open-source and extensible end-to-end framework for comparing clinical NLP system performance across corpora even when the annotation categories do not align. As a use case for this framework, we use 6 off-the-shelf text deidentification systems (ie, CliniDeID, deid from PhysioNet, MITRE Identity Scrubber Toolkit [MIST], NeuroNER, National Library of Medicine [NLM] Scrubber, and Philter) across 3 standard clinical text corpora for the task (2 of which are publicly available) and 1 private corpus (all in English), with annotation categories that are not directly analogous. The framework is built on shell scripts that can be extended to include new systems, corpora, and performance metrics. We present this open tool, multiple means for aligning PII categories during evaluation, and our initial timing and performance metric findings. Code for running this framework with all settings needed to run all pairs are available via Codeberg and GitHub. From this case study, we found large differences in processing speed between systems. The fastest system (ie, MIST) processed an average of 24.57 (SD 26.23) notes per second, while the slowest (ie, CliniDeID) processed an average of 1.00 notes per second. No system uniformly outperformed the others at identifying PII across corpora and categories. Instead, a rich tapestry of performance trade-offs emerged for PII categories. CliniDeID and Philter prioritize recall over precision (with an average recall 6.9 and 11.2 points higher, respectively, for partially matching spans of text matching any PII category), while the other 4 systems consistently have higher precision (with MIST's precision scoring 20.2 points higher, NLM Scrubber scoring 4.4 points higher, NeuroNER scoring 7.2 points higher, and deid scoring 17.1 points higher). The macroaverage recall across corpora for identifying names, one of the more sensitive PII categories, included deid (48.8%) and MIST (66.9%) at the low end and NeuroNER (84.1%), NLM Scrubber (88.1%), and CliniDeID (95.9%) at the high end. A variety of metrics across categories and corpora are reported with a wider variety (eg, F2-score) available via the tool. NLP systems in general and deidentification systems and corpora in our use case tend to be evaluated in stand-alone research articles that only include a limited set of comparators. We hold that a single evaluation pipeline across multiple systems and corpora allows for more nuanced comparisons. Our open pipeline should reduce barriers to evaluation and system advancement.