Named Entity Recognition Research Articles

Chinese EMR Named Entity Recognition Using Fused Label Relations Based on Machine Reading Comprehension Framework.

Chinese electronic medical record (EMR) presents significant challenges for named entity recognition (NER) due to their specialized nature, unique language features, and diverse expressions. Traditionally, NER is treated as a sequence labeling task, where each token is assigned a label. Recent research has reframed NER within the machine reading comprehension (MRC) framework, extracting entities in a question-answer format, achieving state-of-the-art performance. However, these MRC-based methods have a significant limitation: they extract entities of various types independently, ignoring their interrelations. To address this, we introduce the Fusion Label Relations with MRC (FLR-MRC) model, which enhances the MRC model by implicitly capturing dependencies among entity types. FLR-MRC models interrelations between labels using graph attention networks, integrating these with textual data to identify entities. On the benchmark CMeEE and CCKS2017-CNER datasets, FLR-MRC achieves F1-scores of 0.6652 and 0.9101, respectively, outperforming existing clinical NER methods.

Improving dictionary-based named entity recognition with deep learning.

Dictionary-based named entity recognition (NER) allows terms to be detected in a corpus and normalized to biomedical databases and ontologies. However, adaptation to different entity types requires new high-quality dictionaries and associated lists of blocked names for each type. The latter are so far created by identifying cases that cause many false positives through manual inspection of individual names, a process that scales poorly. In this work, we aim to improve block list s by automatically identifying names to block, based on the context in which they appear. By comparing results of three well-established biomedical NER methods, we generated a dataset of over 12.5 million text spans where the methods agree on the boundaries and type of entity tagged. These were used to generate positive and negative examples of contexts for four entity types (genes, diseases, species, and chemicals), which were used to train a Transformer-based model (BioBERT) to perform entity type classification. Application of the best model (F1-score = 96.7%) allowed us to generate a list of problematic names that should be blocked. Introducing this into our system doubled the size of the previous list of corpus-wide blocked names. In addition, we generated a document-specific list that allows ambiguous names to be blocked in specific documents. These changes boosted text mining precision by ∼5.5% on average, and over 8.5% for chemical and 7.5% for gene names, positively affecting several biological databases utilizing this NER system, like the STRING database, with only a minor drop in recall (0.6%). All resources are available through Zenodo https://doi.org/10.5281/zenodo.11243139 and GitHub https://doi.org/10.5281/zenodo.10289360.

Threat intelligence named entity recognition techniques based on few-shot learning

In today’s digital and internet era, threat intelligence analysis is of paramount importance to ensure network and information security. Named Entity Recognition (NER) is a fundamental task in natural language processing, aimed at identifying and extracting specific types of named entities from text, such as person names, locations, organization names, dates, times, currencies, and more. The quality of entities determines the effectiveness of upper-layer applications such as knowledge graphs. Recently, there has been a scarcity of training data in the threat intelligence field, and single models suffer from poor generalization ability. To address this, we propose a multi-view learning model, named the Few-shot Threat Intelligence Named Entity Recognition Model (FTM). We enhance the fusion method based on FTM, and further propose the FTM-GRU (Gate Recurrent Unit) model. The FTM model is based on the Tri-training algorithm to collaboratively train three few-shot NER models, leveraging the complementary nature of different model views to enable them to capture more threat intelligence domain knowledge at the coding level.FTM-GRU improves the fusion of multiple views. FTM-GRU uses the improved GRU model structure to control the memory and forgetting of view information, and introduces a relevance calculation unit to avoid redundancy of view information while highlighting important semantic features. We label and construct a few-shot Threat Intelligence Dataset (TID), and experiments on TID as well as the publicly available National Vulnerability Database (NVD) validate the effectiveness of our model for NER in the threat intelligence domain. Experimental results demonstrate that our proposed model achieves better recognition results in the task.

Joint Entity and Relation Extraction With Set Prediction Networks.

Joint entity and relation extraction is an important task in natural language processing, which aims to extract all relational triples mentioned in a given sentence. In essence, the relational triples mentioned in a sentence are in the form of a set, which has no intrinsic order between elements and exhibits the permutation invariant feature. However, previous seq2seq-based models require sorting the set of relational triples into a sequence beforehand with some heuristic global rules, which destroys the natural set structure. In order to break this bottleneck, we treat joint entity and relation extraction as a direct set prediction problem, so that the extraction model is not burdened with predicting the order of multiple triples. To solve this set prediction problem, we propose networks featured by transformers with non-autoregressive parallel decoding. In contrast to autoregressive approaches that generate triples one by one in a specific order, the proposed networks are able to directly output the final set of relational triples in one shot. Furthermore, we also design a set-based loss that forces unique predictions through bipartite matching. Compared with cross-entropy loss that highly penalizes small shifts in triple order, the proposed bipartite matching loss is invariant to any permutation of predictions; thus, it can provide the proposed networks with a more accurate training signal by ignoring triple order and focusing on relation types and entities. Various experiments on two benchmark datasets demonstrate that our proposed model significantly outperforms the current state-of-the-art (SoTA) models. Training code and trained models are now publicly available at https://github.com/DianboWork/SPN4RE.

Clinical named entity extraction for extracting information from medical data

Clinical named entity extraction (NER) based on deep learning gained much attention among researchers and data analysts. This paper proposes a NER approach to extract valuable Parkinson’s disease-related information. To develop an effective NER method and to handle problems in disease data analytics, a unique NER technique applies a “recognize-map-extract (RME)” mechanism and aims to deal with complex relationships present in the data. Due to the fast-growing medical data, there is a challenge in the development of suitable deep-learning methods for NER. Furthermore, the traditional machine learning approaches rely on the time-consuming process of creating corpora and cannot extract information for specific needs and locations in certain situations. This paper presents a clinical NER approach based on a convolutional neural network (CNN) for better use of specific features around medical entities and analyzes the performance of the proposed approach through fine-tuning NER with effective pre-training on the BC5CDR dataset. The proposed method uses annotation of entities for various medical concepts. The second stage develops a clinically NER method. This proposed method shows interesting results on the performance measures achieving a precision of 92.57%, recall of 92.22%, and F1- measure of 91.6%.

CarD-T: Interpreting Carcinomic Lexicon via Transformers.

The identification and classification of carcinogens is critical in cancer epidemiology, necessitating updated methodologies to manage the burgeoning biomedical literature. Current systems, like those run by the International Agency for Research on Cancer (IARC) and the National Toxicology Program (NTP), face challenges due to manual vetting and disparities in carcinogen classification spurred by the volume of emerging data. To address these issues, we introduced the Carcinogen Detection via Transformers (CarD-T) framework, a text analytics approach that combines transformer-based machine learning with probabilistic statistical analysis to efficiently nominate carcinogens from scientific texts. CarD-T uses Named Entity Recognition (NER) trained on PubMed abstracts featuring known carcinogens from IARC groups and includes a context classifier to enhance accuracy and manage computational demands. Using this method, journal publication data indexed with carcinogenicity & carcinogenesis Medical Subject Headings (MeSH) terms from the last 25 years was analyzed, identifying potential carcinogens. Training CarD-T on 60% of established carcinogens (Group 1 and 2A carcinogens, IARC designation), CarD-T correctly to identifies all of the remaining Group 1 and 2A designated carcinogens from the analyzed text. In addition, CarD-T nominates roughly 1500 more entities as potential carcinogens that have at least two publications citing evidence of carcinogenicity. Comparative assessment of CarD-T against GPT-4 model reveals a high recall (0.857 vs 0.705) and F1 score (0.875 vs 0.792), and comparable precision (0.894 vs 0.903). Additionally, CarD-T highlights 554 entities that show disputing evidence for carcinogenicity. These are further analyzed using Bayesian temporal Probabilistic Carcinogenic Denomination (PCarD) to provide probabilistic evaluations of their carcinogenic status based on evolving evidence. Our findings underscore that the CarD-T framework is not only robust and effective in identifying and nominating potential carcinogens within vast biomedical literature but also efficient on consumer GPUs. This integration of advanced NLP capabilities with vital epidemiological analysis significantly enhances the agility of public health responses to carcinogen identification, thereby setting a new benchmark for automated, scalable toxicological investigations.

Recognition of Chinese Electronic Medical Records for Rehabilitation Robots: Information Fusion Classification Strategy.

Named entity recognition is a critical task in the electronic medical record management system for rehabilitation robots. Handwritten documents often contain spelling errors and illegible handwriting, and healthcare professionals frequently use different terminologies. These issues adversely affect the robot's judgment and precise operations. Additionally, the same entity can have different meanings in various contexts, leading to category inconsistencies, which further increase the system's complexity. To address these challenges, a novel medical entity recognition algorithm for Chinese electronic medical records is developed to enhance the processing and understanding capabilities of rehabilitation robots for patient data. This algorithm is based on a fusion classification strategy. Specifically, a preprocessing strategy is proposed according to clinical medical knowledge, which includes redefining entities, removing outliers, and eliminating invalid characters. Subsequently, a medical entity recognition model is developed to identify Chinese electronic medical records, thereby enhancing the data analysis capabilities of rehabilitation robots. To extract semantic information, the ALBERT network is utilized, and BILSTM and MHA networks are combined to capture the dependency relationships between words, overcoming the problem of different meanings for the same entity in different contexts. The CRF network is employed to determine the boundaries of different entities. The research results indicate that the proposed model significantly enhances the recognition accuracy of electronic medical texts by rehabilitation robots, particularly in accurately identifying entities and handling terminology diversity and contextual differences. This model effectively addresses the key challenges faced by rehabilitation robots in processing Chinese electronic medical texts, and holds important theoretical and practical value.

3102: NSCLC Electronic Health Records Analysis by Named Entity Recognition in Italian Clinical Reports

3102: NSCLC Electronic Health Records Analysis by Named Entity Recognition in Italian Clinical Reports

KCB-FLAT: Enhancing Chinese Named Entity Recognition with Syntactic Information and Boundary Smoothing Techniques

Named entity recognition (NER) is a fundamental task in Natural Language Processing (NLP). During the training process, NER models suffer from over-confidence, and especially for the Chinese NER task, it involves word segmentation and introduces erroneous entity boundary segmentation, exacerbating over-confidence and reducing the model’s overall performance. These issues limit further enhancement of NER models. To tackle these problems, we proposes a new model named KCB-FLAT, designed to enhance Chinese NER performance by integrating enriched semantic information with the word-Boundary Smoothing technique. Particularly, we first extract various types of syntactic data and utilize a network named Key-Value Memory Network, based on syntactic information to functionalize this, integrating it through an attention mechanism to generate syntactic feature embeddings for Chinese characters. Subsequently, we employed an encoder named Cross-Transformer to thoroughly combine syntactic and lexical information to address the entity boundary segmentation errors caused by lexical information. Finally, we introduce a Boundary Smoothing module, combined with a regularity-conscious function, to capture the internal regularity of per entity, reducing the model’s overconfidence in entity probabilities through smoothing. Experimental results demonstrate that the proposed model achieves exceptional performance on the MSRA, Resume, Weibo, and self-built ZJ datasets, as verified by the F1 score.

Cross-lingual Natural Language Processing on Limited Annotated Case/Radiology Reports in English and Japanese: Insights from the Real-MedNLP Workshop.

Textual datasets (corpora) are crucial for the application of natural language processing (NLP) models. However, corpus creation in the medical field is challenging, primarily because of privacy issues with raw clinical data such as health records. Thus, the existing clinical corpora are generally small and scarce. Medical NLP (MedNLP) methodologies perform well with limited data availability. We present the outcomes of the Real-MedNLP workshop, which was conducted using limited and parallel medical corpora. Real-MedNLP exhibits three distinct characteristics: (1) Limited Annotated Documents: The training data comprises only a small set (approximately 100) of case reports (CRs) and radiology reports (RRs) that have been annotated. (2) Bilingually Parallel: The constructed corpora are parallel in Japanese and English. (3) Practical Tasks: The workshop addresses fundamental tasks, such as named entity recognition and applied practical tasks. We propose three tasks: named entity recognition (NER) of approximately 100 available documents (Task 1), NER based only on annotation guidelines for humans (Task 2), and clinical applications (Task 3) consisting of adverse drug effects (ADE) detection for CRs and identical case identification (CI) for RRs. Nine teams participated in this study. The best systems achieved 0.65 and 0.89 F1-scores for CRs and RRs in Task 1, whereas the top scores in Task 2 decreased by 50-70%. In Task 3, ADE reports were detected by up to 0.64 F1-score, and CI scored up to 0.96 binary accuracy. Most systems adopt medical-domain-specific pre-trained language models using data augmentation methods. Despite the challenge of limited corpus size in Tasks 1 and 2, recent approaches are promising because the partial match scores reached approximately 0.8-0.9 F1-scores. Task 3 applications revealed that the different availabilities of external language resources affected the performance per language.

Automated information extraction for behavioural interventions: evaluation and reflections on interdisciplinary AI development

Background The HBCP knowledge system identifies and extracts entities from randomised controlled trials of behaviour change interventions organised by a behaviour change intervention ontology (BCIO) to populate: 1) an outcome prediction tool; and 2) a research browser tool. This knowledge system requires automated information extraction algorithms to query and interpret evidence from behaviour change intervention (BCI) reports. This paper reports the results of an evaluation of the automated information extraction and reflects on the results in relation to the challenges of interdisciplinary working and collaboration. Methods The evaluation used a dataset of 117 previously unseen BCI reports to assess its performance. The automatically extracted information was compared to the full text PDF by trained annotators on essential BCIO entities required for the outcome prediction tool and research browser tool and whether the extracted information was assigned to the correct arm of the randomised trial. Essential entities were the outcome value, a selection of the most common Behaviour Change Techniques (BCTs), the mode of intervention delivery, and key population characteristics. Results The evaluation found an outcome value present in 53.85% (n=63) of the output from the information extraction system but it never extracted both the correct outcome values (interventions and control arms) and assigned them to the correct study arms (intervention and control). Although 84.62% (n=99) of the papers contained information relevant to Behaviour Change Techniques (BCTs), the information extraction algorithm correctly extracted only 58.59% (n=58) of BCTs. Conclusions The evaluation found that the information extraction algorithm did not extract the outcome values and key BCIO entities correctly against the correct arms in any of the papers in our sample, making it unsuitable for deployment in the outcome prediction and research browser tools. Several challenges with working in interdisciplinary teams were identified and discussed along with lessons learned for future work.

Integrating deep learning architectures for enhanced biomedical relation extraction: a pipeline approach.

Biomedical relation extraction from scientific publications is a key task in biomedical natural language processing (NLP) and can facilitate the creation of large knowledge bases, enable more efficient knowledge discovery, and accelerate evidence synthesis. In this paper, building upon our previous effort in the BioCreative VIII BioRED Track, we propose an enhanced end-to-end pipeline approach for biomedical relation extraction (RE) and novelty detection (ND) that effectively leverages existing datasets and integrates state-of-the-art deep learning methods. Our pipeline consists of four tasks performed sequentially: named entity recognition (NER), entity linking (EL), RE, and ND. We trained models using the BioRED benchmark corpus that was the basis of the shared task. We explored several methods for each task and combinations thereof: for NER, we compared a BERT-based sequence labeling model that uses the BIO scheme with a span classification model. For EL, we trained a convolutional neural network model for diseases and chemicals and used an existing tool, PubTator 3.0, for mapping other entity types. For RE and ND, we adapted the BERT-based, sentence-bound PURE model to bidirectional and document-level extraction. We also performed extensive hyperparameter tuning to improve model performance. We obtained our best performance using BERT-based models for NER, RE, and ND, and the hybrid approach for EL. Our enhanced and optimized pipeline showed substantial improvement compared to our shared task submission, NER: 93.53 (+3.09), EL: 83.87 (+9.73), RE: 46.18 (+15.67), and ND: 38.86 (+14.9). While the performances of the NER and EL models are reasonably high, RE and ND tasks remain challenging at the document level. Further enhancements to the dataset could enable more accurate and useful models for practical use. We provide our models and code at https://github.com/janinaj/e2eBioMedRE/. Database URL: https://github.com/janinaj/e2eBioMedRE/.

Evaluating Named Entity Recognition: A comparative analysis of mono- and multilingual transformer models on a novel Brazilian corporate earnings call transcripts dataset

Since 2018, when the Transformer architecture was introduced, Natural Language Processing has gained significant momentum with pre-trained Transformer-based models that can be fine-tuned for various tasks. Most models are pre-trained on large English corpora, making them less applicable to other languages, such as Brazilian Portuguese. In our research, we identified two models pre-trained in Brazilian Portuguese (BERTimbau and PTT5) and two multilingual models (mBERT and mT5). BERTimbau and mBERT use only the Encoder module, while PTT5 and mT5 use both the Encoder and Decoder. Our study aimed to evaluate their performance on a financial Named Entity Recognition (NER) task and determine the computational requirements for fine-tuning and inference. To this end, we developed the Brazilian Financial NER (BraFiNER) dataset, comprising sentences from Brazilian banks’ earnings calls transcripts annotated using a weakly supervised approach. Additionally, we introduced a novel approach that reframes the token classification task as a text generation problem. After fine-tuning the models, we evaluated them using performance and error metrics. Our findings reveal that BERT-based models consistently outperform T5-based models. While the multilingual models exhibit comparable macro F1-scores, BERTimbau demonstrates superior performance over PTT5. In terms of error metrics, BERTimbau outperforms the other models. We also observed that PTT5 and mT5 generated sentences with changes in monetary and percentage values, highlighting the importance of accuracy and consistency in the financial domain. Our findings provide insights into the differing performance of BERT- and T5-based models for the NER task.

Recognition and normalization of multilingual symptom entities using in-domain-adapted BERT models and classification layers.

Due to the scarcity of available annotations in the biomedical domain, clinical natural language processing poses a substantial challenge, especially when applied to low-resource languages. This paper presents our contributions for the detection and normalization of clinical entities corresponding to symptoms, signs, and findings present in multilingual clinical texts. For this purpose, the three subtasks proposed in the SympTEMIST shared task of the Biocreative VIII conference have been addressed. For Subtask 1-named entity recognition in a Spanish corpus-an approach focused on BERT-based model assemblies pretrained on a proprietary oncology corpus was followed. Subtasks 2 and 3 of SympTEMIST address named entity linking (NEL) in Spanish and multilingual corpora, respectively. Our approach to these subtasks followed a classification strategy that starts from a bi-encoder trained by contrastive learning, for which several SapBERT-like models are explored. To apply this NEL approach to different languages, we have trained these models by leveraging the knowledge base of domain-specific medical concepts in Spanish supplied by the organizers, which we have translated into the other languages of interest by using machine translation tools. The results obtained in the three subtasks establish a new state of the art. Thus, for Subtask 1 we obtain precision results of 0.804, F1-score of 0.748, and recall of 0.699. For Subtask 2, we obtain performance gains of up to 5.5% in top-1 accuracy when the trained bi-encoder is followed by a WNT-softmax classification layer that is initialized with the mean of the embeddings of a subset of SNOMED-CT terms. For Subtask 3, the differences are even more pronounced, and our multilingual bi-encoder outperforms the other models analyzed in all languages except Swedish when combined with a WNT-softmax classification layer. Thus, the improvements in top-1 accuracy over the best bi-encoder model alone are 13% for Portuguese and 13.26% for Swedish. Database URL: https://doi.org/10.1093/database/baae087.

Characterization of Nanoparticles in Drinking Water Using Field-Flow Fractionation Coupled with Multi-Angle Light Scattering and Inductively Coupled Plasma Mass Spectrometry

The current absence of well-established and standardized methods for characterizing submicrometer- and nano-sized particles in water samples presents a significant analytical challenge. With the increasing utilization of nanomaterials, the potential for unintended exposure escalates. The widespread and persistent pollution of water by micro- and nanoplastics globally is a concern that demands attention, not only to reduce pollution but also to develop methods for analyzing these pollutants. Additionally, the analysis of naturally occurring nano entities such as bubbles and colloidal matter poses challenges due to the lack of systematic and consistent methodologies. This study presents Asymmetric Flow Field-Flow Fractionation (AF4) separation coupled with a UV-VIS spectrometer followed by Multi-Angle Light Scattering (MALS) for detection and size characterization of nanometric entities. It is coupled with an Inductively Coupled Plasma Mass Spectrometer (ICP-MS) for elemental analysis. Water samples from different sources, such as untreated mountain spring water, groundwater, and bottled drinking water, were analyzed. The system was calibrated using pure particle standards of different metallic compositions. Our study demonstrates the capability of AF4-UV-MALS-ICP-MS to detect metals such as Al, Ba, Cu, and Zn in particles of around 200 nm diameter and Mg associated with very small particles between 1.5 and 10 nm in different drinking water samples.

Relation extraction for colorectal cancer via deep learning with entity-aware feature orthogonal decomposition

Relation extraction is significant for text structuring of colorectal cancer (CRC) pathological reports to facilitate doctors’ disease diagnoses. Although many relation extraction methods have been extensively studied for various natural language processing applications, they cannot be well transferred to be applied for CRC pathological reports since CRC pathological reports have some unique characteristics. To this end, a deep learning framework is designed in this paper to extract entity relations in CRC pathological reports, which is based on an encoder–decoder architecture with entity-aware feature orthogonal decomposition. Specifically, to effectively extract semantic features of long and short entities, a two-stream encoder is designed based on an edge-aware convolutional neural network and a dimension-aware dilated convolution residual network. To alleviate the influence of the blending of subject–object features, entity-aware feature orthogonal decomposition is designed to decompose the extracted semantic features into three types, i.e. subject features, object features and subject–object shared features. A stage-wise cross entropy loss is proposed to well train the network. Comparison experiments indicated that our designed network performs well on CRC pathological texts with the performance of 92.3% F1 score, 93.1% Precision, and 91.5% Recall, outperforming the existing relation extraction models.

From zero to hero: Harnessing transformers for biomedical named entity recognition in zero- and few-shot contexts

Supervised named entity recognition (NER) in the biomedical domain depends on large sets of annotated texts with the given named entities. The creation of such datasets can be time-consuming and expensive, while extraction of new entities requires additional annotation tasks and retraining the model. This paper proposes a method for zero- and few-shot NER in the biomedical domain to address these challenges. The method is based on transforming the task of multi-class token classification into binary token classification and pre-training on a large number of datasets and biomedical entities, which allows the model to learn semantic relations between the given and potentially novel named entity labels. We have achieved average F1 scores of 35.44% for zero-shot NER, 50.10% for one-shot NER, 69.94% for 10-shot NER, and 79.51% for 100-shot NER on 9 diverse evaluated biomedical entities with fine-tuned PubMedBERT-based model. The results demonstrate the effectiveness of the proposed method for recognizing new biomedical entities with no or limited number of examples, outperforming previous transformer-based methods, and being comparable to GPT3-based models using models with over 1000 times fewer parameters. We make models and developed code publicly available.

A novel prompting method for few-shot NER via LLMs

In various natural language processing tasks, significant strides have been made by Large Language Models (LLMs). Researchers leverage prompt method to conduct LLMs in accomplishing specific tasks under few-shot conditions. However, the prevalent use of LLMs’ prompt methods mainly focuses on guiding generative tasks, and employing existing prompts may result in poor performance in Named Entity Recognition (NER) tasks. To tackle this challenge, we propose a novel prompting method for few-shot NER. By enhancing existing prompt methods, we devise a standardized prompts tailored for the utilization of LLMs in NER tasks. Specifically, we structure the prompts into three components: task definition, few-shot demonstration, and output format. The task definition conducts LLMs in performing NER tasks, few-shot demonstration assists LLMs in understanding NER task objectives through specific output demonstration, and output format restricts LLMs’ output to prevent the generation of unnecessary results. The content of these components has been specifically tailored for NER tasks. Moreover, for the few-shot demonstration within the prompts, we propose a selection strategy that utilizes feedback from LLMs’ outputs to identify more suitable few-shot demonstration as prompts. Additionally, to enhance entity recognition performance, we enrich the prompts by summarizing error examples from the output process of LLMs and integrating them as additional prompts.

Comparing a Large Language Model with Previous Deep Learning Models on Named Entity Recognition of Adverse Drug Events.

The ability to fine-tune pre-trained deep learning models to learn how to process a downstream task using a large training set allow to significantly improve performances of named entity recognition. Large language models are recent models based on the Transformers architecture that may be conditioned on a new task with in-context learning, by providing a series of instructions or prompt. These models only require few examples and such approach is defined as few shot learning. Our objective was to compare performances of named entity recognition of adverse drug events between state of the art deep learning models fine-tuned on Pubmed abstracts and a large language model using few-shot learning. Hussain et al's state of the art model (PMID: 34422092) significantly outperformed the ChatGPT-3.5 model (F1-Score: 97.6% vs 86.0%). Few-shot learning is a convenient way to perform named entity recognition when training examples are rare, but performances are still inferior to those of a deep learning model fine-tuned with several training examples. Perspectives are to evaluate few-shot prompting with GPT-4 and perform fine-tuning on GPT-3.5.

Accelerating Clinical Text Annotation in Underrepresented Languages: A Case Study on Text De-Identification.

Clinical notes contain valuable information for research and monitoring quality of care. Named Entity Recognition (NER) is the process for identifying relevant pieces of information such as diagnoses, treatments, side effects, etc., and bring them to a more structured form. Although recent advancements in deep learning have facilitated automated recognition, particularly in English, NER can still be challenging due to limited specialized training data. This exacerbated in hospital settings where annotations are costly to obtain without appropriate incentives and often dependent on local specificities. In this work, we study whether this annotation process can be effectively accelerated by combining two practical strategies. First, we convert usually passive annotation tasks into a proactive contest to motivate human annotators in performing a task often considered tedious and time-consuming. Second, we provide pre-annotations for the participants to evaluate how recall and precision of the pre-annotations can boost or deteriorate annotation performance. We applied both strategies to a text de-identification task on French clinical notes and discharge summaries at a large Swiss university hospital. Our results show that proactive contest and average quality pre-annotations can significantly speed up annotation time and increase annotation quality, enabling us to develop a text de-identification model for French clinical notes with high performance (F1 score 0.94).