Language Processing Systems Research Articles

A Survey of Grapheme-to-Phoneme Conversion Methods

Grapheme-to-phoneme conversion (G2P) is the task of converting letters (grapheme sequences) into their pronunciations (phoneme sequences). It plays a crucial role in natural language processing, text-to-speech synthesis, and automatic speech recognition systems. This paper provides a systematical overview of the G2P conversion from different perspectives. The conversion methods are first presented in the paper; detailed discussions are conducted on methods based on deep learning technology. For each method, the key ideas, advantages, disadvantages, and representative models are summarized. This paper then mentioned the learning strategies and multilingual G2P conversions. Finally, this paper summarized the commonly used monolingual and multilingual datasets, including Mandarin, Japanese, Arabic, etc. Two tables illustrated the performance of various methods with relative datasets. After making a general overall of G2P conversion, this paper concluded with the current issues and the future directions of deep learning-based G2P conversion.

Variational Autoencoders-Based Algorithm for Multi-Criteria Recommendation Systems

In recent years, recommender systems have become a crucial tool, assisting users in discovering and engaging with valuable information and services. Multi-criteria recommender systems have demonstrated significant value in assisting users to identify the most relevant items by considering various aspects of user experiences. Deep learning (DL) models demonstrated outstanding performance across different domains: computer vision, natural language processing, image analysis, pattern recognition, and recommender systems. In this study, we introduce a deep learning model using VAE to improve multi-criteria recommendation systems. Specifically, we propose a variational autoencoder-based model for multi-criteria recommendation systems (VAE-MCRS). The VAE-MCRS model is sequentially trained across multiple criteria to uncover patterns that allow for better representation of user–item interactions. The VAE-MCRS model utilizes the latent features generated by the VAE in conjunction with user–item interactions to enhance recommendation accuracy and predict ratings for unrated items. Experiments carried out using the Yahoo! Movies multi-criteria dataset demonstrate that the proposed model surpasses other state-of-the-art recommendation algorithms, achieving a Mean Absolute Error (MAE) of 0.6038 and a Root Mean Squared Error (RMSE) of 0.7085, demonstrating its superior performance in providing more precise recommendations for multi-criteria recommendation tasks.

A Review on the Application of Automatic Text Summarization and Its Bias Analysis

Abstract. In the era of information explosion, the Internet produces a large number of text documents daily, providing rich information but also posing a challenge: how to swiftly extract key information. Traditional manual reading is time-consuming and inadequate for handling vast data. Thus, automatic text summarization technology emerges as a crucial solution. This paper reviews the application and deviation analysis of this technology across various fields, focusing on addressing shortcomings of traditional methods, such as initial cluster center selection and redundancy. An automatic text summarization method based on an improved TextRank algorithm and K-Means clustering is introduced. Existing methods often struggle with inaccurate initial clustering center selection and high summary redundancy, especially with long texts, resulting in summaries that fail to reflect core content accurately. Furthermore, the widespread use of pre-trained language models introduces potential biases that can propagate to downstream tasks, affecting summary accuracy and impartiality. To address these issues, this paper proposes an innovative automatic text summarization method that optimizes initial clustering center selection and clustering refinement strategies to enhance summary accuracy and readability. Additionally, it discusses name-nationality bias in pre-trained language models and its propagation in text summary tasks, offering a theoretical foundation and practical guidance for developing a more just and reliable Natural Language Processing (NLP) system.

Moving towards the use of artificial intelligence in pain management.

While the development of artificial intelligence (AI) technologies in medicine has been significant, their application to acute and chronic pain management has not been well characterized. This systematic review aims to provide an overview of the current state of AI in acute and chronic pain management. This review was registered with PROSPERO (ID# CRD42022307017), the international registry for systematic reviews. The search strategy was prepared by a librarian and run in four electronic databases (Embase, Medline, Central, and Web of Science). Collected articles were screened by two reviewers. Included studies described the use of AI for acute and chronic pain management. From the 17,601 records identified in the initial search, 197 were included in this review. Identified applications of AI were described for treatment planning as well as treatment delivery. Described uses include prediction of pain, forecasting of individualized responses to treatment, treatment regimen tailoring, image-guidance for procedural interventions and self-management tools. Multiple domains of AI were used including machine learning, computer vision, fuzzy logic, natural language processing and expert systems. There is growing literature regarding applications of AI for pain management, and their clinical use holds potential for improving patient outcomes. However, multiple barriers to their clinical integration remain including lack validation of such applications in diverse patient populations, missing infrastructure to support these tools and limited provider understanding of AI. This review characterizes current applications of AI for pain management and discusses barriers to their clinical integration. Our findings support continuing efforts directed towards establishing comprehensive systems that integrate AI throughout the patient care continuum.

Online transcranial magnetic stimulation reveals dynamic interactions between language control and processing in bilingual language production.

Language production in bilinguals relies on the collaborative interaction between two neural systems: the language control system (e.g. the right inferior frontal gyrus) and the language processing system (e.g. the left superior temporal gyrus). However, the causal mapping between these brain systems and their cognitive functions, as well as the temporal dynamics between these two systems during bilingual language production, remain unclear. To investigate this, our study applied online transcranial magnetic stimulation to the right inferior frontal gyrus and left superior temporal gyrus in Chinese-English bilinguals performing a language-switching task. Transcranial magnetic stimulation was administered within seven 100 ms time windows (TW1 to TW7) following the picture onset. We observed time window-specific enhancements of language control, as shown by a significant reduction in switching costs upon the stimulations on both sites relative to the sham stimulation: specifically, the right inferior frontal gyrus at TW1, TW2, TW4, and TW5 and the left superior temporal gyrus at TW2 and TW5. These findings indicate a double-strike top-down control mechanism underpinned by the pathway from the right inferior frontal gyrus to the right inferior frontal gyrus, potentially during both stages during production: the language task schema and the lemma selection. Our study provides the first evidence of the dynamic interactions between language control and processing systems causally underpinning bilingual language production.

Arabic word tokenization system using the maximum matching model

Word tokenization is the first stage for higher-order Natural Language Processing (NLP) tasks like Part-of-Speech (PoS) tagging, parsing, and named entity recognition. The amount of text on the World Wide Web is growing daily in the present era of technology, necessitating the use of advanced instruments. Since more and more people speak Arabic around the world, Arabic language processing systems must be improved. Due to the writing style of Arabic with a lack of support for capitalization features and the use of compound words, it is difficult to perform word tokenization. This research paper proposes a novel Arabic word tokenization system based on the knowledge. To develop this system, a maximum matching model with its two variations, namely forward and reverse maximum matching is used. The proposed system is implemented in Python. The results produced during system evaluation report high performance.

Recent One Decade Trends in Mathematical Representation Research: Systematic Mapping Study

Research can change over time and can be influenced by technological developments, educational trends, and current social issues. Developments on trends in mathematical representation research over the past 10 years are discussed in this systematic mapping study (SMS). The question of review research with the systematic mapping study method is how the trend of mathematical representation research over the last 1 decade and what new topics have the opportunity to be researched by future researchers related to the themes of mathematical representation. The method used in this review is the PRISMA method which includes 5 stages carried out: 1) defining eligibility criteria, 2) determining information sources, 3) data selection, 4) data collection, and 5) data collection. In this systematic mapping study, it starts from searching for articles on the topic of mathematical representation during the last 1 decade, namely the range of 2013-2023, articles obtained from the scopus database in July 2023 with the keywords "representations" AND : Mathematics" initial data obtained 1907 articles after the exclusion process obtained 120 articles which were then analyzed using R-Studio and Vos-Viewer tools. The results of SMS obtained the relationship of representation topics in the last 10 years of research with the topics of visual, computer software, computer aid instruction, data mining, open source software, network architecture, natural language processing systems, intelligent systems and information analysis. In addition, research over the past 1 decade on the topic of representation has led more to data mining, security of data and STEM. Artificial Intelligence or artificial intelligence is the latest topic that over the past 3 years has grown in various articles that deal with the theme of mathematical representation and accompanied by the theme of multi-representation. The evolution of research topics related to mathematical representations that used to be only related to visual representations and image representations now leads to computer programs, programming algorithms and artificial intelligence, this shows that over the past 1 decade there has been an evolution of research in the implementation of representations in technology.

Balancing Privacy and Robustness in Prompt Learning for Large Language Models

This paper tackles the critical issue of privacy in Natural Language Processing (NLP) systems that process sensitive data by introducing a novel framework combining differential privacy and adversarial training. The proposed solution ensures formal privacy guarantees by minimizing the influence of individual data points on the model’s behavior, effectively preventing information leakage. Simultaneously, adversarial training is applied to strengthen model robustness against privacy attacks by exposing it to adversarial examples during training. The framework is rigorously evaluated across various NLP tasks, demonstrating its capability to balance privacy preservation with high utility effectively. These results mark a significant advancement in developing secure and reliable NLP systems, particularly for applications requiring stringent data confidentiality, such as healthcare and finance.

Extracting social support and social isolation information from clinical psychiatry notes: comparing a rule-based natural language processing system and a large language model.

Social support (SS) and social isolation (SI) are social determinants of health (SDOH) associated with psychiatric outcomes. In electronic health records (EHRs), individual-level SS/SI is typically documented in narrative clinical notes rather than as structured coded data. Natural language processing (NLP) algorithms can automate the otherwise labor-intensive process of extraction of such information. Psychiatric encounter notes from Mount Sinai Health System (MSHS, n = 300) and Weill Cornell Medicine (WCM, n = 225) were annotated to create a gold-standard corpus. A rule-based system (RBS) involving lexicons and a large language model (LLM) using FLAN-T5-XL were developed to identify mentions of SS and SI and their subcategories (eg, social network, instrumental support, and loneliness). For extracting SS/SI, the RBS obtained higher macroaveraged F1-scores than the LLM at both MSHS (0.89 versus 0.65) and WCM (0.85 versus 0.82). For extracting the subcategories, the RBS also outperformed the LLM at both MSHS (0.90 versus 0.62) and WCM (0.82 versus 0.81). Unexpectedly, the RBS outperformed the LLMs across all metrics. An intensive review demonstrates that this finding is due to the divergent approach taken by the RBS and LLM. The RBS was designed and refined to follow the same specific rules as the gold-standard annotations. Conversely, the LLM was more inclusive with categorization and conformed to common English-language understanding. Both approaches offer advantages, although additional replication studies are warranted.

Exploring noise-induced techniques to strengthen deep learning model resilience

AbstractIn artificial intelligence, combating overfitting and enhancing model generalization is crucial. This research explores innovative noise-induced regularization techniques, focusing on natural language processing tasks. Inspired by gradient noise and Dropout, this study investigates the interplay between controlled noise, model complexity, and overfitting prevention. Utilizing long short-term memory and bidirectional long short term memory architectures, this study examines the impact of noise-induced regularization on robustness to noisy input data. Through extensive experimentation, this study shows that introducing controlled noise improves model generalization, especially in language understanding. This contributes to the theoretical understanding of noise-induced regularization, advancing reliable and adaptable artificial intelligence systems for natural language processing.

Deep-transfer learning inspired natural language processing system for software requirements classification

Deep-transfer learning inspired natural language processing system for software requirements classification

Leveraging Temporal Trends for Training Contextual Word Embeddings to Address Bias in Biomedical Applications: Development Study.

Women have been underrepresented in clinical trials for many years. Machine-learning models trained on clinical trial abstracts may capture and amplify biases in the data. Specifically, word embeddings are models that enable representing words as vectors and are the building block of most natural language processing systems. If word embeddings are trained on clinical trial abstracts, predictive models that use the embeddings will exhibit gender performance gaps. We aim to capture temporal trends in clinical trials through temporal distribution matching on contextual word embeddings (specifically, BERT) and explore its effect on the bias manifested in downstream tasks. We present TeDi-BERT, a method to harness the temporal trend of increasing women's inclusion in clinical trials to train contextual word embeddings. We implement temporal distribution matching through an adversarial classifier, trying to distinguish old from new clinical trial abstracts based on their embeddings. The temporal distribution matching acts as a form of domain adaptation from older to more recent clinical trials. We evaluate our model on 2 clinical tasks: prediction of unplanned readmission to the intensive care unit and hospital length of stay prediction. We also conduct an algorithmic analysis of the proposed method. In readmission prediction, TeDi-BERT achieved area under the receiver operating characteristic curve of 0.64 for female patients versus the baseline of 0.62 (P<.001), and 0.66 for male patients versus the baseline of 0.64 (P<.001). In the length of stay regression, TeDi-BERT achieved a mean absolute error of 4.56 (95% CI 4.44-4.68) for female patients versus 4.62 (95% CI 4.50-4.74, P<.001) and 4.54 (95% CI 4.44-4.65) for male patients versus 4.6 (95% CI 4.50-4.71, P<.001). In both clinical tasks, TeDi-BERT improved performance for female patients, as expected; but it also improved performance for male patients. Our results show that accuracy for one gender does not need to be exchanged for bias reduction, but rather that good science improves clinical results for all. Contextual word embedding models trained to capture temporal trends can help mitigate the effects of bias that changes over time in the training data.

Integration of big data analytics in the investigation of the relationship between acromegaly and cancer

ObjectiveTo evaluate the association between acromegaly and cancer and different types of cancer by using natural language processing systems and big data analytics. Material and methodsWe conducted an observational, retrospective study utilizing data from the electronic health records (EHRs) of Hospital Universitario Puerta de Hierro Majadahonda, Madrid, Spain. Information from the EHRs was extracted using artificial intelligence techniques and analyzed using Savana Manager 4.0 software. ResultsOut of a total of 708,047 registered patients (54.7% females), 544 patients (0.08%; 330 women, 60.7%; mean age at diagnosis 53.0±15.8 yr) were diagnosed with acromegaly. The incidence of cancer was higher in patients with acromegaly vs those without this condition (7.7% vs 3.9%, p<0.001; OR, 2.047, 95%CI, 1.493–2.804). Male acromegalic patients had a higher prevalence of cancer vs females (57.1% vs 42.9%, p=0.012). A significantly higher prevalence of colorectal cancer (2.9% vs 1.4%, p=0.006), bladder cancer (1.1% vs 0.3%, p=0.005), and lymphoma (1.1% vs 0.3%, p=0.009) was observed in patients with acromegaly vs those without the condition. Acromegalic men had significantly higher prevalence rates of colorectal cancer (4.7% vs 1.3%, p=0.001), bladder cancer (2.8% vs 0.4%, p<0.001), breast cancer (0.9% vs 0.2%, p=0.042), gastric cancer (0.9% vs 0.1%, p=0.011), lymphoma (1.4% vs 0.3%, p=0.037), and liver cancer (0.9% vs 0.1%, p=0.012) vs non-acromegalic men. On the other hand, acromegalic women showed a higher prevalence of thyroid cancer (1.2% vs 0.4%, p=0.043) vs non-acromegalic women. ConclusionOur study, based on artificial intelligence techniques and analysis of real-world data and information, revealed a significant association between acromegaly and cancer in our hospital population, mainly acromegalic men, with a higher frequency of colorectal cancer, bladder cancer and lymphoma in particular.

DICTIONARY-BASED DETERMINISTIC METHOD OF GENERATION OF TEXT CORPORA

This paper examines the problem of solving software engineering tasks in developing information systems for natural language processing. Generating corpora of text data is highlighted as a specific task of this problem. An analysis of the basic CorDeGen method was carried out, which is one of the corpus generation methods specially developed for this specific problem. This study shows that this method has a limited scope due to the use of “artificial” terms to fill the texts.The paper proposes a new modified DBCorDeGen method that solves this shortcoming thanks to the use of an additional dictionary of terms that is supplied to the input of the method. The DBCorDeGen method preserves most of the characteristic features of the basic method, which are important for its use in solving software engineering tasks: determinism, speed of operation (including the possibility of combining with parallel modification), the possibility of a priori description of the structure and properties of the generated corpus. The only disadvantage compared to the basic method is the increase in the number of input parameters, however, compared to other methods of generating corpora presented in the literature, it is relatively small, and due to it, the scope of application of corpora generated by this method significantly increases.As an experimental test of the proposed modified DBCorDeGen method, the task of sentiment analysis of the texts of the generated corpus is considered. The study shows that when using the basic CorDeGen method, it is impossible to obtain sentiment analysis results different from neutral polarity for all texts. When using the proposed method, it is possible to obtain different results using different dictionaries. Thus, it is confirmed that the proposed DBCorDeGen method has a larger scope than the basic method.

Token-modification adversarial attacks for natural language processing: A survey

Many adversarial attacks target natural language processing systems, most of which succeed through modifying the individual tokens of a document. Despite the apparent uniqueness of each of these attacks, fundamentally they are simply a distinct configuration of four components: a goal function, allowable transformations, a search method, and constraints. In this survey, we systematically present the different components used throughout the literature, using an attack-independent framework which allows for easy comparison and categorisation of components. Our work aims to serve as a comprehensive guide for newcomers to the field and to spark targeted research into refining the individual attack components.

A natural language processing system for the efficient extraction of cell markers

Single-cell RNA sequencing (scRNA-seq) has emerged as a pivotal tool for exploring cellular landscapes across diverse species and tissues. Precise annotation of cell types is essential for understanding these landscapes, relying heavily on empirical knowledge and curated cell marker databases. In this study, we introduce MarkerGeneBERT, a natural language processing (NLP) system designed to extract critical information from the literature regarding species, tissues, cell types, and cell marker genes in the context of single-cell sequencing studies. Leveraging MarkerGeneBERT, we systematically parsed full-text articles from 3702 single-cell sequencing-related studies, yielding a comprehensive collection of 7901 cell markers representing 1606 cell types across 425 human tissues/subtissues, and 8223 cell markers representing 1674 cell types across 482 mouse tissues/subtissues. Comparative analysis against manually curated databases demonstrated that our approach achieved 76% completeness and 75% accuracy, while also unveiling 89 cell types and 183 marker genes absent from existing databases. Furthermore, we successfully applied the compiled brain tissue marker gene list from MarkerGeneBERT to annotate scRNA-seq data, yielding results consistent with original studies. Conclusions: Our findings underscore the efficacy of NLP-based methods in expediting and augmenting the annotation and interpretation of scRNA-seq data, providing a systematic demonstration of the transformative potential of this approach. The 27323 manual reviewed sentences for training MarkerGeneBERT and the source code are hosted at https://github.com/chengpeng1116/MarkerGeneBERT.

A Case Demonstration of the Open Health Natural Language Processing Toolkit From the National COVID-19 Cohort Collaborative and the Researching COVID to Enhance Recovery Programs for a Natural Language Processing System for COVID-19 or Postacute Sequelae of SARS CoV-2 Infection: Algorithm Development and Validation.

A wealth of clinically relevant information is only obtainable within unstructured clinical narratives, leading to great interest in clinical natural language processing (NLP). While a multitude of approaches to NLP exist, current algorithm development approaches have limitations that can slow the development process. These limitations are exacerbated when the task is emergent, as is the case currently for NLP extraction of signs and symptoms of COVID-19 and postacute sequelae of SARS-CoV-2 infection (PASC). This study aims to highlight the current limitations of existing NLP algorithm development approaches that are exacerbated by NLP tasks surrounding emergent clinical concepts and to illustrate our approach to addressing these issues through the use case of developing an NLP system for the signs and symptoms of COVID-19 and PASC. We used 2 preexisting studies on PASC as a baseline to determine a set of concepts that should be extracted by NLP. This concept list was then used in conjunction with the Unified Medical Language System to autonomously generate an expanded lexicon to weakly annotate a training set, which was then reviewed by a human expert to generate a fine-tuned NLP algorithm. The annotations from a fully human-annotated test set were then compared with NLP results from the fine-tuned algorithm. The NLP algorithm was then deployed to 10 additional sites that were also running our NLP infrastructure. Of these 10 sites, 5 were used to conduct a federated evaluation of the NLP algorithm. An NLP algorithm consisting of 12,234 unique normalized text strings corresponding to 2366 unique concepts was developed to extract COVID-19 or PASC signs and symptoms. An unweighted mean dictionary coverage of 77.8% was found for the 5 sites. The evolutionary and time-critical nature of the PASC NLP task significantly complicates existing approaches to NLP algorithm development. In this work, we present a hybrid approach using the Open Health Natural Language Processing Toolkit aimed at addressing these needs with a dictionary-based weak labeling step that minimizes the need for additional expert annotation while still preserving the fine-tuning capabilities of expert involvement.

Inheriting the glory of PāṆinian grammar into computer science

the integration of Pāṇinian grammar principles into computer science represents a significant advancement in computational linguistics and natural language processing. Pāṇini, an ancient Indian grammarian, formulated an elaborate system of grammar known as Aṣṭādhyāyī around the 4th century BCE, which has been revered for its systematic and concise rules governing Sanskrit language structure. In recent years, there has been a growing interest in adapting Pāṇinian principles to enhance computational frameworks for understanding human languages. This paper explores the theoretical foundations of Pāṇinian grammar and its applicability in the context of modern computational linguistics. The integration of Paninian grammar into computer science enhances the accuracy and efficiency of natural language processing system and contributes to preservation and dissemination of ancient linguistic knowledge. Through this the researchers aim to bridge gap between traditional linguistic scholarship and modern computational methodologies. By embracing the systematic and rigorous framework laid out by Pāṇini millennia ago, researchers are poised to achieve new breakthroughs in understanding and processing human language through innovative computational techniques. The journey from ancient Sanskrit grammar to innovative computer science exemplifies a harmonious blend of tradition and innovation in the quest for linguistic knowledge and technological advancement. Keywords - Pāṇini, Aṣṭādhyāyī, Computational linguistics, Machine translation, semantic analysis, syntactic rules.

From rule-based models to deep learning transformers architectures for natural language processing and sign language translation systems: survey, taxonomy and performance evaluation

With the growing Deaf and Hard of Hearing population worldwide and the persistent shortage of certified sign language interpreters, there is a pressing need for an efficient, signs-driven, integrated end-to-end translation system, from sign to gloss to text and vice-versa. There has been a wealth of research on machine translations and related reviews. However, there are few works on sign language machine translation considering the particularity of the language being continuous and dynamic. This paper aims to address this void, providing a retrospective analysis of the temporal evolution of sign language machine translation algorithms and a taxonomy of the Transformers architectures, the most used approach in language translation. We also present the requirements of a real-time Quality-of-Service sign language machine translation system underpinned by accurate deep learning algorithms. We propose future research directions for sign language translation systems.

Long-term Outcomes of Peripheral Artery Disease In Veterans: Analysis of the PEripheral ARtery Disease Long-term Survival Study (PEARLS).

Contemporary research in peripheral artery disease (PAD) remains limited due to lack of a national registry and low accuracy of diagnosis codes to identify PAD patients in electronic health records. Leveraging a novel natural language processing (NLP) system that identifies PAD with high accuracy using ankle brachial index (ABI) and toe-brachial index (TBI) values, we created a registry of 103,748 patients with new onset PAD patients in the Veterans Health Administration (VHA). Study endpoints include mortality, cardiovascular (hospitalization for acute myocardial infarction or stroke) and limb events (hospitalization for critical limb ischemia or major amputation) and were identified using VA and non-VA encounters. The mean age was 70.6 years; 97.3% were males, and 18.5% self-identified as Black race. The mean ABI value was 0.78 (SD: 0.26) and the mean TBI value was 0.51 (SD: 0.19). Nearly one-third (32.4%) patients were currently smoking and 35.4% formerly smoked. Prevalence of hypertension (86.6%), heart failure (22.7%), diabetes (54.8%), renal failure (23.6%), and chronic obstructive pulmonary disease (35.4%) was high. At 1-year, 9.4% of patients had died. The 1-year incidence of cardiovascular events was 5.6 per 100 patient-years and limb events was 4.5 per 100 patient-years. We have successfully launched a registry of >100,000 patients with a new diagnosis of PAD in the VHA, the largest integrated health system in the U.S. The ncidence of death and clinical events in our cohort is high. Ongoing studies will yield important insights regarding improving care and outcomes in this high-risk group.