Neural Language Processing Research Articles

Training-Free Transformer Architecture Search With Zero-Cost Proxy Guided Evolution.

Transformers have shown remarkable performance, however, their architecture design is a time-consuming process that demands expertise and trial-and-error. Thus, it is worthwhile to investigate efficient methods for automatically searching high-performance Transformers via Transformer Architecture Search (TAS). In order to improve the search efficiency, training-free proxy based methods have been widely adopted in Neural Architecture Search (NAS). Whereas, these proxies have been found to be inadequate in generalizing well to Transformer search spaces, as confirmed by several studies and our own experiments. This paper presents an effective scheme for TAS called TRansformer Architecture search with ZerO-cost pRoxy guided evolution (T-Razor) that achieves exceptional efficiency. First, through theoretical analysis, we discover that the synaptic diversity of multi-head self-attention (MSA) and the saliency of multi-layer perceptron (MLP) are correlated with the performance of corresponding Transformers. The properties of synaptic diversity and synaptic saliency motivate us to introduce the ranks of synaptic diversity and saliency that denoted as DSS++ for evaluating and ranking Transformers. DSS++ incorporates correlation information among sampled Transformers to provide unified scores for both synaptic diversity and synaptic saliency. We then propose a block-wise evolution search guided by DSS++ to find optimal Transformers. DSS++ determines the positions for mutation and crossover, enhancing the exploration ability. Experimental results demonstrate that our T-Razor performs competitively against the state-of-the-art manually or automatically designed Transformer architectures across four popular Transformer search spaces. Significantly, T-Razor improves the searching efficiency across different Transformer search spaces, e.g., reducing required GPU days from more than 24 to less than 0.4 and outperforming existing zero-cost approaches. We also apply T-Razor to the BERT search space and find that the searched Transformers achieve competitive GLUE results on several Neural Language Processing (NLP) datasets. This work provides insights into training-free TAS, revealing the usefulness of evaluating Transformers based on the properties of their different blocks.

Application of Artificial Intelligence Models in Nursing Research

In recent years, the rapid development of artificial intelligence has enhanced the efficiency of medical services, accuracy of disease prediction, and innovation in the healthcare industry. Among the many advances, machine learning has become a focal point of development in various fields. Although its use in nursing research and clinical care has been limited, technological progress promises broader applications of machine learning in these areas in the future. In this paper, the authors discuss the application of machine learning in nursing research and care. First, the types and classifications of machine learning are introduced. Next, common neural machine learning models, including recurrent neural networks, transformers, and natural language processing, are described and analyzed. Subsequently, the principles and steps of machine learning are explored and compared to traditional statistical methods, highlighting the quality-monitoring strategies used by machine learning models and the potential limitations and challenges of using machine learning. Finally, interdisciplinary collaboration is encouraged to share knowledge between information technology and nursing disciplines, analyze the advantages and disadvantages of various analytical models, continuously review the research process, and reflect on methodological limitations. Following this course, can help maximize the potential of artificial-intelligence-based technologies to drive innovation and progress in nursing research.

Linear symmetric self-selecting 14-bit kinetic molecular memristors.

Artificial Intelligence (AI) is the domain of large resource-intensive data centres that limit access to a small community of developers1,2. Neuromorphic hardware promises greatly improved space and energy efficiency for AI but is presently only capable of low-accuracy operations, such as inferencing in neural networks3-5. Core computing tasks of signal processing, neural network training and natural language processing demand far higher computing resolution, beyond that of individual neuromorphic circuit elements6-8. Here we introduce an analog molecular memristor based on a Ru-complex of an azo-aromatic ligand with 14-bit resolution. Precise kinetic control over a transition between two thermodynamically stable molecular electronic states facilitates 16,520 distinct analog conductance levels, which can be linearly and symmetrically updated or written individually in one time step, substantially simplifying the weight update procedure over existing neuromorphic platforms3. The circuit elements are unidirectional, facilitating a selector-less 64 × 64 crossbar-based dot-product engine that enables vector-matrix multiplication, including Fourier transform, in a single time step. We achieved more than 73 dB signal-to-noise-ratio, four orders of magnitude improvement over the state-of-the-art methods9-11, while consuming 460× less energy than digital computers12,13. Accelerators leveraging these molecular crossbars could transform neuromorphic computing, extending it beyond niche applications and augmenting the core of digital electronics from the cloud to the edge12,13.

The Promising Role of Artificial Intelligence in Navigating Lung Cancer Prognosis

Incorporating AI in lung cancer management is a disruptive innovation that has improved diagnosis accuracy, prognosis prediction and treatment modalities. In this literature review, we seek to identify the role of artificial intelligence (AI) and machine learning (ML) in lung cancer detection, diagnosis and treatment between 2010 and 2023. A total of 55 studies were selected systematically from databases such as IEEE Xplore, Scopus and PubMed via a PRISMA-based approach. The analysis reveals that artificial intelligence (AI) techniques, specifically convolutional neural networks (CNNs) and natural language processing (NLP), highly improve the precision of initial detection and imaging of lung cancer. Also, CNN distinguishes between benign and malignant nodules, thus aiding early diagnosis and reducing unnecessary biopsies. On the other hand, NLP is utilized to extract relevant clinical information from electronic health records and unstructured medical texts, thereby enhancing the understanding of patient histories and improving treatment planning. Sensitive and specific scores usually higher than standard techniques characterize these technologies. Results show that traditional statistical approaches couldn’t match AI models whose predictive accuracies are outstanding while providing better care to patients through personalised treatment plans. Furthermore, multi-omics data analysis for personalised treatment planning and clinical decision-making optimisation via Clinical Decision Support Systems (CDSS) powered by AI are some ways artificial intelligence has exhibited its potential in this area. Given this, future studies should aim to fine-tune AI algorithms, improve data integration, and address ethical issues promoting responsible use of AI technologies in clinical practice settings. Despite these advances, data quality, model interpretability, and integration into clinical workflows persist. This review demonstrates the demand for continued research and collaboration from different disciplines so that the complete possibilities of AI in fighting lung cancer may be realised.

Brain and grammar: revealing electrophysiological basic structures with competing statistical models.

Acoustic, lexical, and syntactic information are simultaneously processed in the brain requiring complex strategies to distinguish their electrophysiological activity. Capitalizing on previous works that factor out acoustic information, we could concentrate on the lexical and syntactic contribution to language processing by testing competing statistical models. We exploited electroencephalographic recordings and compared different surprisal models selectively involving lexical information, part of speech, or syntactic structures in various combinations. Electroencephalographic responses were recorded in 32 participants during listening to affirmative active declarative sentences. We compared the activation corresponding to basic syntactic structures, such as noun phrases vs. verb phrases. Lexical and syntactic processing activates different frequency bands, partially different time windows, and different networks. Moreover, surprisal models based on part of speech inventory only do not explain well the electrophysiological data, while those including syntactic information do. By disentangling acoustic, lexical, and syntactic information, we demonstrated differential brain sensitivity to syntactic information. These results confirm and extend previous measures obtained with intracranial recordings, supporting our hypothesis that syntactic structures are crucial in neural language processing. This study provides a detailed understanding of how the brain processes syntactic information, highlighting the importance of syntactic surprisal in shaping neural responses during language comprehension.

Sensing Human Emotion using Emerging Machine Learning Techniques

Human emotion recognition using machine learning is a new field that has the potential to improve user experience, lower crime, and target advertising. The ability of today's emotion detection systems to identify human emotions is essential. Applications ranging from security cameras to emotion detection are readily accessible. Machine learning-based emotion detection recognises and deciphers human emotions from text and visual data. In this study, we use convolutional neural networks and natural language processing approaches to create and assess models for emotion detection. Instead of speaking clearly, these human face expressions visually communicate a lot of information. Recognising facial expressions is important for human-machine interaction. Applications for automatic facial expression recognition systems are numerous and include, but are not limited to, comprehending human conduct, identifying mental health issues, and creating artificial human emotions. It is still difficult for computers to recognise facial expressions with a high recognition rate. Geometry and appearance-based methods are two widely used approaches for automatic FER systems in the literature. Pre-processing, face detection, feature extraction, and expression classification are the four steps that typically make up facial expression recognition. The goal of this research is to recognise the seven main human emotions anger, disgust, fear, happiness, sadness, surprise, and neutrality using a variety of deep learning techniques (convolutional neural networks).

A computer-assisted tool for automatically measuring non-native Japanese oral proficiency

This study presents the development of an automated tool for assessing the speech of non-native Japanese speakers. The tool integrated an accent-trained automatic speech recording system based on deep neural networks, audio-text transcription, and natural language processing technologies to perform annotation, analysis, calculation, and scoring. A dataset consisting of audio recordings from 1000 Japanese learners across 16 countries was used for machine training. Validity and reliability tests were parallelly developed to evaluate: (a) the measures’ ability to distinguish test takers with different oral proficiency levels; (b) the agreement between the annotations and scoring performed by the tool and human raters. The results, obtained through precision, recall, F-Score, quadratically-weighted kappa, proportional reduction of mean squared error, Pearson correlation, and multi-faceted Rasch measurement, demonstrated a relatively high reliability of the automated tool. The tool provides users with both a holistic score and individual scores for each criterion, allowing them to gain insights into their overall performance as well as detailed evaluations for each aspect of their proficiency. Consequently, it can be utilized for employment, education, and self-learning purposes.

Leveraging artificial intelligence to enhance ESG models: Transformative impacts and implementation challenges

The integration of Artificial Intelligence (AI) with Environmental, Social, and Governance (ESG) models represents a significant shift in corporate strategy and sustainability efforts. This paper explores the transformative role of deep learning and machine learning technologies in enhancing the precision, efficiency, and effectiveness of ESG frameworks. By utilizing convolutional neural networks (CNNs) and natural language processing (NLP), businesses can now process vast amounts of data, gaining insights that were previously unattainable. The study delves into quantitative analyses involving regression models and scenario analyses, backed by Monte Carlo simulations, to demonstrate the predictive power of AI-enhanced ESG models. Furthermore, the paper discusses the challenges and solutions related to data quality, computational demands, and ethical considerations in implementing AI in ESG assessments. The empirical evidence and theoretical analysis presented underline the superiority of AI-integrated models over traditional methods, showcasing improvements in time-to-insight, predictive accuracy, and cost efficiency. This study not only highlights the practical applications of AI in corporate sustainability efforts but also addresses the ethical and operational challenges faced during implementation.

Towards Faithful Model Explanation in NLP: A Survey

Abstract End-to-end neural Natural Language Processing (NLP) models are notoriously difficult to understand. This has given rise to numerous efforts towards model explainability in recent years. One desideratum of model explanation is faithfulness, that is, an explanation should accurately represent the reasoning process behind the model’s prediction. In this survey, we review over 110 model explanation methods in NLP through the lens of faithfulness. We first discuss the definition and evaluation of faithfulness, as well as its significance for explainability. We then introduce recent advances in faithful explanation, grouping existing approaches into five categories: similarity-based methods, analysis of model-internal structures, backpropagation-based methods, counterfactual intervention, and self-explanatory models. For each category, we synthesize its representative studies, strengths, and weaknesses. Finally, we summarize their common virtues and remaining challenges, and reflect on future work directions towards faithful explainability in NLP.

Fine-grained food image classification and recipe extraction using a customized deep neural network and NLP

Global eating habits cause health issues leading people to mindful eating. This has directed attention to applying deep learning to food-related data. The proposed work develops a new framework integrating neural network and natural language processing for classification of food images and automated recipe extraction. It address the challenges of intra-class variability and inter-class similarity in food images that have received shallow attention in the literature. Firstly, a customized lightweight deep convolution neural network model, MResNet-50 for classifying food images is proposed. Secondly, automated ingredient processing and recipe extraction is done using natural language processing algorithms: Word2Vec and Transformers in conjunction. Thirdly, a representational semi-structured domain ontology is built to store the relationship between cuisine, food item, and ingredients. The accuracy of the proposed framework on the Food-101 and UECFOOD256 datasets is increased by 2.4% and 7.5%, respectively, outperforming existing models in literature such as DeepFood, CNN-Food, Wiser, and other pre-trained neural networks.

Tabib: Chatbot for Healthcare Automation with Audio Assistance using Artificial Intelligence

To live a happy and healthy life, people must prioritize their health care. It's not always easy to contact a doctor when you have a health issue, though. The idea is to build a chatbot for healthcare that functions as a virtual assistant using artificial intelligence. The healthcare chatbot system's medical database will be trained using neural networks and natural language processing. The purpose of the healthcare chatbot was to increase access to medical information while reducing overall healthcare costs. There are certain chatbots that function as medical encyclopedias; educating patients about their conditions and helping them take better care of themselves. In the proposed project, a chatbot system converses with patients about their health issues via voice-to-voice and text- to-text and provides a personalized diagnosis based on their symptoms. In addition to text assistance, this chatbot also offers an audio assistance feature that makes it easier for disabled (visually impaired) patients to access the system. This feature eliminates the restrictions that visually impaired patients face with the currently available text-enabled healthcare chat bots. We used the user's current location to locate the specialists nearby and present the information about them. Users can schedule appointments with various physicians in their area. As a result, people will be aware of their health status, receive rapid treatment, and be informed about the best medicines.

An innovative GPT-based open-source intelligence using historical cyber incident reports

In contemporary discourse, the pervasive influences of Generative Pre-Trained (GPT) and Large Language Models (LLM) are evident, showcasing diverse applications. GPT-based technologies, transcending mere summarization, exhibit adeptness in discerning critical information from extensive textual corpuses. Through prudent extraction of semantically meaningful content from textual representations, GPT technologies engender automated feature extraction, a departure from the fallible manual extraction methodologies. This study posits an innovative paradigm for extracting multidimensional cyber threat-related features from textual depictions of cyber events, leveraging the prowess of GPT. These extracted features serve as inputs for artificial intelligence (AI) and deep learning algorithms, including Convolutional Neural Network (CNN), Decomposition analysis, and Natural Language Processing (NLP)-based modalities tailored for non-technical cyber strategists. The proposed framework empowers cyber strategists or analysts to articulate inquiries regarding historical cyber incidents in plain English, with the NLP-based interaction facet of the system proffering cogent AI-driven insights in natural language. Furthermore, salient insights, often elusive in dynamic visualizations, are succinctly presented in plain language. Empirical validation of the entire system ensued through autonomous acquisition of semantically enriched contextual information concerning 214 major cyber incidents spanning from 2016 to 2023. GPT-based responses on Actor Type, Target, Attack Source (i.e., Country Originating Attack), Attack Destination (i.e., Targeted Country), Attack Level, Attack Type, and Attack Timeline, underwent critical AI-driven analysis. This comprehensive 7-dimensional information gleaned from the corpus of 214 incidents yielded a corpus of 1498 informative outputs, attaining a commendable precision of 96%, a recall rate of 98%, and an F1-Score of 97%.

Morphosyntactic prediction in automatic neural processing of spoken language: EEG evidence

Automatic parsing of syntactic information by the human brain is a well-established phenomenon, but its mechanisms remain poorly understood. Its best-known neurophysiological reflection is the so-called early left-anterior negativity (ELAN) component of event-related potentials (ERPs), with two alternative hypotheses for its origin: (1) error detection, or (2) morphosyntactic prediction/priming. To test these alternatives, we conducted two experiments using a non-attend passive design with visual distraction and recorded ERPs to spoken pronoun-verb phrases with/without agreement violations and to the same critical verbs presented in isolation without preceding pronouns. The results revealed an ELAN at ∼130-220 ms for pronoun-verb gender agreement violations, confirming a high degree of automaticity in early morphosyntactic parsing. Critically, the strongest ELAN was elicited by verbs outside phrasal context, which suggests that the typical ELAN pattern is underpinned by a reduction of ERP amplitudes for felicitous combinations, reflecting syntactic priming/predictability between related words/morphemes (potentially mediated by associative links formed during previous linguistic experience) rather than specialised error-detection processes.

Speech -To -Text Translation Using Hugging Face Model

The internet has revolutionized communication, but it's particularly challenging for those with hearing impairments. A multilingual speech-to-text conversion system using Hugging Face algorithms is being developed to assist these individuals in various tasks. This system, which uses advanced neural network models and natural language processing algorithms, will translate spoken language into text, enabling seamless communication and engagement in various contexts, thereby enhancing the quality of life for those with hearing impairments. Keywords—Speech-to-Text (STT)

The Role of Artificial Intelligence in Male Infertility: Evaluation and Treatment: A Narrative Review

Male infertility has affected an increasingly large population over the past few decades, affecting over 186 million people globally. The advent of assisted reproductive technologies (ARTs) and artificial intelligence (AI) has changed the landscape of diagnosis and treatment of male infertility. Through an extensive literature review encompassing the PubMed, Google Scholar, and Scopus databases, various AI techniques such as machine learning (ML), artificial neural networks (ANNs), deep learning (DL), and natural language processing (NLP) were examined in the context of evaluating seminal quality, predicting fertility potential, and improving semen analysis. Research indicates that AI models can accurately estimate the quality of semen, diagnose problems with sperm, and provide guidance on reproductive health decisions. In addition, developments in smartphone-based semen analyzers and computer-assisted semen analysis (CASA) are indicative of initiatives to improve the price, portability, and accuracy of results. Future directions point to possible uses for AI in ultrasonography assessment, microsurgical testicular sperm extraction (microTESE), and home-based semen analysis. Overall, AI holds significant promise in revolutionizing the diagnosis and treatment of male infertility, offering standardized, objective, and efficient approaches to addressing this global health challenge.

Early identification of suspected serious infection among patients afebrile at initial presentation using neural network models and natural language processing: A development and external validation study in the emergency department

ObjectiveTo develop and externally validate models based on neural networks and natural language processing (NLP) to identify suspected serious infections in emergency department (ED) patients afebrile at initial presentation. MethodsThis retrospective study included adults who visited the ED afebrile at initial presentation. We developed four models based on artificial neural networks to identify suspected serious infection. Patient demographics, vital signs, laboratory test results and information extracted from initial ED physician notes using term frequency-inverse document frequency were used as model variables. Models were trained and internally validated with data from one hospital and externally validated using data from a different hospital. Model discrimination was evaluated using area under the receiver operating characteristic curve (AUC) and 95% confidence intervals (CIs). ResultsThe training, internal validation, and external validation datasets comprised 150,699, 37,675, and 85,098 patients, respectively. The AUCs (95% CIs) for Models 1 (demographics + vital signs), 2 (demographics + vital signs + initial ED physician note), 3 (demographics + vital signs + laboratory tests), and 4 (demographics + vital signs + laboratory tests + initial ED physician note) in the internal validation dataset were 0.789 (0.782–0.796), 0.867 (0.862–0.872), 0.881 (0.876–0.887), and 0.911 (0.906–0.915), respectively. In the external validation dataset, the AUCs (95% CIs) of Models 1, 2, 3, and 4 were 0.824 (0.817–0.830), 0.895 (0.890–0.899), 0.879 (0.873–0.884), and 0.913 (0.909–0.917), respectively. Model 1 can be utilized immediately after ED triage, Model 2 can be utilized after the initial physician notes are recorded (median time from ED triage: 28 min), and Models 3 and 4 can be utilized after the initial laboratory tests are reported (median time from ED triage: 68 min). ConclusionsWe developed and validated models to identify suspected serious infection in the ED. Extracted information from initial ED physician notes using NLP contributed to increased model performance, permitting identification of suspected serious infection at early stages of ED visits.

Improving Automated Program Repair with Domain Adaptation

Automated Program Repair (APR) is defined as the process of fixing a bug/defect in the source code, by an automated tool. APR tools have recently experienced promising results by leveraging state-of-the-art Neural Language Processing (NLP) techniques. APR tools such as TFix and CodeXGLUE that combine text-to-text transformers with software-specific techniques are outperforming alternatives, these days. However, in most APR studies, the train and test sets are chosen from the same set of projects (i.e., when APR fixes a bug in the test set from project A, the model has already seen example fixed bugs from project A in the training set). In the real world, however, APR models are meant to be generalizable to new and different projects. Therefore, there is a potential threat that reported APR models with high effectiveness perform poorly when the characteristics of the new project or its bugs are different than the training set’s (“Domain Shift”). In this study, we first define the problem of domain shift in automated program repair. Next, we measure the potential damage of domain shift on two recent APR models (TFix and CodeXGLUE). Based on this observation, we then propose a domain adaptation framework that can adapt an APR model for a given target project. We conduct an empirical study with three domain adaptation methods FullFineTuning , TuningWithLightWeightAdapterLayers , and CurriculumLearning and two APR models on 2,672 bugs from 12 projects. The results show that our proposed framework on average can improve the effectiveness of TFix by 13.05% and CodeXGLUE by 48.78%, in terms of “Exact Match”. Through experiments, we also show that the framework provides high efficiency and reliability (in terms of “Exposure Bias”). Using synthetic data to domain adapt TFix and CodeXGLUE on the projects with no data (Zero-shot learning), also results in an average improvement of 5.76% and 17.62% for TFix and CodeXGLUE, respectively.

Neural natural language processing for long texts: A survey on classification and summarization

The adoption of Deep Neural Networks (DNNs) has greatly benefited Natural Language Processing (NLP) during the past decade. However, the demands of long document analysis are quite different from those of shorter texts, while the ever increasing size of documents uploaded online renders automated understanding of lengthy texts a critical issue. Relevant applications include automated Web mining, legal document review, medical records analysis, financial reports analysis, contract management, environmental impact assessment, news aggregation, etc. Despite the relatively recent development of efficient algorithms for analyzing long documents, practical tools in this field are currently flourishing. This article serves as an entry point into this dynamic domain and aims to achieve two objectives. First of all, it provides an introductory overview of the relevant neural building blocks, serving as a concise tutorial for the field. Secondly, it offers a brief examination of the current state-of-the-art in two key long document analysis tasks: document classification and document summarization. Sentiment analysis for long texts is also covered, since it is typically treated as a particular case of document classification. Consequently, this article presents an introductory exploration of document-level analysis, addressing the primary challenges, concerns, and existing solutions. Finally, it offers a concise definition of “long text/document”, presents an original overarching taxonomy of common deep neural methods for long document analysis and lists publicly available annotated datasets that can facilitate further research in this area.

Advanced Techniques for Geospatial Referencing in Online Media Repositories

In the digital transformation era, video media libraries’ untapped potential is immense, restricted primarily by their non-machine-readable nature and basic search functionalities limited to standard metadata. This study presents a novel multimodal methodology that utilizes advances in artificial intelligence, including neural networks, computer vision, and natural language processing, to extract and geocode geospatial references from videos. Leveraging the geospatial information from videos enables semantic searches, enhances search relevance, and allows for targeted advertising, particularly on mobile platforms. The methodology involves a comprehensive process, including data acquisition from ARD Mediathek, image and text analysis using advanced machine learning models, and audio and subtitle processing with state-of-the-art linguistic models. Despite challenges like model interpretability and the complexity of geospatial data extraction, this study’s findings indicate significant potential for advancing the precision of spatial data analysis within video content, promising to enrich media libraries with more navigable, contextually rich content. This advancement has implications for user engagement, targeted services, and broader urban planning and cultural heritage applications.

Revolution of Medical Review: The Application of Meta-Analysis and Convolutional Neural Network-Natural Language Processing in Classifying the Literature for Head and Neck Cancer Radiotherapy.

This study explored the application of meta-analysis and convolutional neural network-natural language processing (CNN-NLP) technologies in classifying literature concerning radiotherapy for head and neck cancer. It aims to enhance both the efficiency and accuracy of literature reviews. By integrating statistical analysis with deep learning, this research successfully identified key studies related to the probability of normal tissue complications (NTCP) from a vast corpus of literature. This demonstrates the advantages of these technologies in recognizing professional terminology and extracting relevant information. The findings not only improve the quality of literature reviews but also offer new insights for future research on optimizing medical studies through AI technologies. Despite the challenges related to data quality and model generalization, this work provides clear directions for future research.