Natural Language Processing Results Research Articles

The Relationship between Interpreting Anxiety and Learning Motivation Assisted by Long Short-Term Memory Neural Network Algorithms

The Long Short-Term Memory (LSTM) neural network algorithm has achieved significant results in natural language processing and speech recognition. This study explores the application of the LSTM algorithm in the study of the relationship between interpreting anxiety and learning motivation. The article mainly uses questionnaire survey method to obtain research data. The participates include students engaged in translation research and professional translators. After statistical analysis of the research data, it was found that there was a significant correlation between translation anxiety and learning motivation (r=-0.62, p<0.001). According to calculation results, the average score is 3.42 and the standard deviation is 0.76. The average score of evaluating the learning enthusiasm of participants through corresponding questionnaires is 4.27, with an SD of 0.92. Further regression analysis of the data was conducted, and the results showed that respondents with low translation anxiety had stronger learning motivation（β=- 0.43, p<0.001). It can be seen that reducing learners' fear of translation can enhance their learning motivation, thereby stimulating their interest in learning. This indicates that reducing translation anxiety may be helpful to improve the learning motivation level of interpreting learners. The study is of great significance for improving the learning enthusiasm of translation learners and reducing translation anxiety and provides a new perspective and research method for future related research.

A lightweight Transformer‐based neural network for large‐scale masonry arch bridge point cloud segmentation

AbstractTransformer architecture based on the attention mechanism achieves impressive results in natural language processing (NLP) tasks. This paper transfers the successful experience to a 3D point cloud segmentation task. Inspired by newly proposed 3D Transformer neural networks, this paper introduces a new Transformer‐based module, which is called Local Geo‐Transformer. To alleviate the heavy memory consumption of the original Transformer, Local Geo‐Transformer only performs the attention mechanism in local regions. It is designed to mitigate the information loss caused by the subsampling of point clouds for segmentation. Global Geo‐Transformer is proposed to exploit the global relationships between features with the lowest resolution. The new architecture is validated on a masonry bridge dataset developed by the authors for their earlier work on a previous segmentation network called BridgeNet. The new version of the network with Transformer architecture, BridgeNetv2, outperforms BridgeNet in all metrics. BridgeNetv2 is also shown to be lightweight and memory efficient, well‐adapted to large‐scale point cloud segmentation tasks in civil engineering.

Evaluation of Machine Learning Approach for Sentiment Analysis using Yelp Dataset

Due to the abundance of text data representing public opinion, the Sentiment Analysis study is getting more and more important. Various techniques and methods have been proposed to address the issues. One of those techniques is deep learning algorithms which have been used to achieve great results in Natural Language Processing (NLP) applications. Sentiment Analysis is a part of NLP application that extracts emotional information from texts. In this study, we investigate the performance of sequence-based model, i.e., LSTM, compared with multi-layer perceptron Neural Network (NN) to classify the polarity of the text review based on negative or positive. The dataset used in this study is a restaurant review taken from the Yelp website. The dataset is trained using Word2vec word embedding to convert words contained in the dataset into numerical vector representation which is used as the deep learning model input. Based on the experiment results, it is shown that the LSTM model is outperformed compared to the multi-layer NN model. The best accuracy performance provided by LSTM model is 91%, whereas the best accuracy performance of multi-layer NN model is 76%.

MY ChatGPT TEACHER. TOWARDS A METHODOLOGY FOR USING ARTIFICIAL INTELLIGENCE IN SELF-DIRECTED LEARNING

Language is the distinctive feature separating humans from the rest of the animal world. Therefore, throughout history, inventions and technologies that changed the way we use language have been crucial turning points. These include the invention of writing, which allowed the communication across space and time, the printing press, which facilitated the rapid and widespread dissemination of knowledge, and the advent of the Internet and mobile phone technology, which resulted in an explosion of communications and faster diffusion of knowledge. The latest revolutionary innovation in the field of language processing is ChatGPT, a generative AI model launched by OpenAI in November 2022, which produced spectacular results in natural language processing (NLP). It appears to be one of the most significant milestones in this field and has the potential to produce major changes in numerous domains, particularly in education. Starting from the available literature and from several experiments with ChatGPT, this paper outlines the main methodological requirements for using this instrument as an aid for self-directed learners.

Material transformers: deep learning language models for generative materials design

Pre-trained transformer language models (LMs) on large unlabeled corpus have produced state-of-the-art results in natural language processing, organic molecule design, and protein sequence generation. However, no such models have been applied to learn the composition patterns for the generative design of material compositions. Here we train a series of seven modern transformer models (GPT, GPT-2, GPT-Neo, GPT-J, BLMM, BART, and RoBERTa) for materials design using the expanded formulas of the ICSD, OQMD, and Materials Projects databases. Six different datasets with/out non-charge-neutral or EB samples are used to benchmark the generative design performances and uncover the biases of modern transformer models for the generative design of materials compositions. Our experiments show that the materials transformers based on causal LMs can generate chemically valid material compositions with as high as 97.61% to be charge neutral and 91.22% to be electronegativity balanced, which has more than six times higher enrichment compared to the baseline pseudo-random sampling algorithm. Our LMs also demonstrate high generation novelty and their potential in new materials discovery is proved by their capability to recover the leave-out materials. We also find that the properties of the generated compositions can be tailored by training the models with selected training sets such as high-bandgap samples. Our experiments also show that different models each have their own preference in terms of the properties of the generated samples and their running time complexity varies a lot. We have applied our materials transformers to discover a set of new materials as validated using density functional theory calculations. All our trained materials transformer models and code can be accessed freely at http://www.github.com/usccolumbia/MTransformer.

Comparison of Applicability of Startup Life Cycle Theories Based on Natural Language Processing

ABSTRACT In the economies of Eastern and Central Europe, startups are expected to become the engine of economic development. Based on interviews with the founders, we examine the extent to which startup life cycle theories (SLCT) can be synthesized and what different or identical narrative conceptual spaces can be drawn. Based on our natural language processing results, it can be concluded that SLCTs cannot be clearly synthesized in the case of semi-peripheral countries, as they draw different narrative spaces. SLCTs can adequately capture enterprises in the initial stages, but in the later stages, the practical applicability of the theories is reduced.

BERT Probe: A python package for probing attention based robustness evaluation of BERT models

<h2>Abstract</h2> Transformer models based on attention-based architectures have been significantly successful in establishing state-of-the-art results in natural language processing (NLP). However, recent work about adversarial robustness of attention-based models show that their robustness is susceptible to adversarial inputs causing spurious outputs thereby raising questions about trustworthiness of such models. In this paper, we present BERT Probe which is a python-based package for evaluating robustness to attention attribution based on character-level and word-level evasion attacks and empirically quantifying potential vulnerabilities for sequence classification tasks. Additionally, BERT Probe also provides two out-of-the-box defenses against character-level attention attribution-based evasion attacks.

White-box attacks on hate-speech BERT classifiers in German with explicit and implicit character-level defense

Attention-based transformer models have achieved state-of-the-art results in natural language processing (NLP). However, recent work shows that the underlying attention mechanism can be exploited by adversaries to craft malicious inputs designed to induce spurious outputs, thereby harming model performance and trustworthiness. Unlike in the vision domain, the literature examining neural networks under adversarial conditions in the NLP domain is limited and most of it focuses mainly on the English language. In this article, we first analyze the adversarial robustness of Bidirectional Encoder Representations from Transformers (BERT) models for German data sets. Second, we introduce two novel NLP attacks: a character-level and a word-level attacks, both of which utilize attention scores to calculate where to inject character-level and word-level noise, respectively. Finally, we present two defense strategies against the attacks above. The first implicit character-level defense is a variant of adversarial training, which trains a new classifier capable of abstaining/rejecting certain (ideally adversarial) inputs. The other explicit character-level defense learns a latent representation of the complete training data vocabulary and then maps all tokens of an input example to the same latent space, enabling the replacement of all out-of-vocabulary tokens with the most similar in-vocabulary tokens based on the cosine similarity metric.

Can MOOC Instructor Be Portrayed by Semantic Features? Using Discourse and Clustering Analysis to Identify Lecture-Style of Instructors in MOOCs

Nowadays, most courses in massive open online course (MOOC) platforms are xMOOCs, which are based on the traditional instruction-driven principle. Course lecture is still the key component of the course. Thus, analyzing lectures of the instructors of xMOOCs would be helpful to evaluate the course quality and provide feedback to instructors and researchers. The current study aimed to portray the lecture styles of instructors in MOOCs from the perspective of natural language processing. Specifically, 129 course transcripts were downloaded from two major MOOC platforms. Two semantic analysis tools (linguistic inquiry and word count and Coh-Metrix) were used to extract semantic features including self-reference, tone, effect, cognitive words, cohesion, complex words, and sentence length. On the basis of the comments of students, course video review, and the results of cluster analysis, we found four different lecture styles: “perfect,” “communicative,” “balanced,” and “serious.” Significant differences were found between the different lecture styles within different disciplines for notes taking, discussion posts, and overall course satisfaction. Future studies could use fine-grained log data to verify the results of our study and explore how to use the results of natural language processing to improve the lecture of instructors in both MOOCs and traditional classes.

White-Box Attacks on Hate-speech BERT Classifiers in German with Explicit and Implicit Character Level Defense

Attention based Transformer models have achieved state-of-the-art results in natural language processing (NLP). However, recent work shows that the underlying attention mechanism can be exploited by adversaries to craft malicious inputs designed to induce spurious outputs, thereby harming model performance and trustworthiness. Unlike in the vision domain, the literature examining neural networks under adversarial conditions in the NLP domain is limited and most of it focuses mainly on the English language. In this paper, we first analyze the adversarial robustness of Bidirectional Encoder Representations from Transformers (BERT) models for German datasets. Second, we introduce two novel NLP attacks. Namely, a character-level and a word-level attacks, both of which utilize attention scores to calculate where to inject character-level and word-level noise, respectively. Finally, we present two defense strategies against the attacks above. The first implicit character-level defense is a variant of adversarial training, which trains a new classifier capable of abstaining/rejecting certain (ideally adversarial) inputs. The other explicit character-level defense learns a latent representation of the complete training data vocabulary and then maps all tokens of an input example to the same latent space, enabling the replacement of all out of vocabulary tokens with the most similar in-vocabulary tokens based on the cosine similarity metric.

White-Box Attacks on Hate-speech BERT Classifiers in German with Explicit and Implicit Character Level Defense

Attention based Transformer models have achieved state-of-the-art results in natural language processing (NLP). However, recent work shows that the underlying attention mechanism can be exploited by adversaries to craft malicious inputs designed to induce spurious outputs, thereby harming model performance and trustworthi- ness. Unlike in the vision domain, the literature examining neural networks under adversarial conditions in the NLP domain is limited and most of it focuses mainly on the English language. In this paper, we first analyze the adver- sarial robustness of Bidirectional Encoder Representations from Transformers (BERT) models for German datasets. Second, we introduce two novel NLP attacks. Namely, a character-level and a word-level attacks, both of which uti- lize attention scores to calculate where to inject character-level and word-level noise, respectively. Finally, we present two defense strategies against the attacks above. The first implicit character-level defense is a variant of adversar- ial training, which trains a new classifier capable of abstaining/rejecting certain (ideally adversarial) inputs. The other explicit character-level defense learns a latent representation of the complete training data vocabulary and then maps all tokens of an input example to the same latent space, enabling the replacement of all out of vocabulary tokens with the most similar in-vocabulary tokens based on the cosine similarity metric.

Exploring the Influence of Focal Loss on Transformer Models for Imbalanced Maintenance Data in Industry 4.0

Abstract Harnessing data from historical maintenance databases may be challenging, as they tend to rely on text data provided by operators. Thus, they often include acronyms, jargon, typos, and other irregularities that complicate the automated analysis of such reports. Furthermore, maintenance datasets may present highly imbalanced distributions: some situations happen more often than others, which hinders the effective application of classic Machine Learning (ML) models. Hence, this paper explores the use of a recent Deep Learning (DL) architecture called Transformer, which has provided cutting-edge results in Natural Language Processing (NLP). To tackle the class imbalance, a loss function called Focal Loss (FL) is explored. Results suggests that when all the classes are equally important, the FL does not improve the classification performance. However, if the objective is to detect the minority class, the FL achieves the best performance, although by degrading the detection capacity for the majority class.

Investigation of Dual-Flow Deep Learning Models LSTM-FCN and GRU-FCN Efficiency against Single-Flow CNN Models for the Host-Based Intrusion and Malware Detection Task on Univariate Times Series Data

Intrusion and malware detection tasks on a host level are a critical part of the overall information security infrastructure of a modern enterprise. While classical host-based intrusion detection systems (HIDS) and antivirus (AV) approaches are based on change monitoring of critical files and malware signatures, respectively, some recent research, utilizing relatively vanilla deep learning (DL) methods, has demonstrated promising anomaly-based detection results that already have practical applicability due low false positive rate (FPR). More complex DL methods typically provide better results in natural language processing and image recognition tasks. In this paper, we analyze applicability of more complex dual-flow DL methods, such as long short-term memory fully convolutional network (LSTM-FCN), gated recurrent unit (GRU)-FCN, and several others, for the task specified on the attack-caused Windows OS system calls traces dataset (AWSCTD) and compare it with vanilla single-flow convolutional neural network (CNN) models. The results obtained do not demonstrate any advantages of dual-flow models while processing univariate times series data and introducing unnecessary level of complexity, increasing training, and anomaly detection time, which is crucial in the intrusion containment process. On the other hand, the newly tested AWSCTD-CNN-static (S) single-flow model demonstrated three times better training and testing times, preserving the high detection accuracy.

Impact of Different Electronic Cohort Definitions to Identify Patients With Atrial Fibrillation From the Electronic Medical Record.

BackgroundElectronic medical records (EMRs) allow identification of disease‐specific patient populations, but varying electronic cohort definitions could result in different populations. We compared the characteristics of an electronic medical record–derived atrial fibrillation (AF) patient population using 5 different electronic cohort definitions.Methods and ResultsAdult patients with at least 1 AF billing code from January 1, 2010, to December 31, 2017, were included. Based on different electronic cohort definitions, we trained 5 different logistic regression models using a labeled training data set (n=786). Each model yielded a predicted probability; patients were classified as having AF if the probability was higher than a specified cut point. Test characteristics were calculated for each model. These models were then applied to the full cohort and resulting characteristics were compared. In the training set, the comprehensive model (including demographics, billing codes, and natural language processing results) performed best, with an area under the curve of 0.89, sensitivity of 0.90, and specificity of 0.87. Among a candidate population (n=22 000), the proportion of patients identified as having AF varied from 61% in the model using diagnosis or procedure International Classification of Diseases (ICD) billing codes to 83% in the model using natural language processing of clinical notes. Among identified AF patients, the proportion of patients with a CHA 2 DS 2‐VASc score ≥2 varied from 69% to 85%; oral anticoagulant treatment rates varied from 50% to 66% depending on the model.ConclusionsDifferent electronic cohort definitions result in substantially different AF study samples. This difference threatens the quality and reproducibility of electronic medical record–based research and quality initiatives.

A Review of Soft Computing Technique for Real-Time Data Forecasting

Weather forecasting attracted so many researchers from different research communities because of its tremendous effect on global human life. Mechanistic models rather than traditional numerical model are gaining more attention. Hence, it is of interest to implement a model which will be faster and more accurate than traditional meteorological models. The recent advancement in deep learning techniques achieved notable results in natural language processing, audio, video, image processing, etc. However, not much research has been yet conducted in such neural networks using real-time data. Hence, it is of interest to study whether deep neural network can be a good solution to predict weather parameters combined with enormously available weather time series data. This paper provides brief review about various attempts like Soft Computing, Machine Learning and Artificial Neural Network made for weather forecasting.

Diagnoses Detection in Short Snippets of Narrative Medical Texts

Data extraction from narrative medical texts is a significant task to enable secondary use of medical data. Supervised learning algorithms show good results in natural language processing (NLP) tasks. We have developed a NLP framework based on supervised machine learning for entity extraction from medical texts. The framework is language independent and entities independent as long as an appropriately labeled dataset is given. The framework is based on vector representation of words and a neural network as a classifier. We have trained and evaluated the framework on two different text corpuses: diagnoses paragraphs written in German and medical records written in Russian. The neural network hyperparameters were adjusted for every dataset to get better classification results. Finally, accuracy, standard deviation, and standard error were calculated for both network models engaging 10-folds cross-validation. The obtained accuracy is 97,64% for Russian texts and 96,81% for German ones.

Design of a generic, open platform for machine learning-assisted indexing and clustering of articles in PubMed, a biomedical bibliographic database

AbstractMany investigators have carried out text mining of the biomedical literature for a variety of purposes, ranging from the assignment of indexing terms to the disambiguation of author names. A common approach is to define positive and negative training examples, extract features from article metadata, and use machine learning algorithms. At present, each research group tackles each problem from scratch, in isolation of other projects, which causes redundancy and a great waste of effort. Here, we propose and describe the design of a generic platform for biomedical text mining, which can serve as a shared resource for machine learning projects and as a public repository for their outputs. We initially focus on a specific goal, namely, classifying articles according to publication type and emphasize how feature sets can be made more powerful and robust through the use of multiple, heterogeneous similarity measures as input to machine learning models. We then discuss how the generic platform can be extended to include a wide variety of other machine learning-based goals and projects and can be used as a public platform for disseminating the results of natural language processing (NLP) tools to end-users as well.

Application of Natural Language Processing for Information Retrieval

Through a comprehensive analysis of using natural language processing in information retrieval, we compared the effects with the various natural language techniques for information retrieval precision in this paper. This is for the tasks of more suitable as well as accurate results of natural language processing.

Automatic Detection of Metonymic Expressions Using Associative Relation between Words

Metonymy is a figure of speech, where one item's name represents another item which usually has a close relation with the first one. Metonymic expressions need to be correctly detected and interpreted because sentences including such expressions have different meanings from literal ones; computer systems may output inappropriate results in natural language processing.

Informatics in Radiology

Radiology departments are a rich source of information in the form of digital radiology reports and images obtained in patients with a wide spectrum of clinical conditions. A free text radiology report and image search application known as Render was created to allow users to find pertinent cases for a variety of purposes. Render is a radiology report and image repository that pools researchable information derived from multiple systems in near real time with use of (a) Health Level 7 links for radiology information system data, (b) periodic file transfers from the picture archiving and communication system, and (c) the results of natural language processing (NLP) analysis. Users can perform more structured and detailed searches with this application by combining different imaging and patient characteristics such as examination number; patient age, gender, and medical record number; and imaging modality. Use of NLP analysis allows a more effective search for reports with positive findings, resulting in the retrieval of more cases and terms having greater relevance. From the retrieved results, users can save images, bookmark examinations, and navigate to an external search engine such as Google. Render has applications in the fields of radiology education, research, and clinical decision support.