Deep Learning-based Natural Language Processing Research Articles

Deep Learning-Based Natural Language Processing in Radiology: The Impact of Report Complexity, Disease Prevalence, Dataset Size, and Algorithm Type on Model Performance

In radiology, natural language processing (NLP) allows the extraction of valuable information from radiology reports. It can be used for various downstream tasks such as quality improvement, epidemiological research, and monitoring guideline adherence. Class imbalance, variation in dataset size, variation in report complexity, and algorithm type all influence NLP performance but have not yet been systematically and interrelatedly evaluated. In this study, we investigate these factors on the performance of four types [a fully connected neural network (Dense), a long short-term memory recurrent neural network (LSTM), a convolutional neural network (CNN), and a Bidirectional Encoder Representations from Transformers (BERT)] of deep learning-based NLP. Two datasets consisting of radiologist-annotated reports of both trauma radiographs (n = 2469) and chest radiographs and computer tomography (CT) studies (n = 2255) were split into training sets (80%) and testing sets (20%). The training data was used as a source to train all four model types in 84 experiments (Fracture-data) and 45 experiments (Chest-data) with variation in size and prevalence. The performance was evaluated on sensitivity, specificity, positive predictive value, negative predictive value, area under the curve, and F score. After the NLP of radiology reports, all four model-architectures demonstrated high performance with metrics up to > 0.90. CNN, LSTM, and Dense were outperformed by the BERT algorithm because of its stable results despite variation in training size and prevalence. Awareness of variation in prevalence is warranted because it impacts sensitivity and specificity in opposite directions.

Deep learning-based natural language processing in ophthalmology: applications, challenges and future directions.

Artificial intelligence (AI) is the fourth industrial revolution in mankind's history. Natural language processing (NLP) is a type of AI that transforms human language, to one that computers can interpret and process. NLP is still in the formative stages of development in healthcare, with promising applications and potential challenges in its applications. This review provides an overview of AI-based NLP, its applications in healthcare and ophthalmology, next-generation use case, as well as potential challenges in deployment. The integration of AI-based NLP systems into existing clinical care shows considerable promise in disease screening, risk stratification, and treatment monitoring, amongst others. Stakeholder collaboration, greater public acceptance, and advancing technologies will continue to shape the NLP landscape in healthcare and ophthalmology. Healthcare has always endeavored to be patient centric and personalized. For AI-based NLP systems to become an eventual reality in larger-scale applications, it is pertinent for key stakeholders to collaborate and address potential challenges in application. Ultimately, these would enable more equitable and generalizable use of NLP systems for the betterment of healthcare and society.

Early prediction of diagnostic-related groups and estimation of hospital cost by processing clinical notes

As healthcare providers receive fixed amounts of reimbursement for given services under DRG (Diagnosis-Related Groups) payment, DRG codes are valuable for cost monitoring and resource allocation. However, coding is typically performed retrospectively post-discharge. We seek to predict DRGs and DRG-based case mix index (CMI) at early inpatient admission using routine clinical text to estimate hospital cost in an acute setting. We examined a deep learning-based natural language processing (NLP) model to automatically predict per-episode DRGs and corresponding cost-reflecting weights on two cohorts (paid under Medicare Severity (MS) DRG or All Patient Refined (APR) DRG), without human coding efforts. It achieved macro-averaged area under the receiver operating characteristic curve (AUC) scores of 0·871 (SD 0·011) on MS-DRG and 0·884 (0·003) on APR-DRG in fivefold cross-validation experiments on the first day of ICU admission. When extended to simulated patient populations to estimate average cost-reflecting weights, the model increased its accuracy over time and obtained absolute CMI error of 2·40 (1·07%) and 12·79% (2·31%), respectively on the first day. As the model could adapt to variations in admission time, cohort size, and requires no extra manual coding efforts, it shows potential to help estimating costs for active patients to support better operational decision-making in hospitals.

Phishing websites detection via CNN and multi-head self-attention on imbalanced datasets

Phishing websites belong to a social engineering attack where perpetrators fake legitimate websites to lure people to access so as to illegally acquire user’s identity, password, privacy and even properties. This attack imposes a great threat to people and becomes more and more severe. In order to identify phishing websites, many proposals have shown their merits. For example, the classical proposal CNN-LSTM received a very high precision by combining Convolutional Neural Network (CNN) and Long Short Term Memory (LSTM) together. However, despite CNN achieved great success in AI area, LSTM still exists the biases issue since it always treats the later features much more important than the former ones. In the meanwhile, as the self-attention mechanism can discover the text’s inner dependency relationships, it has been widely applied to various tasks of deep learning-based Natural Language Processing (NLP). If we treat a URL as a text string, this mechanism can learn comprehensive URL representations. In order to improve the accuracy for phishing websites detection further, in this paper, we propose a novel Convolutional Neural Network (CNN) with self-attention named self-attention CNN for phishing Uniform Resource Locators (URLs) identification. Specifically, self-attention CNN first leverages Generative Adversarial Network (GAN) to generate phishing URLs so as to balance the datasets of legitimate and phishing URLs. Then it utilizes CNN and multi-head self-attention to construct our new classifier which is comprised of four blocks, namely the input block, the attention block, the feature block and the output block. Finally, the trained classifier can give a high-accuracy result for an unknown website URL. Overall thorough experiments indicate that self-attention CNN achieves 95.6% accuracy, which outperforms CNN-LSTM, single CNN and single LSTM by 1.4%, 4.6% and 2.1% respectively.

Deep Learning-based Natural Language Processing Methods Comparison for Presumptive Detection of Cyberbullying in Social Networks

Due to TIC development in the last years, users have managed to satisfy many social experiences through several digital media like blogs, web and especially social networks. However, not all social media users have had good experiences with these media. Since there are malicious users that are able to cause negative psychological effects over people, this is called cyberbullying. For this reason, social networks such as Twitter are looking to implement models based on deep learning or machine learning capable of recognizing harassing comments on their platforms. However, most of these models are focused on the use of English language and there are very few models adapted for Spanish language. This is why, in this paper we propose the evaluation of an RNN+LSTM neural network, as well as a BERT model through sentiment analysis, to perform the detection of cyberbullying based on Spanish language for Ecuadorian accounts of the social network Twitter. The results obtained show a balance between execution time and accuracy obtained for the RNN + LSTM model. In addition, evaluations of comments that are not explicitly offensive show a better performance for the BERT model, which outperforms its counterpart by 20%.

Online critical review classification in response strategy and service provider rating: Algorithms from heuristic processing, sentiment analysis to deep learning

This research proposes and tests mechanisms for defining and identifying the critical online consumer reviews that firms could prioritize to optimize their online response strategies, while incorporating the latest artificial intelligence (AI) technology to deal with the overwhelming volume of information. Three sets of analytical tools are introduced: Heuristic Processing, Linguistic Feature Analysis, and Deep Learning-based Natural Language Processing (NLP), to extract review information. Twelve algorithms to classify critical reviews were developed accordingly and empirically tested for their effectiveness. Our econometric analysis of 110,146 online reviews from a chain operation in hospitality industry over seven years identifies six outstanding algorithms. Firm value rating, comment length, valence, and certain consumer emotions, in addition to past comment-response behavior, are found to be superior in predicting incoming review criticality. However, the service attributes such as urgency to reply and the feasibility of actions to take are not as informative.

Understanding spatial language in radiology: Representation framework, annotation, and spatial relation extraction from chest X-ray reports using deep learning.

Radiology reports contain a radiologist's interpretations of images, and these images frequently describe spatial relations. Important radiographic findings are mostly described in reference to an anatomical location through spatial prepositions. Such spatial relationships are also linked to various differential diagnoses and often described through uncertainty phrases. Structured representation of this clinically significant spatial information has the potential to be used in a variety of downstream clinical informatics applications. Our focus is to extract these spatial representations from the reports. For this, we first define a representation framework based on the Spatial Role Labeling (SpRL) scheme, which we refer to as Rad-SpRL. In Rad-SpRL, common radiological entities tied to spatial relations are encoded through four spatial roles: Trajector, Landmark, Diagnosis, and Hedge, all identified in relation to a spatial preposition (or Spatial Indicator). We annotated a total of 2,000 chest X-ray reports following Rad-SpRL. We then propose a deep learning-based natural language processing (NLP) method involving word and character-level encodings to first extract the Spatial Indicators followed by identifying the corresponding spatial roles. Specifically, we use a bidirectional long short-term memory (Bi-LSTM) conditional random field (CRF) neural network as the baseline model. Additionally, we incorporate contextualized word representations from pre-trained language models (BERT and XLNet) for extracting the spatial information. We evaluate both gold and predicted Spatial Indicators to extract the four types of spatial roles. The results are promising, with the highest average F1 measure for Spatial Indicator extraction being 91.29 (XLNet); the highest average overall F1 measure considering all the four spatial roles being 92.9 using gold Indicators (XLNet); and 85.6 using predicted Indicators (BERT pre-trained on MIMIC notes). The corpus is available in Mendeley at http://dx.doi.org/10.17632/yhb26hfz8n.1 and https://github.com/krobertslab/datasets/blob/master/Rad-SpRL.xml.

Deep learning-based extraction of predicate-argument structure (PAS) in building design rule sentences☆

Abstract This paper describes an approach to extracting a predicate-argument structure (PAS) in building design rule sentences using natural language processing (NLP) and deep learning models. For the computer to reason about the compliance of building design, design rules represented by natural language must be converted into a computer-readable format. The rule interpretation and translation processes are challenging tasks because of the vagueness and ambiguity of natural language. Many studies have proposed approaches to address this problem, but most of these are dependent on manual tasks, which is the bottleneck to expanding the scope of design rule checking to design requirements from various documents. In this paper, we apply deep learning-based NLP techniques for translating design rule sentences into a computer-readable data structure. To apply deep learning-based NLP techniques to the rule interpretation process, we identified the semantic role elements of building design requirements and defined a PAS for design rule checking. Using a bidirectional long short-term memory model with a conditional random field layer, the computer can intelligently analyze constituents of building design rule sentences and automatically extract the logical elements. The proposed approach contributes to broadening the scope of building information modeling-enabled rule checking to any natural language-based design requirements.

Using deep learning and visual analytics to explore hotel reviews and responses

Abstract This study aims to use computational linguistics, visual analytics, and deep learning techniques to analyze hotel reviews and responses collected on TripAdvisor and to identify response strategies. To this end, we collected and analyzed 113,685 hotel reviews and responses and their semantic and syntactic relations. We are among the first to use visual analytics and deep learning-based natural language processing to empirically identify managerial responses. The empirical results indicate that our proposed multi-feature fusion, convolutional neural network model can make different types of data complement each other, thereby outperforming the comparisons. The visualization results can also be used to improve the performance of the proposed model and provide insights into response strategies, which further shows the theoretical and technical contributions of this study.

Testing Contextualized Word Embeddings to Improve NER in Spanish Clinical Case Narratives

In the Big Data era, there is an increasing need to fully exploit and analyze the huge quantity of information available about health. Natural Language Processing (NLP) technologies can contribute by extracting relevant information from unstructured data contained in Electronic Health Records (EHR) such as clinical notes, patients’ discharge summaries and radiology reports. The extracted information can help in health-related decision making processes. The Named Entity Recognition (NER) task, which detects important concepts in texts (e.g., diseases, symptoms, drugs, etc.), is crucial in the information extraction process yet has received little attention in languages other than English. In this work, we develop a deep learning-based NLP pipeline for biomedical entity extraction in Spanish clinical narratives. We explore the use of contextualized word embeddings, which incorporate context variation into word representations, to enhance named entity recognition in Spanish language clinical text, particularly of pharmacological substances, compounds, and proteins. Various combinations of word and sense embeddings were tested on the evaluation corpus of the PharmacoNER 2019 task, the Spanish Clinical Case Corpus (SPACCC). This data set consists of clinical case sections extracted from open access Spanish-language medical publications. Our study shows that our deep-learning-based system with domain-specific contextualized embeddings coupled with stacking of complementary embeddings yields superior performance over a system with integrated standard and general-domain word embeddings. With this system, we achieve performance competitive with the state-of-the-art.

딥러닝 기반 자연어 처리에서 도메인 지식의 역할

딥러닝 기반 자연어 처리에서 도메인 지식의 역할

Deep Learning Natural Language Processing Successfully Predicts the Cerebrovascular Cause of Transient Ischemic Attack-Like Presentations.

Background and Purpose- Triaging of referrals to transient ischemic attack (TIA) clinics is aided by risk stratification. Deep learning-based natural language processing, a type of machine learning, may be able to assist with the prediction of cerebrovascular cause of TIA-like presentations from free-text information. Methods- Consecutive TIA clinic notes were retrieved from existing databases. Texts associated with cerebrovascular and noncerebrovascular diagnoses were preprocessed before classification experiments, using a variety of classifier models, based on only the free-text description of the history of presenting complaint. The primary outcome was area under the curve (AUC) of the receiver operator curve. The model with the greatest AUC was then used in classification experiments in which it was provided with additional clinical information. Results- Of the classifier models trialed on the history of presenting complaint, the convolutional neural network achieved the greatest predictive capability (AUC±SD; 81.9±2.0). The effects of additional clinical information on AUC were variable. The greatest AUC was achieved when the convolutional neural network was provided with the history of presenting complaint and magnetic resonance imaging report (88.3±3.6). Conclusions- Deep learning-based natural language processing, in particular convolutional neural networks, based on medical free-text, may prove effective in prediction of the cause of TIA-like presentations. Future research investigating the role of the application of deep learning-based natural language processing to the automated triaging of clinic referrals in TIA, and potentially other specialty areas, is indicated.

Deep Learning and Natural Language Processing

The field of natural language processing (NLP) has seen rapid advances in the past several years since the introduction of deep learning techniques. A variety of NLP tasks including syntactic parsing, machine translation, and summarization can now be performed by relatively simple combinations of general neural network models such as recurrent neural networks and attention mechanisms. This manuscript gives a brief introduction to deep learning and an overview of the current deep learning-based NLP technology.

Wave2Vec: Vectorizing Electroencephalography Bio-Signal for Prediction of Brain Disease.

Interest in research involving health-medical information analysis based on artificial intelligence, especially for deep learning techniques, has recently been increasing. Most of the research in this field has been focused on searching for new knowledge for predicting and diagnosing disease by revealing the relation between disease and various information features of data. These features are extracted by analyzing various clinical pathology data, such as EHR (electronic health records), and academic literature using the techniques of data analysis, natural language processing, etc. However, still needed are more research and interest in applying the latest advanced artificial intelligence-based data analysis technique to bio-signal data, which are continuous physiological records, such as EEG (electroencephalography) and ECG (electrocardiogram). Unlike the other types of data, applying deep learning to bio-signal data, which is in the form of time series of real numbers, has many issues that need to be resolved in preprocessing, learning, and analysis. Such issues include leaving feature selection, learning parts that are black boxes, difficulties in recognizing and identifying effective features, high computational complexities, etc. In this paper, to solve these issues, we provide an encoding-based Wave2vec time series classifier model, which combines signal-processing and deep learning-based natural language processing techniques. To demonstrate its advantages, we provide the results of three experiments conducted with EEG data of the University of California Irvine, which are a real-world benchmark bio-signal dataset. After converting the bio-signals (in the form of waves), which are a real number time series, into a sequence of symbols or a sequence of wavelet patterns that are converted into symbols, through encoding, the proposed model vectorizes the symbols by learning the sequence using deep learning-based natural language processing. The models of each class can be constructed through learning from the vectorized wavelet patterns and training data. The implemented models can be used for prediction and diagnosis of diseases by classifying the new data. The proposed method enhanced data readability and intuition of feature selection and learning processes by converting the time series of real number data into sequences of symbols. In addition, it facilitates intuitive and easy recognition, and identification of influential patterns. Furthermore, real-time large-capacity data analysis is facilitated, which is essential in the development of real-time analysis diagnosis systems, by drastically reducing the complexity of calculation without deterioration of analysis performance by data simplification through the encoding process.