Recurrent Neural Network Framework Research Articles

CNSMolGen: A Bidirectional Recurrent Neural Network-Based Generative Model for De Novo Central Nervous System Drug Design.

Central nervous system (CNS) drugs have had a significant impact on treating a wide range of neurodegenerative and psychiatric disorders. In recent years, deep learning-based generative models have shown great potential for accelerating drug discovery and improving efficacy. However, specific applications of these techniques in CNS drug discovery have not been widely reported. In this study, we developed the CNSMolGen model, which uses a framework of bidirectional recurrent neural networks (Bi-RNNs) for de novo molecular design of CNS drugs. Results showed that the pretrained model was able to generate more than 90% of completely new molecular structures, which possessed the properties of CNS drug molecules and were synthesizable. In addition, transfer learning was performed on small data sets with specific biological activities to evaluate the potential application of the model for CNS drug optimization. Here, we used drugs against the classical CNS disease target serotonin transporter (SERT) as a fine-tuned data set and generated a focused database against the target protein. The potential biological activities of the generated molecules were verified by using the physics-based induced-fit docking study. The success of this model demonstrates its potential in CNS drug design and optimization, which provides a new impetus for future CNS drug development.

Deterministic convergence analysis for regularized long short-term memory and its application to regression and multi-classification problems

Long short-term memory (LSTM) is a recurrent neural network (RNN) framework designed to solve the gradient disappearance and explosion problems of traditional RNNs. In recent years, LSTM has become a state-of-the-art model for solving various machine-learning problems. This paper propose a novel regularized LSTM based on the batch gradient method. Specifically, the L2 regularization is appended to the objective function as a systematic external force, effectively controlling the excessive growth of weights in the network and preventing the overfitting phenomenon. In addition, a rigorous convergence analysis of the proposed method is carried out, i.e., monotonicity, weak convergence, and strong convergence results are obtained. Finally, comparative simulations are conducted on the benchmark data set for regression and classification problems, and the simulation results verify the effectiveness of the method.

Factorized Recurrent Neural Network with Attention for Language Identification and Content Detection

Language identification and content detection are essential for ensuring effective digital communication, and content moderation. While extensive research has primarily focused on well-known and widely spoken languages, challenges persist when dealing with indigenous and resource-limited languages, especially between closely similar languages such as Ethiopian languages. This article aims to simultaneously identify the language of a given text and detect its content, and to achieve this, we propose a novel attention-based recurrent neural network framework. The proposed method has an attention-embedded Bidirectional-LSTM architecture with two classifiers that identify the language of a given text and content within the text. The two classifiers share a common feature space before they branched at their task-specific layers where both layers are assisted by attention mechanism. We use five different topics in Six Ethiopian Languages the dataset consists of nearly 22,624 sentences. We compared our result with the classical NLP techniques, the proposed method shortened the data prepossessing steps. We evaluated the model performance using the accuracy metric, achieving results of 98.88% for language identification and 96.5% for text content detection. The dataset, source code, and pretrained model are available at https://github.com/bdu-birhanu/LID_TCD .

A data driven recurrent neural network approach for reproduction of variable visuo-haptic force feedback in surgical tool insertion

The surge of interest in haptic technology is due to inspirational advances in the robotic-assisted surgical system, where haptics has the role of delivering tactile feedback for enhancement of user experience. This work presents a Long Short Term Memory (LSTM) based Recurrent Neural Network (RNN) framework with Dimensionality Reduction (DR) and a Cyclical Learning Rate (CLR) optimizer for reproducing variable forces produced in different skin layers during the performance of various surgical procedures. This paper deals with online estimation of the force parameters of original porcine skin, and the same has been tested in real-time and Visuo-haptic environment for training surgeons. The proposed model has processed both spatial and temporal information acquired from three different dataset, surgical tools and manipulator. The results of proposed framework RNN-LSTM + DR + CLR show a 9.23 % & 3.8 % improvement in force prediction accuracy in real-time and 7.11 % & 1.68 % improvement in Visuo-haptic simulation compared to the RNN and RNN-LSTM prediction frameworks, respectively. The sensitivity analysis shows that torque (97.62 %), position (94.54 %), deformation (93.20 %), stiffness (89.23 %), tool diameter (87.25 %), rotation (63.21 %), and orientation (62.56 %) features have a respective impact on the predicted force. The performance of RNN-LSTM was better when the network was optimized with dimensionality reduction, loss function as Root Mean Square Error (RMSE), and learning rate as Cyclical Learning Rate (CLR). The research outcomes show the effectiveness of the method for estimating the force on the surface and internal layers of the skin. Also, the method has applications in real-time surgical tasks and surgeon training.

Transfer learning-based adaptive recursive neural network for short-term non-stationary building heating load prediction

Building energy consumption is a non-stationary time series, and its distribution law changes over time. Traditional machine-learning models are prone to model shift, which leads to a reduction of their prediction accuracy when applied to building energy consumption forecasting. Therefore, in this paper, an adaptive neural network framework is proposed for non-stationary building energy consumption prediction based on transfer learning, in which the training datasets are divided into the most dissimilar periods, and based on the transfer learning mechanism, the different periods of data are learned to obtain the minimum overall loss to improve the model generalization. Taking LSTM and GRU models as examples, adaptive long short-term memory (adaLSTM) and adaptive gated recurrent unit (adaGRU) building energy consumption prediction models are established. The models were trained and verified using heating load data from Xi'an, China. The results show that compared with LSTM model, the coefficient of determination, root mean square error, the coefficient of variation of the root mean squared error and mean absolute error of the adaLSTM model were improved by 0.61%, 37.78%, 38.05% and 30.69%, respectively, and the over-fitting degree was reduced by 227.7%. Compared with the traditional GRU model, the corresponding evaluation indexes of adaGRU were improved by 2.50%, 70.58%, 70.64% and 68.83%, respectively, and the over-fitting degree was improved by 505.7% points. The adaptive recurrent neural network framework proposed in this paper is a generalized approach which can be applied to other non-stationary time series prediction models.

Deep Learning-Based DAS to Geophone Data Transformation

Seismic data are the primary way to study the subsurface structure and properties. A conventional seismic sensor like geophone or accelerometer measures particle velocity or acceleration at a local point only, while distributed acoustic sensing (DAS) measures dynamic strain along the fiber optic cable at densely spaced sample points, where strain rate is obtained over certain gauge length interval. Therefore, DAS measures subsurface properties with high sampling resolution and large coverage. When an optical fiber is installed in a well, DAS can provide continuous, dense downhole recording. However, currently, most of the seismic processing, imaging, and inversion techniques are developed for geophone data. These well-established techniques can be readily and properly utilized if DAS data are transformed into geophone measurements, such as particle velocity. In this study, we present a recurrent neural network (RNN) framework to perform this transformation. This effectiveness of the deep learning-based mapping is then demonstrated with a field measurement data, showing that DAS data can be transformed into particle velocity accurately and robustly using the proposed deep-learning approach.

Wi-CL: Low-Cost WiFi-Based Detection System for Nonmotorized Traffic Travel Mode Classification

Traffic travel mode identification and classification are crucial for the development of intelligent transportation systems (ITSs). At present, scholars have investigated the classification of motorized and nonmotorized traffic travel in various road environments; however, the classification of walking and bicycle modes in nonmotorized travel has been largely ignored. Therefore, in this paper, we investigate nonmotorized traffic travel and propose a new low-cost nonmotorized traffic travel mode classification system, known as the Wi-Fi classification (Wi-CL) system that uses Wi-Fi signal detectors and the refined characteristics of nonmotorized travel modes. The Wi-CL system includes four modules: data acquisition module, data processing module, feature extraction module, and mode classification module. In the data acquisition module, the proposed system detects the Wi-Fi signals of traffic participants in road environments. In addition, we propose a received signal strength indicator (RSSI) filtering algorithm for hybrid traffic networks that effectively addresses surrounding obstacles and environmental noise. In the feature extraction module, we extract relevant traffic features to construct a mode classification model. Finally, a recurrent neural network (RNN) framework based on the long short-term memory (LSTM) algorithm is successfully implemented in the mode classification module for traffic travel mode identification. To validate the effectiveness of the Wi-CL system, extensive experiments were conducted using field data collected by Wi-Fi detectors installed at the South China University of Technology (SCUT). The experimental results show that the proposed RSSI filtering algorithm achieves excellent signal filtering results in real road traffic environments. In addition, the constructed travel speed estimation algorithm outperforms other baseline models in four different scenarios (flat-peak walking, midday peak walking, flat-peak cycling, and midday peak cycling), achieving an overall classification accuracy of 97.92%. In summary, our Wi-CL system is a feasible approach for nonmotorized traffic travel mode classification.

Machine learning for the prediction of safe and biologically active organophosphorus molecules

Drug discovery is a complex process with a large molecular space to be considered. By constraining the search space, the fragment based drug design is an approach that can effectively sample the chemical space of interest. Here we propose a framework of Recurrent Neural Networks (RNN) with an attention model to sample the chemical space of organophosphorus molecules using the fragment-based approach. The framework is trained with a ZINC dataset that is screened for high druglikeness scores. The goal is to predict molecules with similar biological action modes as organophosphorus pesticides or chemical warfare agents yet less toxic to humans. The generated molecules contain a starting fragment of PO2F but have a bulky hydrocarbon side chain limiting its binding effectiveness to the targeted protein.

Novel Heuristic Recurrent Neural Network Framework to Handle Automatic Telugu Text Categorization from Handwritten Text Image

In the near future, the digitization and processing of the current paper documents describe efficient role in the creation of a paperless environment. Deep learning techniques for handwritten recognition have been extensively studied by various researchers. Deep neural networks can be trained quickly thanks to a lot of data and other algorithmic advancements. Various methods for extracting text from handwritten manuscripts have been developed in literature. To extract features from written Telugu Text image having some other neural network approaches like convolution neural network (CNN), recurrent neural networks (RNN), long short-term memory (LSTM). Different deep learning related approaches are widely used to identification of handwritten Telugu Text; various techniques are used in literature for the identification of Telugu Text from documents. For automatic identification of Telugu written script efficiently to eliminate noise and other semantic features present in Telugu Text, in this paper, proposes Novel Heuristic Advanced Neural Network based Telugu Text Categorization Model (NHANNTCM) based on sequence-to-sequence feature extraction procedure. Proposed approach extracts the features using RNN and then represents Telugu Text in sequence-to-sequence format for the identification advanced neural network performs both encoding and decoding to identify and explore visual features from sequence of Telugu Text in input data. The classification accuracy rates for Telugu words, Telugu numerals, Telugu characters, Telugu sentences, and the corresponding Telugu sentences were 99.66%, 93.63%, 91.36%, 99.05%, and 97.73% consequently. Experimental evaluation describe extracted with revealed which are textured i.e. TENG shown considerable operations in applications such as private information protection, security defense, and personal handwriting signature identification.

Novel Generative Recurrent Neural Network Framework to Produce Accurate, Applicable, and Deidentified Synthetic Medical Data for Patients With Metastatic Cancer

Sensitive patient data cannot be easily shared/analyzed, severely limiting the innovative progress of research, specifically for marginalized/under-represented populations. Existing methods of deidentification are subject to data breaches. The objective of this study was to develop a neural network capable of generating a synthetic version of data for patients with novel postoperative metastatic cancer. We analyzed a metastatic cancer patient cohort of 167,474 patients obtained from the National Surgical Quality Improvement Program. Twenty-seven clinical features were analyzed. We created a volume-matched synthetic cohort of 167,474 patients and a reduced-size synthetic cohort of 5,000 patients. The volume-matched and reduced-size synthetic cohorts were compared against the ground truth data to analyze differences in principal component distribution, underlying statistical properties/associations, intervariable correlations, and machine learning classifier performance when developed on the synthetic data. Among 167,474 patients with metastatic cancer in the original data, 50,669 (30.3%) died within 30 days of their index surgery. Our model was able to accurately capture underlying statistical properties, principal components, and intervariable correlations within the ground truth data, yielding an accuracy of 93.2% with a loss of 0.21%, and develop synthetic data capable of training accurate machine learning classifiers. The reduced-size synthetic data accurately replicated all categorical variables and every continuous variable with statistically similar records (P > .05), with the sole exception of preoperative albumin (P < .05). The volume-matched synthetic data frame was able to accurately replicate all categorical variables (P > .05). This described methodology can be applied to any structured medical data from any setting, significantly expedite scientific analysis/innovation, and be used to develop improved predictive classifiers with boosted tree-based algorithms, serving as the potential new gold standard of medical data sharing and data augmentation.

Prediction of wing buffet pressure loads using a convolutional and recurrent neural network framework

AbstractIn the present study, a hybrid deep learning reduced-order model (ROM) is applied for the prediction of wing buffet pressure distributions on a civil aircraft configuration. The hybrid model is compound of a convolutional variational neural network autoencoder (CNN-VAR-AE) and a long short-term memory (LSTM) neural network. The CNN-VAR-AE is used for the reduction of the high-dimensional flow field data, whereas the LSTM is applied to predict the temporal evolution of the pressure distributions. For training the neural network, experimental buffet data obtained by unsteady pressure sensitive paint measurement (iPSP), is applied. As a test case, the Airbus XRF-1 configuration is selected, considering two different experimental setups. The first setup is defined by a wind tunnel model with a clean wing, whereas the second setup includes an ultra high bypass ratio engine nacelle on each wing. Both configurations have been tested in the European Transonic Windtunnel, considering several transonic buffet conditions. Finalizing the training of the hybrid neural networks, the trained models are applied for the prediction of buffet flow conditions which are not included in the training data set. A comparison of the experimental results and the pressure distributions predicted by the hybrid ROMs indicate a precise prediction performance. Considering both aircraft configurations, the main buffet flow features are captured by the hybrid ROMs.

Multitasking recurrent neural network for photovoltaic power generation prediction

With the increased uptake of renewable energy resources (RES), power generation from Photovoltaics (PV) cells is gaining momentum and heavily depends on environmental and meteorological conditions. Predicting PV power generation plays an important role in power management and dispatch optimization. Due to the scales of PV systems, the generation may vary significantly, which is difficult to have accurate PV power generation prediction over different categories of customers (e.g., residential, agricultural, industrial, and commercial) using a single model. In this paper, we define the PV power generation prediction as a multitasking prediction problem, where PV generation over each of the categories is modeled as a separate prediction task. To address this problem, we employ a recurrent neural network (RNN) as the predictor for each prediction task and propose a multitasking RNN (MT-RNN) framework. Instead of addressing each task individually, MT-RNN performs knowledge transfer among different tasks to improve the prediction accuracy of each task, where the knowledge is represented via connection weights and biases in each RNN. In comparison to several state-of-art deep neural network (DNN) models that solve each task individually, the superior performance of MT-RNN in terms of prediction accuracy is demonstrated.

A Hybrid CRNN Model for Multi-Class Violence Detection in Text and Video

Gender-based violence is a critical issue that not only poses a threat to physical safety but also has significant impacts on mental health. Shockingly, up to 1 billion children aged 2-17 years are estimated to have experienced gender-based violence globally, making it a pressing concern for the machine learning and deep learning communities to address. To end this, a novel approach has been proposed in the form of a Convolutional Neural Network and bi-directional LSTM (CRNN) to classify three types of violence present in both text and video data, thereby making the internet a safer space for individuals. The proposed approach utilises two datasets consisting of 400 and 600 samples each for videos and text, respectively, to improve the precision and accuracy of the model. The use of a Convolutional Recurrent Neural Network framework combined with LSTM layers has resulted in an accuracy of 97% on text and 96% on videos, surpassing the performance of existing RNN models. Additionally, the inclusion of dropout and regularizer layers has helped the model avoid overfitting and generalise better on unseen data. Overall, the CRNN-based approach presents a promising solution to the problem of gender-based violence detection, with the potential to significantly improve the safety of individuals online. By leveraging the power of machine learning and deep learning, we can contribute towards creating a safer and more equitable world for all.

Group-based bi-directional recurrent wavelet neural network for efficient video super-resolution (VSR)

Video super-resolution (VSR) is an important technology for enhancing the quality of video frames. The recurrent neural network (RNN)-based approach is suitable for sequential data because it can use accumulated temporal information. However, since existing methods only tend to capture slow and symmetrical motion with low frame rate, there are still limitations to restore the missing details for more dynamic motion. Most of the previous methods using spatial information treat different types of the spatial features identically. It leads to lack of obtaining meaningful information and enhancing the fine details. We propose a group-based bi-directional recurrent wavelet neural network (GBR-WNN) to exploit spatio-temporal information effectively. The proposed group-based bi-directional RNN (GBR) framework is built on the well-structured process with the group of pictures (GOP). In a GOP, we resolves the low-resolution (LR) frames from border frames to center target frame. Because super-resolved features in a GOP are cumulative, neighboring features are improved progressively and asymmetrical motion can be dealt with. Also, we propose a temporal wavelet attention (TWA) adopting attention module for both spatial and temporal features simultaneously based on discrete wavelet transform. Experiments show that the proposed scheme achieves superior performance compared with state-of-the-art methods.

Ensemble Support Vector Recurrent Neural Network for Brain Signal Detection.

The brain-computer interface (BCI) P300 speller analyzes the P300 signals from the brain to achieve direct communication between humans and machines, which can assist patients with severe disabilities to control external machines or robots to complete expected tasks. Therefore, the classification method of P300 signals plays an important role in the development of BCI systems and technologies. In this article, a novel ensemble support vector recurrent neural network (E-SVRNN) framework is proposed and developed to acquire more accurate and efficient electroencephalogram (EEG) signal classification results. First, we construct a support vector machine (SVM) to formulate EEG signals recognizing model. Second, the SVM formulation is transformed into a standard convex quadratic programming (QP) problem. Third, the convex QP problem is solved by combining a varying parameter recurrent neural network (VPRNN) with a penalty function. Experimental results on BCI competition II and BCI competition III datasets demonstrate that the proposed E-SVRNN framework can achieve accuracy rates as high as 100% and 99%, respectively. In addition, the results of comparison experiments verify that the proposed E-SVRNN possesses the best recognition accuracy and information transfer rate (ITR) compared with most of the state-of-the-art algorithms.

A Transferred Recurrent Neural Network for Battery Calendar Health Prognostics of Energy-Transportation Systems

Battery-based energy storage system is a key component to achieve low carbon industrial and social economy, where battery health status plays a vital role in determining the safety and reliability of energy-transportation nexus. This article proposes a transferred recurrent neural network (RNN)-based framework to achieve efficient calendar capacity prognostics under both witnessed and unwitnessed storage conditions. Specifically, this transferred RNN framework contains a base model part and a transfer model part. The base model is first trained by using the easily collected and time-saving accelerated ageing dataset from high temperature and state-of-charge (SOC) cases. Then the transfer part is tuned by using only a small portion of starting capacity data from unwitnessed condition of interest. The developed framework is evaluated under a well-rounded ageing dataset with three different storage SOCs (20%, 50%, and 90%) and temperatures (10 °C, 25 °C, and 45 °C). Experimental results demonstrate that the derived transferred RNN framework is capable of providing satisfactory calendar capacity health prognostics under different storage cases. A model structure with the impact factor terms of SOC and temperature outperforms other counterparts especially for the unwitnessed conditions. The proposed framework could assist engineers to significantly reduce battery ageing experiment burden and is also promising to capture future capacity information for battery health and life-cycle cost analysis of energy-transportation applications.

Guiding principle of reservoir computing based on “small-world” network

Reservoir computing is a computational framework of recurrent neural networks and is gaining attentions because of its drastically simplified training process. For a given task to solve, however, the methodology has not yet been established how to construct an optimal reservoir. While, “small-world” network has been known to represent networks in real-world such as biological systems and social community. This network is categorized amongst those that are completely regular and totally disordered, and it is characterized by highly-clustered nodes with a short path length. This study aims at providing a guiding principle of systematic synthesis of desired reservoirs by taking advantage of controllable parameters of the small-world network. We will validate the methodology using two different types of benchmark tests—classification task and prediction task.

A novel deep neural network for detection of Atrial Fibrillation using ECG signals

In healthcare practice, one of the most predominantly occurring dysrhythmia is atrial fibrillation (AF). The anomalous heart rhythm and the deficiency of an evident P-wave signal are the consequences of AF, several cerebral apoplexies, thrombus, blood coagulation, cognitive impairments, and strokes. It is arduous to ascertain the symptoms of AF and clinically silent that might cause death. There are certain liabilities for diagnosis of AF in manual Electrocardiogram (ECG) since it demands high expertise; it is a time demanding and tedious process which is also accompanied by variations between intra- and inter-observer. Hence, to combat with this issue, a novel AF detection models has been proposed a conglomerate parallel structure of Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) framework which can deepen the understanding of features and classification. To test and legitimize the system, we utilize the data of the MIT-BIH Atrial Fibrillation Database. Parallel models for control subjects have been designed specifically to validate performance in terms of classification for more voracious categorization of 3 classes namely: Non-Atrial Fibrillation (N-AF), Atrial Fibrillation (AF) and Normal Sinus Rhythm (NSR). The model obtained an Accuracy, Sensitivity and Specificity of 99.6%, 98.64% and 99.01% respectively.

Continuum-Body-Pose Estimation From Partial Sensor Information Using Recurrent Neural Networks

Soft continuum arms have significant potential for use in various applications due to their extremely high degrees of freedom. For example, these soft arms can be used for grasping and manipulating fragile materials in the deep sea or carrying a human to rescue in unstructured environments. However, in these situations, the environment is often dark and visual cues are not always usable. Therefore, these arms must estimate their pose from proprioceptive sensors to control their behavior and execute their tasks in dark places. Estimating the pose in a dynamic situation is still challenging because of the arms' high dimensionality and the complex structural changes in the body shape. Therefore, this study demonstrates a novel method for estimating the pose of proprioceptive bending sensors using recurrent neural networks (RNNs). In particular, an RNN framework known as deep reservoir computing was used for this purpose. Results from experiments using an octopus-inspired soft robotic arm clearly indicate that the proposed method significantly outperforms existing methods using long short-term memory models or linear models. We expect that our proposed method will enable behavioral control of these arms in dark places such as the deep sea, space, and inside the human body in future applications.

Correlated RNN Framework to Quickly Generate Molecules with Desired Properties for Energetic Materials in the Low Data Regime.

Motivated by the challenging of deep learning on the low data regime and the urgent demand for intelligent design on highly energetic materials, we explore a correlated deep learning framework, which consists of three recurrent neural networks (RNNs) correlated by the transfer learning strategy, to efficiently generate new energetic molecules with a high detonation velocity in the case of very limited data available. To avoid the dependence on the external big data set, data augmentation by fragment shuffling of 303 energetic compounds is utilized to produce 500,000 molecules to pretrain RNN, through which the model can learn sufficient structure knowledge. Then the pretrained RNN is fine-tuned by focusing on the 303 energetic compounds to generate 7153 molecules similar to the energetic compounds. In order to more reliably screen the molecules with a high detonation velocity, the SMILE enumeration augmentation coupled with the pretrained knowledge is utilized to build an RNN-based prediction model, through which R2 is boosted from 0.4446 to 0.9572. The comparable performance with the transfer learning strategy based on an existing big database (ChEMBL) to produce the energetic molecules and drug-like ones further supports the effectiveness and generality of our strategy in the low data regime. High-precision quantum mechanics calculations further confirm that 35 new molecules present a higher detonation velocity and lower synthetic accessibility than the classic explosive RDX, along with good thermal stability. In particular, three new molecules are comparable to caged CL-20 in the detonation velocity. All the source codes and the data set are freely available at https://github.com/wangchenghuidream/RNNMGM.