Independently Recurrent Neural Network Research Articles

Application of CNN-IndRNN model combined with residual structure in aircraft engine EGT prediction

The convolutional neural network (CNN), independently recurrent neural network (IndRNN), and residual structure are used to establish a deep learning model for predicting the exhaust gas temperature (EGT) of aircraft engines. Compared to traditional RNN, the IndRNN has an independent recurrent weight matrix in each unit unfolded in the time dimension, which can more efficiently capture spatiotemporal information. Meanwhile, CNN can capture salient features from the multi-feature and multi-time-step data of the original input. Subsequently, the salient features extracted are combined with the initial input using the residual structure before being fed into the IndRNN module. A case study is conducted on the flight data of two civil aircraft engines to verify the effectiveness of IndRNN compared to traditional RNN. The proposed CNN-IndRNN model can accurately predict the EGT of aircraft engines, with a mean absolute relative error below 1.0%.

Multi-Task Classification of Physical Activity and Acute Psychological Stress for Advanced Diabetes Treatment

Wearable sensor data can be integrated and interpreted to improve the treatment of chronic conditions, such as diabetes, by enabling adjustments in treatment decisions based on physical activity and psychological stress assessments. The challenges in using biological analytes to frequently detect physical activity (PA) and acute psychological stress (APS) in daily life necessitate the use of data from noninvasive sensors in wearable devices, such as wristbands. We developed a recurrent multi-task deep neural network (NN) with long-short-term-memory architecture to integrate data from multiple sensors (blood volume pulse, skin temperature, galvanic skin response, three-axis accelerometers) and simultaneously detect and classify the type of PA, namely, sedentary state, treadmill run, stationary bike, and APS, such as non-stress, emotional anxiety stress, mental stress, and estimate the energy expenditure (EE). The objective was to assess the feasibility of using the multi-task recurrent NN (RNN) rather than independent RNNs for detection and classification of AP and APS. The multi-task RNN achieves comparable performance to independent RNNs, with the multi-task RNN having F1 scores of 98.00% for PA and 98.97% for APS, and a root mean square error (RMSE) of 0.728 calhr.kg for EE estimation for testing data. The independent RNNs have F1 scores of 99.64% for PA and 98.83% for APS, and an RMSE of 0.666 calhr.kg for EE estimation. The results indicate that a multi-task RNN can effectively interpret the signals from wearable sensors. Additionally, we developed individual and multi-task extreme gradient boosting (XGBoost) for separate and simultaneous classification of PA types and APS types. Multi-task XGBoost achieved F1 scores of 99.89% and 98.31% for the classification of PA types and APS types, respectively, while the independent XGBoost achieved F1 scores of 99.68% and 96.77%, respectively. The results indicate that both multi-task RNN and XGBoost can be used for the detection and classification of PA and APS without loss of performance with respect to individual separate classification systems. People with diabetes can achieve better outcomes and quality of life by including physical activity and psychological stress assessments in treatment decision-making.

A Two-Stream Neural Network for Pose-Based Hand Gesture Recognition

Pose-based hand gesture recognition has been widely studied in the recent years. Compared with full body action recognition, hand gesture involves joints that are more spatially closely distributed with stronger collaboration. This nature requires a different approach from action recognition to capturing the complex spatial features. Many gesture categories, such as “Grab” and “Pinch,” have very similar motion or temporal patterns posing a challenge on temporal processing. To address these challenges, this article proposes a two-stream neural network with one stream being a self-attention-based graph convolutional network (SAGCN) extracting the short-term temporal information and hierarchical spatial information, and the other being a residual-connection-enhanced bidirectional independently recurrent neural network (IndRNN) for extracting long-term temporal information. The SAGCN has a dynamic self-attention mechanism to adaptively exploit the relationships of all hand joints in addition to the fixed topology and local feature extraction in the GCNs. The proposed method effectively takes advantage of the GCN and IndRNN to capture the temporal-spatial information. The widely used Dynamic Hand Gesture dataset (two evaluation protocols) and First-Person Hand Action dataset are used to validate its effectiveness and our method achieves state-of-the-art performance with 96.31%, 94.05%, and 90.26%, respectively, in terms of recognition accuracy.

Complementary ensemble empirical mode decomposition and independent recurrent neural network model for predicting air quality index

Establishing a scientific and effective air quality prediction model is of great scientific value and practical significance for protecting people’s health and promoting social harmony and stability. However, existing prediction models have certain shortcomings in various aspects. To address these shortcomings, this paper combines different methods to achieve better prediction accuracy. Outlier analysis is done for air quality index (AQI) using an isolation forest algorithm. An air quality prediction system that consists of data preprocessing, optimization, prediction, and modification is established. The complementary ensemble empirical mode decomposition (CEEMD), modified particle swarm optimization (MPSO) algorithm, independent recurrent neural network (IndRNN), and nonlinear correction strategy are employed for the prediction. We select five cities with different AQI as the experiment sites, and three experiments are designed to test the accuracy of the prediction model. The results reveal that (1) by using CEEMD data decomposition technology to deal with the non-stationarity and nonlinearity of the original data, the prediction accuracy of the original cyclic neural network model can be improved by about 15%. (2) The prediction system with the CEEMD-MPSO-IndRNN model as the core prediction module and with a nonlinear error correction strategy has good scalability and robustness for air quality prediction. (3) The performance of the air quality prediction model constructed with systematic thought is better than other comparable models, and it can effectively predict the time series data of different cities and frequencies.

Detection of Depression Severity Using Bengali Social Media Posts on Mental Health: Study Using Natural Language Processing Techniques.

BackgroundThere are a myriad of language cues that indicate depression in written texts, and natural language processing (NLP) researchers have proven the ability of machine learning and deep learning approaches to detect these cues. However, to date, these approaches bridging NLP and the domain of mental health for Bengali literature are not comprehensive. The Bengali-speaking population can express emotions in their native language in greater detail.ObjectiveOur goal is to detect the severity of depression using Bengali texts by generating a novel Bengali corpus of depressive posts. We collaborated with mental health experts to generate a clinically sound labeling scheme and an annotated corpus to train machine learning and deep learning models.MethodsWe conducted a study using Bengali text-based data from blogs and open source platforms. We constructed a procedure for annotated corpus generation and extraction of textual information from Bengali literature for predictive analysis. We developed our own structured data set and designed a clinically sound labeling scheme with the help of mental health professionals, adhering to the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) during the process. We used 5 machine learning models for detecting the severity of depression: kernel support vector machine (SVM), random forest, logistic regression K-nearest neighbor (KNN), and complement naive Bayes (NB). For the deep learning approach, we used long short-term memory (LSTM) units and gated recurrent units (GRUs) coupled with convolutional blocks or self-attention layers. Finally, we aimed for enhanced outcomes by using state-of-the-art pretrained language models.ResultsThe independent recurrent neural network (RNN) models yielded the highest accuracies and weighted F1 scores. GRUs, in particular, produced 81% accuracy. The hybrid architectures could not surpass the RNNs in terms of performance. Kernel SVM with term frequency–inverse document frequency (TF-IDF) embeddings generated 78% accuracy on test data. We used validation and training loss curves to observe and report the performance of our architectures. Overall, the number of available data remained the limitation of our experiment.ConclusionsThe findings from our experimental setup indicate that machine learning and deep learning models are fairly capable of assessing the severity of mental health issues from texts. For the future, we suggest more research endeavors to increase the volume of Bengali text data, in particular, so that modern architectures reach improved generalization capability.

Construction and Application of Music Style Intelligent Learning System Based on Situational Awareness

Contextual representation recommendation directly uses contextual prefiltering technology when processing user contextual data, which is not the integration of context and model in the true sense. To this end, this paper proposes a context-aware recommendation model based on probability matrix factorization. We design a music genre style recognition and generation network. In this network, all the sub-networks of music genres share the explanation layer, which can greatly reduce the learning of model parameters and improve the learning efficiency. Each music genre sub-network analyzes music of different genres, realizing the effect of multitasking simultaneous processing. In this paper, a music style recognition method using a combination of independent recurrent neural network and scattering transform is proposed. The relevant characteristics of traditional audio processing methods are analyzed, and their suitable application scenarios and inapplicability in this task scenario are expounded. Starting from the principle of scattering transform, the superiority and rationality of using scattering transform in this task are explained. This paper proposes a music style recognition method combining two strategies of scattering transform and independent recurrent neural network. In the case that the incremental data set is all labeled, this paper introduces the solution of the convex hull vector, which reduces the training time of the initial sample. According to the error push strategy, an incremental learning algorithm based on convex hull vector and error push strategy is proposed, which can effectively filter historical useful information and at the same time eliminate useless information in new samples. Experiments show that this method improves the accuracy of music style recognition to a certain extent. Music style recognition based on independent recurrent neural network can achieve better performance.

Recognition of musical beat and style and applications in interactive humanoid robot.

The musical beat and style recognition have high application value in music information retrieval. However, the traditional methods mostly use a convolutional neural network (CNN) as the backbone and have poor performance. Accordingly, the present work chooses a recurrent neural network (RNN) in deep learning (DL) to identify musical beats and styles. The proposed model is applied to an interactive humanoid robot. First, DL-based musical beat and style recognition technologies are studied. On this basis, a note beat recognition method combining attention mechanism (AM) and independent RNN (IndRNN) [AM-IndRNN] is proposed. The AM-IndRNN can effectively avoid gradient vanishing and gradient exploding. Second, the audio music files are divided into multiple styles using the music signal's temporal features. A human dancing robot using a multimodal drive is constructed. Finally, the proposed method is tested. The results show that the proposed AM-IndRNN outperforms multiple parallel long short-term memory (LSTM) models and IndRNN in recognition accuracy (88.9%) and loss rate (0.0748). Therefore, the AM-optimized LSTM model has gained a higher recognition accuracy. The research results provide specific ideas for applying DL technology in musical beat and style recognition.

Forecasting oil consumption with attention-based IndRNN optimized by adaptive differential evolution.

Accurate prediction of oil consumption plays a dominant role in oil supply chain management. However, because of the effects of the coronavirus disease 2019 (COVID-19) pandemic, oil consumption has exhibited an uncertain and volatile trend, which leads to a huge challenge to accurate predictions. The rapid development of the Internet provides countless online information (e.g., online news) that can benefit predict oil consumption. This study adopts a novel news-based oil consumption prediction methodology–convolutional neural network (CNN) to fetch online news information automatically, thereby illustrating the contribution of text features for oil consumption prediction. This study also proposes a new approach called attention-based JADE-IndRNN that combines adaptive differential evolution (adaptive differential evolution with optional external archive, JADE) with an attention-based independent recurrent neural network (IndRNN) to forecast monthly oil consumption. Experimental results further indicate that the proposed news-based oil consumption prediction methodology improves on the traditional techniques without online oil news significantly, as the news might contain some explanations of the relevant confinement or reopen policies during the COVID-19 period.

Variable Rate Independently Recurrent Neural Network (IndRNN) for Action Recognition

Recurrent neural networks (RNNs) have been widely used to solve sequence problems due to their capability of modeling temporal dependency. Despite the rich varieties of RNN models proposed in the literature, the problem of different sampling rates or performing speeds in sequence tasks has not been explicitly considered in the network and the corresponding training and testing processes. This paper addresses the problem of different sampling rates or performing speeds in the skeleton-based action recognition with RNNs. Specifically, the recently proposed independently recurrent neural network (IndRNN) is used as the RNN network due to its well-behaved and easily regulated gradient backpropagation through time. Samples are extracted with variable sampling rates and thus of different lengths, then processed by IndRNN with different time steps. In order to accommodate the differences in terms of gradients introduced by the backpropagation through time under variable time steps, a learning rate adjustment method is further proposed in the paper. Different learning rate adjustment factors are obtained for different layers by analyzing the gradient behavior under IndRNN. Experiments on skeleton-based action recognition are conducted to verify its effectiveness, and the results show that the proposed variable rate IndRNN network can significantly improve the performance over the RNN models under the conventional training strategies.

An Abnormal Traffic Detection Model Combined BiIndRNN With Global Attention

As time series data with internal correlation, networks traffic data can be used for abnormal detection using Recurrent Neural Network (RNN) and its variants, but existing models are difficult to calculate in parallel, and gradient explosion or vanishing easily occurs. To address this problem, we propose a Bidirectional Independent Recurrent Neural Network (BiIndRNN) with parallel computation and adjustable gradient, which can extract the bidirectional structural features of networks traffic by forward and backward input and capture the spatial influence in the data flow. To establish the dependencies on the forward and backward moments of networks traffic, a model combining Global Attention (GA) with BiIndRNN is proposed to pay more attention to the moments containing essential information. Taking the UNSW-NB15 dataset as the object, the GA expression of the packets feature vector of the networks is derived, feature fusion, as well as loss calculation, is performed for multiple fully connected layers. The experimental results show that, compared with traditional deep and shallow machine learning and other state-of-the-art technologies, our GA-BiIndRNN model converges faster, the accuracy, precision, and F1 scores are all above 99%, and the false positive rate (FPR) is close to 0.36%, which can effectively identify normal and malicious network activities. These results provide a theoretical basis for the rapid implementation of protective measures.

Complementary Ensemble Empirical Mode Decomposition and Independent Recurrent Neural Network Model for Predicting Air Quality Index

Complementary Ensemble Empirical Mode Decomposition and Independent Recurrent Neural Network Model for Predicting Air Quality Index

A Novel Approach for Malicious URL Detection Based on the Joint Model

The number of malicious websites is increasing yearly, and many companies and individuals worldwide have suffered losses. Therefore, the detection of malicious websites is a task that needs continuous development. In this study, a joint neural network algorithm model combining the attention mechanism, bidirectional independent recurrent neural network (Bi-IndRNN), and capsule network (CapsNet) is proposed. The word vector tool word2vec trains the character- and word-level uniform resource locator (URL) static embedding vector features. At the same time, the algorithm will also extract texture fingerprint features that can compare the content differences of different malicious web URL binary files. Then, the extracted features are fused and input into the joint neural network algorithm model. First, the multihead attention mechanism is used to extract contextual semantic features by adjusting weights and Bi-IndRNN. Second, CapsNet with dynamic routing is used to extract deep semantic information. Finally, the sigmoid classifier is used for classification. This study uses different methods from different angles to extract more comprehensive features. From the experimental results, the method proposed in this study improves the classification accuracy of malicious web page detection compared with other researchers.

Case Study of Deep Learning Model of Temperature-Induced Deflection of a Cable-Stayed Bridge Driven by Data Knowledge

A cable-stayed bridge is a typical symmetrical structure, and symmetry affects the deformation characteristics of such bridges. The main girder of a cable-stayed bridge will produce obvious deflection under the inducement of temperature. The regression model of temperature-induced deflection is hoped to provide a comparison value for bridge evaluation. Based on the temperature and deflection data obtained by the health monitoring system of a bridge, establishing the correlation model between temperature and temperature-induced deflection is meaningful. It is difficult to complete a high-quality model only by the girder temperature. The temperature features based on prior knowledge from the mechanical mechanism are used as the input information in this paper. At the same time, to strengthen the nonlinear ability of the model, this paper selects an independent recurrent neural network (IndRNN) for modeling. The deep learning neural network is compared with machine learning neural networks to prove the advancement of deep learning. When only the average temperature of the main girder is input, the calculation accuracy is not high regardless of whether the deep learning network or the machine learning network is used. When the temperature information extracted by the prior knowledge is input, the average error of IndRNN model is only 2.53%, less than those of BPNN model and traditional RNN. Combining knowledge with deep learning is undoubtedly the best modeling scheme. The deep learning model can provide a comparison value of bridge deformation for bridge management.

Prediction of COVID-19 epidemic situation via fine-tuned IndRNN.

The COVID-19 pandemic is the most serious catastrophe since the Second World War. To predict the epidemic more accurately under the influence of policies, a framework based on Independently Recurrent Neural Network (IndRNN) with fine-tuning are proposed for predict the epidemic development trend of confirmed cases and deaths in the United Stated, India, Brazil, France, Russia, China, and the world to late May, 2021. The proposed framework consists of four main steps: data pre-processing, model pre-training and weight saving, the weight fine-tuning, trend predicting and validating. It is concluded that the proposed framework based on IndRNN and fine-tuning with high speed and low complexity, has great fitting and prediction performance. The applied fine-tuning strategy can effectively reduce the error by up to 20.94% and time cost. For most of the countries, the MAPEs of fine-tuned IndRNN model were less than 1.2%, the minimum MAPE and RMSE were 0.05%, and 1.17, respectively, by using Chinese deaths, during the testing phase. According to the prediction and validation results, the MAPEs of the proposed framework were less than 6.2% in most cases, and it generated lowest MAPE and RMSE values of 0.05% and 2.14, respectively, for deaths in China. Moreover, Policies that play an important role in the development of COVID-19 have been summarized. Timely and appropriate measures can greatly reduce the spread of COVID-19; untimely and inappropriate government policies, lax regulations, and insufficient public cooperation are the reasons for the aggravation of the epidemic situations. The code is available at https://github.com/zhhongsh/COVID19-Precdiction. And the prediction by IndRNN model with fine-tuning are now available online (http://47.117.160.245:8088/IndRNNPredict).

Analysis of machine learning based LEACH robust routing in the Edge Computing systems

Wireless sensor networks (WSN) are used to detect real-time changes in the deployed environment. This dynamic behaviour is either triggered by the deployed environment or by the user from outside. Because of their ability to monitor complex scenarios that change rapidly over time, wireless sensor networks are critical components of most advanced computing systems. These complex activities are influenced by different methods or even by the designers of their networks. Machine learning encourages many real solutions that optimise resource use and increase the network's lifespan in sensor networks. LEACH routing protocol has many limitations due to sudden energy utilisation & cluster head nodes due to direct communication with the base station node. This fast node energy leak creates several black hole structures in the networks, resulting in data redundancy, data packets transmission, node upgrade costs, and end-to-end delay for WSN. The proposed model with LEACH protocol functionality has improved network performance, network (WSN) efficiency, and solving data redundancy issues. By using an independent Recurrent Neural Network (IRNN)-based data fusion algorithm, namely, DFAIRNN. The simulation and comparative results indicate that the mean method & minimum distance method used in the LEACH-DFAIRNN protocol can effectively resolve data redundancy issues caused by the adjacent sensor nodes by flooding data simultaneously to a single node.

State of Charge Estimation of Lithium-ion Battery Based on Improved Recurrent Neural Network

Aiming at the problem of gradient disappearance and gradient explosion in the recurrent neural network (RNN) algorithm in the battery estimation model, this paper proposes a state of charge (SOC) estimation model developed using an independent recurrent neural network (IndRNN). Firstly, the Thevenin equivalent model of the battery is established, and the parameters of the equivalent model are identified through experiments. Then, the Relu activation function is introduced into the RNN to separate the neurons in each layer. Finally, the model was trained on various experimental data sets collected from the lithium iron phosphate battery experimental platform under different discharge conditions. Without any prior knowledge about the battery interior, the proposed battery model successfully characterizes the non-linear behavior of the battery. The results show that under different discharge conditions, IndRNN is better than RNN in terms of maximum error, the mean error, and the root mean square error (RMSE), which greatly improves the accuracy of SOC estimation.

Deep Learning Based Modulation Recognition With Multi-Cue Fusion

In this letter, a novel multi-cue fusion (MCF) network for automatic modulation recognition is proposed. To exploit the potentially non-trivial and informative contents that previous works ignored, multi-cue learning is injected into neural network design. The MCF network consists of a signal cue multi-stream (SCMS) module and a visual cue discrimination (VCD) module. The SCMS module based on Convolutional Neural Network (CNN) and Independently Recurrent Neural Network (IndRNN) is built for modeling spatial-temporal correlations from two signal cues (i.e., In-phase/Quadrature (I/Q) and amplitude-phase (A/Φ)), which aims to explore various differences and leverage the complements from multiple data-form. In VCD module, raw I/Q data is converted to constellation diagrams as the visual cue to exploit the structural information utilizing the feature extracting capability of CNN. Experimental results on RadioML2016.10a and RadioML2018 datasets achieve 97.8% and 96.1% respectively, which outperform the state-of-the-art works.

Malicious URL Detection Algorithm Based on Multi Neural Network Series

Convolutional neural network based on attention mechanism and a bidirectional independent recurrent neural network tandem joint algorithm (CATIR) are proposed. In natural language processing related technologies, word vector features are extracted based on URLs, and the extracted URL information features and host information features are merged. The proposed CATIR algorithm uses CNN (Convolutional Neural Network) to obtain the deep local features in the data, uses the Attention mechanism to adjust the weights, and uses IndRNN (Independent Recurrent Neural Network) to obtain the global features in the data. The experimental results shows that the CATIR algorithm has significantly improved the accuracy of malicious URL detection based on traditional algorithms to 96.9%.

Design of English Automatic Translation System Based on Machine Intelligent Translation and Secure Internet of Things

With the rapid development of Internet technology and the development of economic globalization, international exchanges in various fields have become increasingly active, and the need for communication between languages has become increasingly clear. As an effective tool, automatic translation can perform equivalent translation between different languages while preserving the original semantics. This is very important in practice. This paper focuses on the Chinese-English machine translation model based on deep neural networks. In this paper, we use the end-to-end encoder and decoder framework to create a neural machine translation model, the machine automatically learns its function, and the data is converted into word vectors in a distributed method and can be directly through the neural network perform the mapping between the source language and the target language. Research experiments show that, by adding part of the voice information to verify the effectiveness of the model performance improvement, the performance of the translation model can be improved. With the superimposition of the number of network layers from two to four, the improvement ratios of each model are 5.90%, 6.1%, 6.0%, and 7.0%, respectively. Among them, the model with an independent recurrent neural network as the network structure has the largest improvement rate and a higher improvement rate, so the system has high availability.

Research on epileptic EEG recognition based on improved residual networks of 1-D CNN and indRNN

BackgroundEpilepsy is one of the diseases of the nervous system, which has a large population in the world. Traditional diagnosis methods mostly depended on the professional neurologists’ reading of the electroencephalogram (EEG), which was time-consuming, inefficient, and subjective. In recent years, automatic epilepsy diagnosis of EEG by deep learning had attracted more and more attention. But the potential of deep neural networks in seizure detection had not been fully developed.MethodsIn this article, we used a one-dimensional convolutional neural network (1-D CNN) to replace the residual network architecture’s traditional convolutional neural network (CNN). Moreover, we combined the Independent recurrent neural network (indRNN) and CNN to form a new residual network architecture-independent convolutional recurrent neural network (RCNN). Our model can achieve an automatic diagnosis of epilepsy EEG. Firstly, the important features of EEG were learned by using the residual network architecture of 1-D CNN. Then the relationship between the sequences were learned by using the recurrent neural network. Finally, the model outputted the classification results.ResultsOn the small sample data sets of Bonn University, our method was superior to the baseline methods and achieved 100% classification accuracy, 100% classification specificity. For the noisy real-world data, our method also exhibited powerful performance.ConclusionThe model we proposed can quickly and accurately identify the different periods of EEG in an ideal condition and the real-world condition. The model can provide automatic detection capabilities for clinical epilepsy EEG detection. We hoped to provide a positive significance for the prediction of epileptic seizures EEG.