Deep Bidirectional Recurrent Neural Networks Research Articles

A Comprehensive Survey of Predictive Maintenance Techniques for Aircraft Engines Utilizing the C-MAPSS Dataset

The application of deep learning and sophisticated machine learning techniques is driving the rapid advancement of aircraft engine prognostics and predictive maintenance. Remain ing Useful Life (RUL) of aviation engines has been the subject of numerous studies aimed at improving prediction accuracy and efficacy to improve aviation safety and maintenance plans. Innovative approaches and technologies are demonstrated by these projects, which use a variety of methodologies and datasets, including C-MAPSS and N-CMAPSS. Combining feature engi neering, ensemble learning, and deep learning models such as Restricted Boltzmann Machines (RBMs), Long Short-Term Mem ory (LSTM) networks, Convolutional Neural Networks (CNNs), and Deep Bidirectional Recurrent Neural Networks (DBRNNs) is one prominent method. Features are chosen and models are optimized using a variety of methods, including Genetic Algorithms, Recursive Feature Elimination, Lasso, and Feature Importances. In prognostic modeling, the research emphasize the need of interpretability, model adaptability, and measuring uncertainty. Additionally, in order to pinpoint important features and improve model transparency, the study investigates the use of explainable AI techniques such aggregated feature importances with cross-validation (AFICv) and Shapley additive explanation (SHAP). In order to capture prediction uncertainties, the inte gration of Gaussian Processes (GPs) and Bayesian Deep Neural Networks (DNNs) is also investigated. This provides insights into uncertainty-aware prognosis and predictive analytics for industrial assets. The development and publication of datasets such as the N-CMAPSS dataset also makes it possible to conduct more comprehensive and realistic assessments of prognostic models under real-world flight conditions, providing useful tools for benchmarking and improving machine learning algorithms in predictive maintenance. All things considered, these research projects highlight current developments and the possibility of combining cutting edge technology to improve system reliability, improve predictive maintenance techniques, and guarantee safer airline operations. Key Words: Aircraft engine, Prognostics, Predictive main tenance, Machine learning, Deep learning, Remaining Useful Life (RUL), Aviation safety, Feature engineering, Ensemble learning, Uncertainty quantification, Explainable AI, Bayesian Deep Neural Networks (DNNs), Gaussian Processes (GPs), N CMAPSS dataset, Real flight conditions, Industrial assets, System reliability, Benchmarking

Task sub-type states decoding via group deep bidirectional recurrent neural network

Decoding brain states under different cognitive tasks from functional magnetic resonance imaging (fMRI) data has attracted great attention in the neuroimaging filed. However, the well-known temporal dependency in fMRI sequences has not been fully exploited in existing studies, due to the limited temporal-modeling capacity of the backbone machine learning algorithms and rigid training sample organization strategies upon which the brain decoding methods are built. To address these limitations, we propose a novel method for fine-grain brain state decoding, namely, group deep bidirectional recurrent neural network (Group-DBRNN) model. We first propose a training sample organization strategy that consists of a group-task sample generation module and a multiple-scale random fragment strategy (MRFS) module to collect training samples that contain rich task-relevant brain activity contrast (i.e., the comparison of neural activity patterns between different tasks) and maintain the temporal dependency. We then develop a novel decoding model by replacing the unidirectional RNNs that are widely used in existing brain state decoding studies with bidirectional stacked RNNs to better capture the temporal dependency, and by introducing a multi-task interaction layer (MTIL) module to effectively model the task-relevant brain activity contrast. Our experimental results on the Human Connectome Project task fMRI dataset (7 tasks consisting of 23 task sub-type states) show that the proposed model achieves an average decoding accuracy of 94.7% over the 23 fine-grain sub-type states. Meanwhile, our extensive interpretations of the intermediate features learned in the proposed model via visualizations and quantitative assessments of their discriminability and inter-subject alignment evidence that the proposed model can effectively capture the temporal dependency and task-relevant contrast.

Improved deep bidirectional recurrent neural network for learning the cross-sensitivity rules of gas sensor array

Cross-sensitivity among chemical gas sensors leads to inaccurate identification of mixed gas. Pattern recognition algorithms are usually applied to improve the recognition accuracy. However, the abilities of current methods to process sequence property of response data are not strong enough. Especially, they cannot deal with bidirectional cross-sensitive issue, leading to recognition errors. Bidirectional Recurrent Neural Network (BRNN) could well learn bidirectional association features between word sequences in natural language processing field, which is similar to the bidirectional interaction of cross-sensitivity. In this study, an improved deep BRNN model was constructed to solve the cross-sensitivity problem of chemical gas sensor array. A chemical gas sensor array with four units was fabricated and response data was thoroughly obtained. Data preprocessing methods, model structure hyperparameters and optimizers were studied. Finally, an improved deep BRNN model was developed with 3 layers and 100 hidden_size, training with Adamax optimizer. A recognition accuracy of 98.93% was achieved, attributing to the model’s excellent learning ability to the bidirectional cross-sensitivity rules among gas sensors. This improved BRNN model provided a novel idea to eliminate cross-sensitivity, exhibiting good potential for recognizing mixed gas analyte accurately.

BIDRN: A Method of Bidirectional Recurrent Neural Network for Sentiment Analysis

Text mining research has grown in importance in recent years due to the tremendous increase in the volume of unstructured textual data. This has resulted in immense potential as well as obstacles in the sector, which may be efficiently addressed with adequate analytical and study methods. Deep Bidirectional Recurrent Neural Networks are used in this study to analyze sentiment. The method is categorized as sentiment polarity analysis because it may generate a dataset with sentiment labels. This dataset can be used to train and evaluate sentiment analysis models capable of extracting impartial opinions. This paper describes the Sentiment Analysis-Deep Bidirectional Recurrent Neural Networks (SA-BDRNN) Scheme, which seeks to overcome the challenges and maximize the potential of text mining in the context of Big Data. The current study proposes a SA-DBRNN Scheme that attempts to give a systematic framework for sentiment analysis in the context of student input on institution choice. The purpose of this study is to compare the effectiveness of the proposed SA- DBRNN Scheme to existing frameworks to establish a robust deep neural network that might serve as an adequate classification model in the field of sentiment analysis.

The application of machine learning algorithms in speech recognition error detection

The field of speech recognition technology has experienced a significant progress in recent years, with error detection being a crucial research domain. The present study offers a comprehensive overview of this area. Prior to the emergence of neural networks, Hidden Markov models (HMM) have been widely employed as a primary framework for speech recognition. However, this model only achieves local optimality, leading to its gradual replacement by neural network models, which have attracted considerable attention from researchers aiming to enhance their recognition performance. Various models, such as variant RNNs, bidirectional recurrent neural networks, and Fastcorrect2 have been developed. This paper introduces HMM, followed by a presentation of several neural network models, which entail a detailed description of their respective framework, principle, and idea. The variant RNN model is designed to enhance the recursive connection between input and output layers, while the deep bidirectional recurrent neural network model simulates the nonlinear relationship between input feature vectors and output labels using two models, namely bidirectional and deep. Additionally, the FastCorrect2 model enhances the voting effect of candidate words and the alignment algorithm. Finally, the study highlights the application of speech recognition error detection in everyday life, emphasizing the importance of speech recognition.

Deep Bidirectional Recurrent Neural Networks Ensemble for Remaining Useful Life Prediction of Aircraft Engine.

Remaining useful life (RUL) prediction of aircraft engine (AE) is of great importance to improve its reliability and availability, and reduce its maintenance costs. This article proposes a novel deep bidirectional recurrent neural networks (DBRNNs) ensemble method for the RUL prediction of the AEs. In this method, several kinds of DBRNNs with different neuron structures are built to extract hidden features from sensory data. A new customized loss function is designed to evaluate the performance of the DBRNNs, and a series of the RUL values is obtained. Then, these RUL values are reencapsulated into a predicted RUL domain. By updating the weights of elements in the domain, multiple regression decision tree (RDT) models are trained iteratively. These models integrate the predicted results of different DBRNNs to realize the final RUL prognostics with high accuracy. The proposed method is validated by using C-MAPSS datasets from NASA. The experimental results show that the proposed method has achieved more superior performance compared with other existing methods.

Improving the Robustness of Loanword Identification in Social Media Texts

As a potential bilingual resource, loanwords play a very important role in many natural language processing tasks. If loanwords in a low-resource language can be identified effectively, the generated donor-receipt word pairs will benefit many cross-lingual natural language processing tasks. However, most studies on loanword identification mainly focus on formal texts such as news and government documents. Loanword identification in social media texts is still an under-studied field. Since it faces many challenges and can be widely used in several downstream tasks, more efforts should be put on loanword identification in social media texts. In this study, we present a multi-task learning architecture with deep bi-directional recurrent neural networks for loanword identification in social media texts, where different task supervision can happen at different layers. The multi-task neural network architecture learns higher-order feature representations from word and character sequences along with basic spell error checking, part-of-speech tagging, and named entity recognition information. Experimental results on Uyghur loanword identification in social media texts in five donor languages (Chinese, Arabic, Russian, Turkish, and Farsi) show that our method achieves the best performance compared with several strong baseline systems. We also combine the loanword detection results into the training data of neural machine translation for low-resource language pairs. Experiments show that models trained on the extended datasets achieve significant improvements compared with the baseline models in all language pairs.

Resource scheduling in edge computing IoT networks using hybrid deep learning algorithm

The proliferation of the Internet of Things (IoT) and wireless sensor networks enhances data communication. The demand for data communication rapidly increases, which calls the emerging edge computing paradigm. Edge computing plays a major role in IoT networks and provides computing resources close to the users. Moving the services from the cloud to users increases the communication, storage, and network features of the users. However, massive IoT networks require a large spectrum of resources for their computations. In order to attain this, resource scheduling algorithms are employed in edge computing. Statistical and machine learning-based resource scheduling algorithms have evolved in the past decade, but the performance can be improved if resource requirements are analyzed further. A deep learning-based resource scheduling in edge computing IoT networks is presented in this research work using deep bidirectional recurrent neural network (BRNN) and convolutional neural network algorithms. Before scheduling, the IoT users are categorized into clusters using a spectral clustering algorithm. The proposed model simulation analysis verifies the performance in terms of delay, response time, execution time, and resource utilization. Existing resource scheduling algorithms like a genetic algorithm (GA), Improved Particle Swarm Optimization (IPSO), and LSTM-based models are compared with the proposed model to validate the superior performances.

Exploiting variable length segments with coarticulation effect in online speech recognition based on deep bidirectional recurrent neural network and context-sensitive segment

Exploiting variable length segments with coarticulation effect in online speech recognition based on deep bidirectional recurrent neural network and context-sensitive segment

JEDI: circular RNA prediction based on junction encoders and deep interaction among splice sites

MotivationCircular RNA (circRNA) is a novel class of long non-coding RNAs that have been broadly discovered in the eukaryotic transcriptome. The circular structure arises from a non-canonical splicing process, where the donor site backspliced to an upstream acceptor site. These circRNA sequences are conserved across species. More importantly, rising evidence suggests their vital roles in gene regulation and association with diseases. As the fundamental effort toward elucidating their functions and mechanisms, several computational methods have been proposed to predict the circular structure from the primary sequence. Recently, advanced computational methods leverage deep learning to capture the relevant patterns from RNA sequences and model their interactions to facilitate the prediction. However, these methods fail to fully explore positional information of splice junctions and their deep interaction.ResultsWe present a robust end-to-end framework, Junction Encoder with Deep Interaction (JEDI), for circRNA prediction using only nucleotide sequences. JEDI first leverages the attention mechanism to encode each junction site based on deep bidirectional recurrent neural networks and then presents the novel cross-attention layer to model deep interaction among these sites for backsplicing. Finally, JEDI can not only predict circRNAs but also interpret relationships among splice sites to discover backsplicing hotspots within a gene region. Experiments demonstrate JEDI significantly outperforms state-of-the-art approaches in circRNA prediction on both isoform level and gene level. Moreover, JEDI also shows promising results on zero-shot backsplicing discovery, where none of the existing approaches can achieve.Availability and implementationThe implementation of our framework is available at https://github.com/hallogameboy/JEDI.Supplementary information Supplementary data are available at Bioinformatics online.

Predicting Proteolysis in Complex Proteomes Using Deep Learning.

Both protease- and reactive oxygen species (ROS)-mediated proteolysis are thought to be key effectors of tissue remodeling. We have previously shown that comparison of amino acid composition can predict the differential susceptibilities of proteins to photo-oxidation. However, predicting protein susceptibility to endogenous proteases remains challenging. Here, we aim to develop bioinformatics tools to (i) predict cleavage site locations (and hence putative protein susceptibilities) and (ii) compare the predicted vulnerabilities of skin proteins to protease- and ROS-mediated proteolysis. The first goal of this study was to experimentally evaluate the ability of existing protease cleavage site prediction models (PROSPER and DeepCleave) to identify experimentally determined MMP9 cleavage sites in two purified proteins and in a complex human dermal fibroblast-derived extracellular matrix (ECM) proteome. We subsequently developed deep bidirectional recurrent neural network (BRNN) models to predict cleavage sites for 14 tissue proteases. The predictions of the new models were tested against experimental datasets and combined with amino acid composition analysis (to predict ultraviolet radiation (UVR)/ROS susceptibility) in a new web app: the Manchester proteome susceptibility calculator (MPSC). The BRNN models performed better in predicting cleavage sites in native dermal ECM proteins than existing models (DeepCleave and PROSPER), and application of MPSC to the skin proteome suggests that: compared with the elastic fiber network, fibrillar collagens may be susceptible primarily to protease-mediated proteolysis. We also identify additional putative targets of oxidative damage (dermatopontin, fibulins and defensins) and protease action (laminins and nidogen). MPSC has the potential to identify potential targets of proteolysis in disparate tissues and disease states.

Identifying protein subcellular localisation in scientific literature using bidirectional deep recurrent neural network

The increased diversity and scale of published biological data has to led to a growing appreciation for the applications of machine learning and statistical methodologies to gain new insights. Key to achieving this aim is solving the Relationship Extraction problem which specifies the semantic interaction between two or more biological entities in a published study. Here, we employed two deep neural network natural language processing (NLP) methods, namely: the continuous bag of words (CBOW), and the bi-directional long short-term memory (bi-LSTM). These methods were employed to predict relations between entities that describe protein subcellular localisation in plants. We applied our system to 1700 published Arabidopsis protein subcellular studies from the SUBA manually curated dataset. The system combines pre-processing of full-text articles in a machine-readable format with relevant sentence extraction for downstream NLP analysis. Using the SUBA corpus, the neural network classifier predicted interactions between protein name, subcellular localisation and experimental methodology with an average precision, recall rate, accuracy and F1 scores of 95.1%, 82.8%, 89.3% and 88.4% respectively (n = 30). Comparable scoring metrics were obtained using the CropPAL database as an independent testing dataset that stores protein subcellular localisation in crop species, demonstrating wide applicability of prediction model. We provide a framework for extracting protein functional features from unstructured text in the literature with high accuracy, improving data dissemination and unlocking the potential of big data text analytics for generating new hypotheses.

Derin Öğrenme ile Otomatik Hata Triyajlama: Bir Replikasyon Çalışması

Bug management is the process to identify and fix bugs. In the bug management process, after a bug is identified, it needs to triaged. Bug triaging is the process of prioritizing bugs and assigning an appropriate developer for a given bug. The main task in bug triaging is to predict the most appropriate developer to fix a software bug from a given bug report. This problem can be defined as a classification problem in which textual bug attributes (bug title, description etc.) are inputs and the available developer (class label) is the output. Since manual bug triaging is a time consuming process, there have een several bug triaging algorithms to automate this process. One of the latest successful algorithms to address this problem is the Deep Triage. It employs Deep Bidirectional Recurrent Neural Network with Attention (DBRNN-A) approach for this classification task. In this study, we implement an improved version of the automated bug triaging method, DeepTriage. To improve the performance of the model, three contributions are made to the original implementation: (1) Using GRU instead of LSTM to fasten the training process by using a larger batch size with the same memory usage, (2) Using a corpus combining the data from different datasets to create a more generalized model, (3) Adding extra dense layers before the multiclass classification to improve the results. After running the experiments, we achieved the state of the art results in Mozilla Firefox dataset, an accuracy of 46.6%. In the Chromium dataset, we get a higher accuracy (44.0%) than the original accuracy from the paper (42.7%). The resulting model and its source code is made publicly available for future research in this area.

A deep bidirectional recurrent neural network for identification of SARS-CoV-2 from viral genome sequences.

In this work, Deep Bidirectional Recurrent Neural Networks (BRNNs) models were implemented based on both Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) cells in order to distinguish between genome sequence of SARS-CoV-2 and other Corona Virus strains such as SARS-CoV and MERS-CoV, Common Cold and other Acute Respiratory Infection (ARI) viruses. An investigation of the hyper-parameters including the optimizer type and the number of unit cells, was also performed to attain the best performance of the BRNN models. Results showed that the GRU BRNNs model was able to discriminate between SARS-CoV-2 and other classes of viruses with a higher overall classification accuracy of 96.8% as compared to that of the LSTM BRNNs model having a 95.8% overall classification accuracy. The best hyper-parameters producing the highest performance for both models was obtained when applying the SGD optimizer and an optimum number of unit cells of 80 in both models. This study proved that the proposed GRU BRNN model has a better classification ability for SARS-CoV-2 thus providing an efficient tool to help in containing the disease and achieving better clinical decisions with high precision.

Near-infrared Search for Fundamental-mode RR Lyrae Stars toward the Inner Bulge by Deep Learning

Abstract Aiming to extend the census of RR Lyrae stars to highly reddened low-latitude regions of the central Milky Way, we performed a deep near-IR variability search using data from the VISTA Variables in the Vía Láctea (VVV) survey of the bulge, analyzing the photometric time series of over a hundred million point sources. In order to separate fundamental-mode RR Lyrae (RRab) stars from other periodically variable sources, we trained a deep bidirectional long short-term memory recurrent neural network (RNN) classifier using VVV survey data and catalogs of RRab stars discovered and classified by optical surveys. Our classifier attained a ∼99% precision and recall for light curves with signal-to-noise ratio above 60, and is comparable to the best-performing classifiers trained on accurate optical data. Using our RNN classifier, we identified over 4300 hitherto unknown bona fide RRab stars toward the inner bulge. We provide their photometric catalog and VVV photometric time series.

Meter classification of Arabic poems using deep bidirectional recurrent neural networks

Poetry is an important component of any language. Much of a nation’s history and culture are documented in poems. A poem has a rhythmic flow which is quite different as compared to a prose. Each language has its own set of rhythmical structures for poems, called meters. Identifying the meters of Arabic poems is a lengthy and complicated process. To classify a poem’s meter, the text of the poem should be encoded in a special Arudi form which needs complex rule-based transformations before another set of rules can be used to finally classify the meters. This paper introduces a novel method for classifying poem meters of Arabic poems using RNN-based deep learning. It bypasses the need to transform the poem to the Arudi form as well as the need to explicitly encode the complex rules that are usually followed to determine the meter. The presented method was evaluated on a large dataset collected specifically for this purpose. We are able to classify the poem meters with an accuracy of 94.32% on an independent test set.

A Deep Multi-task Model for Dialogue Act Classification, Intent Detection and Slot Filling

An essential component of any dialogue system is understanding the language which is known as spoken language understanding (SLU). Dialogue act classification (DAC), intent detection (ID) and slot filling (SF) are significant aspects of every dialogue system. In this paper, we propose a deep learning-based multi-task model that can perform DAC, ID and SF tasks together. We use a deep bi-directional recurrent neural network (RNN) with long short-term memory (LSTM) and gated recurrent unit (GRU) as the frameworks in our multi-task model. We use attention on the LSTM/GRU output for DAC and ID. The attention outputs are fed to individual task-specific dense layers for DAC and ID. The output of LSTM/GRU is fed to softmax layer for slot filling as well. Experiments on three datasets, i.e. ATIS, TRAINS and FRAMES, show that our proposed multi-task model performs better than the individual models as well as all the pipeline models. The experimental results prove that our attention-based multi-task model outperforms the state-of-the-art approaches for the SLU tasks. For DAC, in relation to the individual model, we achieve an improvement of more than 2% for all the datasets. Similarly, for ID, we get an improvement of 1% on the ATIS dataset, while for TRAINS and FRAMES dataset, there is a significant improvement of more than 3% compared to individual models. We also get a 0.8% enhancement for ATIS and a 4% enhancement for TRAINS and FRAMES dataset for SF with respect to individual models. Results obtained clearly show that our approach is better than existing methods. The validation of the obtained results is also demonstrated using statistical significance t tests.

PHTNet: Characterization and Deep Mining of Involuntary Pathological Hand Tremor using Recurrent Neural Network Models

The global aging phenomenon has increased the number of individuals with age-related neurological movement disorders including Parkinson’s Disease (PD) and Essential Tremor (ET). Pathological Hand Tremor (PHT), which is considered among the most common motor symptoms of such disorders, can severely affect patients’ independence and quality of life. To develop advanced rehabilitation and assistive technologies, accurate estimation/prediction of nonstationary PHT is critical, however, the required level of accuracy has not yet been achieved. The lack of sizable datasets and generalizable modeling techniques that can fully represent the spectrotemporal characteristics of PHT have been a critical bottleneck in attaining this goal. This paper addresses this unmet need through establishing a deep recurrent model to predict and eliminate the PHT component of hand motion. More specifically, we propose a machine learning-based, assumption-free, and real-time PHT elimination framework, the PHTNet, by incorporating deep bidirectional recurrent neural networks. The PHTNet is developed over a hand motion dataset of 81 ET and PD patients collected systematically in a movement disorders clinic over 3 years. The PHTNet is the first intelligent systems model developed on this scale for PHT elimination that maximizes the resolution of estimation and allows for prediction of future and upcoming sub-movements.

Multi-task and multi-view training for end-to-end relation extraction

Transfer learning has shown promising results for transferring knowledge ofsource tasks to target tasks in natural language processing (NLP). In this paper, we investigate a multi-task and multi-view learning (MTMVL) framework for end-to-end neural relation extraction, using large noisy data as one auxiliary task for improving manually labeled data, and a dependency parsing objective as a second auxiliary task for leveraging syntax. Two views are taken for relation extraction, consisting of a joint tagging view and a novel context relation view. To capture sentence multi-level information, we explore a weighted average approach to represent shared deep bi-directional recurrent neural networks and encourage auxiliary tasks to accommodate relation extraction task via a time-related scheduled sampling strategy. We evaluate MTMVL framework on manual-labeled ACE2005 dataset, and experimental results show that our model outperforms the state-of-the-art methods, which indicates the effectiveness of multi-auxiliary information for knowledge transferring.

End-to-end optimized transmission over dispersive intensity-modulated channels using bidirectional recurrent neural networks.

We propose an autoencoding sequence-based transceiver for communication over dispersive channels with intensity modulation and direct detection (IM/DD), designed as a bidirectional deep recurrent neural network (BRNN). The receiver uses a sliding window technique to allow for efficient data stream estimation. We find that this sliding window BRNN (SBRNN), based on end-to-end deep learning of the communication system, achieves a significant bit-error-rate reduction at all examined distances in comparison to previous block-based autoencoders implemented as feed-forward neural networks (FFNNs), leading to an increase of the transmission distance. We also compare the end-to-end SBRNN with a state-of-the-art IM/DD solution based on two level pulse amplitude modulation with an FFNN receiver, simultaneously processing multiple received symbols and approximating nonlinear Volterra equalization. Our results show that the SBRNN outperforms such systems at both 42 and 84 Gb/s, while training fewer parameters. Our novel SBRNN design aims at tailoring the end-to-end deep learning-based systems for communication over nonlinear channels with memory, such as the optical IM/DD fiber channel.