Bidirectional Long Short-term Memory Network Research Articles

Chinese Named Entity Recognition Based on Character-Word Vector Fusion

Due to the lack of explicit markers in Chinese text to define the boundaries of words, it is often more difficult to identify named entities in Chinese than in English. At present, the pretreatment of the character or word vector models is adopted in the training of the Chinese named entity recognition model. Aimed at the problems that taking character vector as an input of the neural network cannot use the words’ semantic meanings and give up the words’ explicit boundary information, and taking the word vector as an input of the neural network relies on the accuracy of the segmentation algorithms, a Chinese named entity recognition model based on character word vector fusion CWVF-BiLSTM-CRF (Character Word Vector Fusion-Bidirectional Long-Short Term Memory Networks-Conditional Random Field) is proposed in this paper. First, the Word2Vec is used to obtain the corresponding dictionaries of character-character vector and word-word vector. Second, the character-word vector is integrated as the input unit of the BiLSTM (Bidirectional Long-Short Term Memory) network, and then, the problem of an unreasonable tag sequence is solved using the CRF (conditional random field). By using the presented model, the dependence on the accuracy of the word segmentation algorithm is reduced, and the words’ semantic characteristics are effectively applied. The experimental results show that the model based on character-word vector fusion improves the recognition effect of the Chinese named entity.

DeepCINAC: A Deep-Learning-Based Python Toolbox for Inferring Calcium Imaging Neuronal Activity Based on Movie Visualization.

Two-photon calcium imaging is now widely used to infer neuronal dynamics from changes in fluorescence of an indicator. However, state-of-the-art computational tools are not optimized for the reliable detection of fluorescence transients from highly synchronous neurons located in densely packed regions such as the CA1 pyramidal layer of the hippocampus during early postnatal stages of development. Indeed, the latest analytical tools often lack proper benchmark measurements. To meet this challenge, we first developed a graphical user interface (GUI) allowing for a precise manual detection of all calcium transients from imaged neurons based on the visualization of the calcium imaging movie. Then, we analyzed movies from mouse pups using a convolutional neural network (CNN) with an attention process and a bidirectional long-short term memory (LSTM) network. This method is able to reach human performance and offers a better F1 score (harmonic mean of sensitivity and precision) than CaImAn to infer neural activity in the developing CA1 without any user intervention. It also enables automatically identifying activity originating from GABAergic neurons. Overall, DeepCINAC offers a simple, fast and flexible open-source toolbox for processing a wide variety of calcium imaging datasets while providing the tools to evaluate its performance.

Human gesture recognition under degraded environments using 3D-integral imaging and deep learning.

In this paper, we propose a spatio-temporal human gesture recognition algorithm under degraded conditions using three-dimensional integral imaging and deep learning. The proposed algorithm leverages the advantages of integral imaging with deep learning to provide an efficient human gesture recognition system under degraded environments such as occlusion and low illumination conditions. The 3D data captured using integral imaging serves as the input to a convolutional neural network (CNN). The spatial features extracted by the convolutional and pooling layers of the neural network are fed into a bi-directional long short-term memory (BiLSTM) network. The BiLSTM network is designed to capture the temporal variation in the input data. We have compared the proposed approach with conventional 2D imaging and with the previously reported approaches using spatio-temporal interest points with support vector machines (STIP-SVMs) and distortion invariant non-linear correlation-based filters. Our experimental results suggest that the proposed approach is promising, especially in degraded environments. Using the proposed approach, we find a substantial improvement over previously published methods and find 3D integral imaging to provide superior performance over the conventional 2D imaging system. To the best of our knowledge, this is the first report that examines deep learning algorithms based on 3D integral imaging for human activity recognition in degraded environments.

A novel hybrid bidirectional unidirectional LSTM network for dynamic hand gesture recognition with Leap Motion

Due to the recent development of machine learning and sensor innovations, hand gesture recognition systems become promisingfor the digital entertainment field. In this paper, we propose a dynamic hand gesture recognition approach using touchless hand motions over a Leap Motion device. First, we analyze the sequential time series data gathered from Leap Motion using Long Short-Term Memory (LSTM) recurrent neural networks for recognition purposes. We exploit basic unidirectional LSTM and bidirectional LSTM separately. Then, we propound novel architecture by combining the aforementioned models with additional components to give a final prediction network, named Hybrid Bidirectional Unidirectional LSTM (HBU-LSTM). The suggested network improves the model performance significantly by considering the spatial and temporal dependencies between the Leap Motion data and the network layers during the forward and backward pass. The recognition models are examined on two available benchmark datasets, named the LeapGestureDB dataset and the RIT dataset. Experiments demonstrate the potential of the proposed HBU-LSTM network for dynamic hand gesture recognition, with an average recognition rate reaching approximately 90%. Our suggested approach reaches superior performance, in terms of accuracy and computational complexity, over some existing methods for hand gesture recognition.

Multimodal multitask deep learning model for Alzheimer’s disease progression detection based on time series data

Early prediction of Alzheimer’s disease (AD) is crucial for delaying its progression. As a chronic disease, ignoring the temporal dimension of AD data affects the performance of a progression detection and medically unacceptable. Besides, AD patients are represented by heterogeneous, yet complementary, multimodalities. Multitask modeling improves progression-detection performance, robustness, and stability. However, multimodal multitask modeling has not been evaluated using time series and deep learning paradigm, especially for AD progression detection. In this paper, we propose a robust ensemble deep learning model based on a stacked convolutional neural network (CNN) and a bidirectional long short-term memory (BiLSTM) network. This multimodal multitask model jointly predicts multiple variables based on the fusion of five types of multimodal time series data plus a set of background (BG) knowledge. Predicted variables include AD multiclass progression task, and four critical cognitive scores regression tasks. The proposed model extracts local and longitudinal features of each modality using a stacked CNN and BiLSTM network. Concurrently, local features are extracted from the BG data using a feed-forward neural network. Resultant features are fused to a deep network to detect common patterns which jointly used to predict the classification and regression tasks. To validate our model, we performed six experiments on five modalities from Alzheimer’s Disease Neuroimaging Initiative (ADNI) of 1536 subjects. The results of the proposed approach achieve state-of-the-art performance for both multiclass progression and regression tasks. Moreover, our approach can be generalized in other medial domains to analyze heterogeneous temporal data for predicting patient’s future status.

Human activity recognition using improved complete ensemble EMD with adaptive noise and long short-term memory neural networks

The recognition of human activities is a topic of great relevance due to its wide range of applications. Different approaches have been proposed to recognize human activities, ranging from the comparison of signals with thresholds to the application of deep and machine learning techniques. In this work, the classification of six human activities (walking, walking downstairs, walking upstairs, standing, sitting, and lying down) is performed using bidirectional LSTM networks that exploit intrinsic mode function (IMF) representation of inertial signals. Records with inertial signals (accelerometer and gyroscope) of 2.56s, available at the UCI Machine Learning Repository, were collected from 30 subjects using a smartphone. First, inertial signals were standardized to take them to the same scale and were decomposed into IMF using the improved complete ensemble empirical mode decomposition with adaptive noise (ICEEMDAN). IMF were then segmented (split) into nine segments of 1.28s with 12.5% overlap and introduced to a first network with four outputs to identify the dynamic activities and the statics as a single class called “statics”, giving 98.86% accuracy. Then, the non-segmented IMF of the records assigned to the statics class were introduced to a second network to classify their three activities, giving an accuracy of 88.46%. In total, 92.91% accuracy was obtained to classify the six human activities. This performance is because ICEEMDAN allowed the extraction of information that was embedded in the signal, and the segmentation of the IMF allowed the network to discriminate between static and dynamic activities.

Neonatal sleep stage identification using long short-term memory learning system.

Neonatal sleep analysis at the neonatal intensive care units (NICU) is critical for the diagnosis of any brain growth risks during the early stages of life. In this paper, an investigation is carried out on the use of a long short-term memory (LSTM) learning system in automatic sleep stage scoring in neonates. The developed algorithm automatically classifies sleep stages based on inputs from a single channel EEG recording. Up to this date, only a single study have developed an approach for automatic sleep stage scoring in neonatal sleep signals using deep neural network (DNN). A total of 5095 sleep stages signals acquired from EEG recordings of the University of Pittsburgh are used in this study. The sleep stages are annotated by a medical doctor from the Pediatric Neurology Department of Case Western Reserve University for three neonatal sleep stages including the awake (W), active sleep (AS), and quiet sleep (QS) stages on every 60-s epoch. The signals are pre-processed through normalization and filtering. The resulted signals are divided following 4-, 6-, and 10-fold cross-validation schemes. The training and classification process is done using a bi-directional LSTM network classifier built with pre-defined training parameters. At the end, the developed algorithm is evaluated along with a complete summary table that reports the results of this study and other state-of-the-art studies. The current study achieved high levels of Cohen's kappa (κ), accuracy, and F1 score with 91.37%, 96.81%, and 94.43%, respectively. Based on the confusion matrix, the overall true positives percentage reached 95.21%. The developed algorithm gave promising results in automatic sleep stage scoring in neonatal sleep signals. Future work include LSTM architecture and training parameters improvements to enhance the overall accuracy of the classifier.

Noise-Robust Speech Recognition System based on Multimodal Audio-Visual Approach Using Different Deep Learning Classification Techniques

This paper extends an earlier work on designing a speech recognition system based on Hidden Markov Model (HMM) classification technique of using visual modality in addition to audio modality[1]. Improved off traditional HMM-based Automatic Speech Recognition (ASR) accuracy is achieved by implementing a technique using either RNN-based or CNN-based approach. This research is intending to deliver two contributions: The first contribution is the methodology of choosing the visual features by comparing different visual features extraction methods like Discrete Cosine Transform (DCT), blocked DCT, and Histograms of Oriented Gradients with Local Binary Patterns (HOG+LBP), and applying different dimension reduction techniques like Principal Component Analysis (PCA), auto-encoder, Linear Discriminant Analysis (LDA), t-distributed Stochastic Neighbor Embedding (t-SNE) to find the most effective features vector size. Then the obtained visual features are early integrated with the audio features obtained by using Mel Frequency Cepstral Coefficients (MFCCs) and feed the combined audio-visual feature vector to the classification process. The second contribution of this research is the methodology of developing the classification process using deep learning by comparing different Deep Neural Network (DNN) architectures like Bidirectional Long-Short Term Memory (BiLSTM) and Convolution Neural Network (CNN) with the traditional HMM. The proposed model is evaluated on two multi-speakers AV-ASR datasets named AVletters and GRID with different SNR. The model performs speaker-independent experiments in AVlettter dataset and speaker-dependent in GRID dataset.

Using Deep Learning to Forecast Maritime Vessel Flows

Forecasting vessel flows is important to the development of intelligent transportation systems in the maritime field, as real-time and accurate traffic information has favorable potential in helping a maritime authority to alleviate congestion, mitigate emission of GHG (greenhouse gases) and enhance public safety, as well as assisting individual vessel users to plan better routes and reduce additional costs due to delays. In this paper, we propose three deep learning-based solutions to forecast the inflow and outflow of vessels within a given region, including a convolutional neural network (CNN), a long short-term memory (LSTM) network, and the integration of a bidirectional LSTM network with a CNN (BDLSTM-CNN). To apply those solutions, we first divide the given maritime region into grids, then we forecast the inflow and outflow for all the grids. Experimental results based on the real AIS (Automatic Identification System) data of marine vessels in Singapore demonstrate that the three deep learning-based solutions significantly outperform the conventional method in terms of mean absolute error and root mean square error, with the performance of the BDLSTM-CNN-based hybrid solution being the best.

Pattern Recognition of the Vertical Hydraulic Fracture Shapes in Coalbed Methane Reservoirs Based on Hierarchical Bi-LSTM Network

Pattern recognition of the hydraulic fracture shapes is very important and complex for the refracturing design of coalbed methane (CBM) wells. In this paper, we explore a new idea by regarding the pattern recognition process as understanding what a CBM reservoir “says” during hydraulic fracturing. Then we present a hierarchical Bidirectional LSTM (Bi-LSTM) network to recognize the pattern of hydraulic fracture geometry in CBM reservoirs. Inputting the wavelet denoised sequences of data to the presented network, we can extract the implicit features of the hydraulic sand fracturing operation curves and automatically combine them to make the classification of the fracture shapes. With this method, we can cope with the problems happened in early stage of the CBM field development such as the lack of monitoring wells and the information of rock mechanics. Moreover, the experiences of the engineers and the measured data are combinationally used, which can efficiently reduce the subjectivity and assist the engineers to make the refracturing design. The validity of this method is verified by the testing data and comparing with the simulated results of Fracpro PT software.

Driving Behavior Prediction Considering Cognitive Prior and Driving Context

Driving behavior plays a key role in the interaction between vehicle and driver in transportation systems. Some applications about driving behavior in Advanced Driver Assistance Systems (ADAS) improve driving safety significantly. This paper introduces the driving context and models driving behavior in a combination of cognitive perspective and data-driven perspective. First, we use a cognitive fusion method by adding a delay time module to fuse the environmental information and inside information. To better capture the driving context relationship between outside and inside features, we transfer the behavior prediction task to the sequence labeling task by introducing the visual inertia hypothesis. We propose the Predictive-Bi-LSTM-CRF algorithm which used the Bidirectional Long-Short Term Memory Networks (Bi-LSTM) and Conditional Random Field (CRF) as the loss layer to model the driving behavior. Besides, we define a new comprehensive evaluation metric for the prediction task considering F1-score and the prediction time before maneuver together. Our experiment results achieve the state of art performance on the Brain4Cars dataset and demonstrate the applicability of our theory.

A water quality prediction method based on the multi-time scale bidirectional long short-term memory network.

As an important factor affecting the mangrove wetland ecosystem, water quality has become the focus of attention in recent years. Therefore, many studies have focused on the prediction of water quality to help establish a regulatory framework for the assessment and management of water pollution and ecosystem health. To make a more accurate and comprehensive forecast analysis of water quality, we propose a method for water quality prediction based on the multi-time scale bidirectional LSTM network. In the method, we improve data integrity and data volume through data preprocessing. And the network processes input data forward and backward and considers the dependencies at multiple time scales. Besides, we use the Box-Behnken experimental design method to adjust hyper-parameters in the process of modeling. In this study, we apply this method to the water quality prediction research of Beilun Estuary, and the performance of our proposed model is evaluated and compared with other models. The experiment results show that this model has better performance in water quality prediction than that of using LSTM or bidirectional LSTM alone. Graphical Abstract Schematic of research work.

Semantic Segmentation of QRS Complex in Single Channel ECG with Bidirectional LSTM Networks

In this paper, bidirectional Long Short-Term Memory (BiLSTM) networks are designed to realize the semantic segmentation of QRS complex in single channel electrocardiogram (ECG) for the tasks of R peak detection and heart rate estimation. Three types of seq2seq BiLSTM networks are introduced, including the densely connected BiLSTM with attention model, the BiLSTM U-Net, and the BiLSTM U-Net++. To alleviate the sparse problem of the QRS labels, symmetric label expansion is applied by extending the single R peak into a time interval of fixed length. Linear ensemble method is introduced that averages the outputs of different BiLSTM networks. The cross-validation results show significant increase of the accuracy and decrease of discontinuous gaps in the QRS interval prediction by the ensembling over singular neural networks.

Parallel reinforcement learning-based energy efficiency improvement for a cyber-physical system

As a complex and critical cyber-physical system &#x0028 CPS &#x0029 , the hybrid electric powertrain is significant to mitigate air pollution and improve fuel economy. Energy management strategy &#x0028 EMS &#x0029 is playing a key role to improve the energy efficiency of this CPS. This paper presents a novel bidirectional long short-term memory &#x0028 LSTM &#x0029 network based parallel reinforcement learning &#x0028 PRL &#x0029 approach to construct EMS for a hybrid tracked vehicle &#x0028 HTV &#x0029 . This method contains two levels. The high-level establishes a parallel system first, which includes a real powertrain system and an artificial system. Then, the synthesized data from this parallel system is trained by a bidirectional LSTM network. The lower-level determines the optimal EMS using the trained action state function in the model-free reinforcement learning &#x0028 RL &#x0029 framework. PRL is a fully data-driven and learning-enabled approach that does not depend on any prediction and predefined rules. Finally, real vehicle testing is implemented and relevant experiment data is collected and calibrated. Experimental results validate that the proposed EMS can achieve considerable energy efficiency improvement by comparing with the conventional RL approach and deep RL.

Real Network Traffic Collection and Deep Learning for Mobile App Identification

The proliferation of mobile devices over recent years has led to a dramatic increase in mobile traffic. Demand for enabling accurate mobile app identification is coming as it is an essential step to improve a multitude of network services: accounting, security monitoring, traffic forecasting, and quality-of-service. However, traditional traffic classification techniques do not work well for mobile traffic. Besides, multiple machine learning solutions developed in this field are severely restricted by their handcrafted features as well as unreliable datasets. In this paper, we propose a framework for real network traffic collection and labeling in a scalable way. A dedicated Android traffic capture tool is developed to build datasets with perfect ground truth. Using our established dataset, we make an empirical exploration on deep learning methods for the task of mobile app identification, which can automate the feature engineering process in an end-to-end fashion. We introduce three of the most representative deep learning models and design and evaluate our dedicated classifiers, namely, a SDAE, a 1D CNN, and a bidirectional LSTM network, respectively. In comparison with two other baseline solutions, our CNN and RNN models with raw traffic inputs are capable of achieving state-of-the-art results regardless of TLS encryption. Specifically, the 1D CNN classifier obtains the best performance with an accuracy of 91.8% and macroaverage F-measure of 90.1%. To further understand the trained model, sample-specific interpretations are performed, showing how it can automatically learn important and advanced features from the uppermost bytes of an app’s raw flows.

Automatic Assessment of Sentence-Level Dysarthria Intelligibility Using BLSTM

Dysarthria is a motor speech impairment, often characterized by slow and slurred speech that is generally incomprehensible by human listeners. An understanding of the intelligibility level of the patient's dysarthric speech can provide an insight into the progression/status of the underlying cause and is essential for planning therapy. Automatic assessment of dysarthric speech intelligibility can be of immense value and serve to assist speech language pathologists in diagnosis and therapy. However, this is a non-trivial problem due to the high intra and inter speaker variability in dysarthric speech. In this article we propose a machine learning-based method to automatically classify dysarthric speech into intelligible (I) and non-intelligible (NI) using Bidirectional Long-Short Term Memory (BLSTM) Networks. We explored balancing of training data to represent both the classes almost equally and its implications on the binary classification. Additionally, we present a mechanism to use the available pre-trained acoustic models for transfer-learning. It was observed that the transfer learning method was able to handle channel noise. This technique provided significant improvement of roughly 6% as compared to traditional machine learning method.

Video-based person re-identification using a novel feature extraction and fusion technique

Person re-identification has received extensive attention in the academic community. In this paper, a novel multiple feature fusion network (MPFF-Net) is proposed for video-based person re-identification. The proposed network is used to obtain the robust and discriminative feature representation for describing the pedestrian in the video, which contains the hand-crafted and deep-learned parts. First, the image-level features of all consecutive frames are extracted. Then the hand-crafted branch uses these descriptors to obtain the average feature of the video and the information of frame-to-frame differences. The deep-learned branch is based on the bidirectional LSTM (BiLSTM) network. It is responsible for aggregating frame-wise representations of human regions and yielding sequence-level features. Furthermore, the problem of misalignment is taken into account in this branch. Finally, the hand-crafted and deep-learned parts are considered to be complementary, and the fusion of them can help to capture the complete information of the video. Extensive experiments are conducted on the iLIDS-VID, PRID2011 and MARS datasets. The results demonstrate that the proposed algorithm outperforms state-of-the-art video-based re-identification methods.

Speech Emotion Recognition by Combining a Unified First-Order Attention Network With Data Balance

In the domain of speech emotion recognition (SER), generally there is an unbalanced data distribution of emotional samples in existing emotional datasets. Moreover, different fragment areas in an utterance contribute diversely to SER. To address these two issues, this paper proposes a new SER method by combining a unified first-order attention network with data balance. The proposed method firstly utilizes the strategy of data balance to augment and balance the training data. Then, a pre-trained convolutional neural network (CNN) model (i.e., VGGish) is fine-tuned on target emotional datasets to learn segment-level speech features from the extracted Log Mel-spectrograms. Next, the unified first-order attention mechanism, including different feature-pooling strategies such as sum, min, max, mean, and standard deviation (std), is embedded into the output of a bi-directional long short-term memory (Bi-LSTM) network. This is used for learning high-level discriminative segment-level features, and simultaneously aggregating the learned segment-level features into fixed-length utterance-level features for SER. Finally, based on utterance-level features, the softmax layer in a Bi-LSTM network is adopted to conduct final emotion classification task. Extensive experiments are implemented on three public datasets such as BAUM-1s, AFEW5.0, and CHEAVD2.0, demonstrate the advantage of the proposed method.

Polluted gas quantitative detection in multi-gas sensor based on bidirectional long-short term memory network

Quantitative detection of polluted gas by electronic nose can reduce the cost of detection and improve the efficiency of measurement. Through the effective pattern recognition method, the electronic nose can analyse the continuous periodic data and realise the detection of specific tasks. In this paper, the pollution gas concentration prediction method based on bidirectional long-short term memory network (Bi-LSTM) is proposed. And the effect of the Bi-LSTM model with different time steps, hidden layers and different combinations of sensor features on the performance of pollution gas prediction model is investigated. This method can extract deep features by automatically learning the gas response information of the sensor array, and its performance is better. The proposed method is verified on the air quality dataset, which proves that the proposed method has high accuracy in the quantitative detection of gas concentration based on electronic nose information.

Polluted gas quantitative detection in multi-gas sensor based on bidirectional long-short term memory network

Quantitative detection of polluted gas by electronic nose can reduce the cost of detection and improve the efficiency of measurement. Through the effective pattern recognition method, the electronic nose can analyse the continuous periodic data and realise the detection of specific tasks. In this paper, the pollution gas concentration prediction method based on bidirectional long-short term memory network (Bi-LSTM) is proposed. And the effect of the Bi-LSTM model with different time steps, hidden layers and different combinations of sensor features on the performance of pollution gas prediction model is investigated. This method can extract deep features by automatically learning the gas response information of the sensor array, and its performance is better. The proposed method is verified on the air quality dataset, which proves that the proposed method has high accuracy in the quantitative detection of gas concentration based on electronic nose information.