Long Short-term Memory Units Research Articles

DEEPSMP: A deep learning model for predicting the ectodomain shedding events of membrane proteins.

Membrane proteins play essential roles in modern medicine. In recent studies, some membrane proteins involved in ectodomain shedding events have been reported as the potential drug targets and biomarkers of some serious diseases. However, there are few effective tools for identifying the shedding event of membrane proteins. So, it is necessary to design an effective tool for predicting shedding event of membrane proteins. In this study, we design an end-to-end prediction model using deep neural networks with long short-term memory (LSTM) units and attention mechanism, to predict the ectodomain shedding events of membrane proteins only by sequence information. Firstly, the evolutional profiles are encoded from original sequences of these proteins by Position-Specific Iterated BLAST (PSI-BLAST) on Uniref50 database. Then, the LSTM units which contain memory cells are used to hold information from past inputs to the network and the attention mechanism is applied to detect sorting signals in proteins regardless of their position in the sequence. Finally, a fully connected dense layer and a softmax layer are used to obtain the final prediction results. Additionally, we also try to reduce overfitting of the model by using dropout, L2 regularization, and bagging ensemble learning in the model training process. In order to ensure the fairness of performance comparison, firstly we use cross validation process on training dataset obtained from an existing paper. The average accuracy and area under a receiver operating characteristic curve (AUC) of five-fold cross-validation are 81.19% and 0.835 using our proposed model, compared to 75% and 0.78 by a previously published tool, respectively. To better validate the performance of the proposed model, we also evaluate the performance of the proposed model on independent test dataset. The accuracy, sensitivity, and specificity are 83.14%, 84.08%, and 81.63% using our proposed model, compared to 70.20%, 71.97%, and 67.35% by the existing model. The experimental results validate that the proposed model can be regarded as a general tool for predicting ectodomain shedding events of membrane proteins. The pipeline of the model and prediction results can be accessed at the following URL: http://www.csbg-jlu.info/DeepSMP/.

Content Classification With Electroglottograph

Electroglottograph(EGG) is a physiological signal collected from the throat which reflects the vocal cord movement. EGG signals can be still collected from patients without speaking ability or from the extremely noisy environments. Additionally, the trends of the vocal cord movement will be distinctive for long enough Chinese sentences with different contents. Therefore, it is valuable and possible to carry out the research of applying only the EGG signals for content classification or recognition. In this paper, a content classification method with EGG was proposed, which consists of an EGG feature extraction module and a classification network based on LSTM(Long Short-Term Memory) units. The EGG feature extraction module was composed of three parts: the voiced segments extraction, the feature extraction and the F 0 smoothing. The classification network was made of a three-layer bidirectional LSTM encoder. This method achieved 91.12% accuracy on the validation set in 20-class content classification experiment, which provides the reference for further study in content classification and recognition with EGG signals.

Estimation of Hand Motion from Piezoelectric Soft Sensor Using Deep Recurrent Network

Soft sensors are attracting significant attention in human–machine interaction due to their high flexibility and adaptability. However, estimating motion state from these sensors is difficult due to their nonlinearity and noise. In this paper, we propose a deep learning network for a smart glove system to predict the moving state of a piezoelectric soft sensor. We implemented the network using Long-Short Term Memory (LSTM) units and demonstrated its performance in a real-time system based on two experiments. The sensor’s moving state was estimated and the joint angles were calculated. Since we use moving state in the sensor offset calculation and the offset value is used to estimate the angle value, the accurate moving state estimation results in good performance for angle value estimation. The proposed network performed better than the conventional heuristic method in estimating the moving state. It was also confirmed that the calculated values successfully mimic the joint angles measured using a leap motion controller.

Recurrent neural network closure of parametric POD-Galerkin reduced-order models based on the Mori-Zwanzig formalism

Closure modeling based on the Mori-Zwanzig formalism has proven effective to improve the stability and accuracy of projection-based model order reduction. However, closure models are often expensive and infeasible for complex nonlinear systems. Towards efficient model reduction of general problems, this paper presents a recurrent neural network (RNN) closure of parametric POD-Galerkin reduced-order model. Based on the short time history of the reduced-order solutions, the RNN predicts the memory integral which represents the impact of the unresolved scales on the resolved scales. A conditioned long short term memory (LSTM) network is utilized as the regression model of the memory integral, in which the POD coefficients at a number of time steps are fed into the LSTM units, and the physical/geometrical parameters are fed into the initial hidden state of the LSTM. The reduced-order model is integrated in time using an implicit-explicit (IMEX) Runge-Kutta scheme, in which the memory term is integrated explicitly and the remaining right-hand-side term is integrated implicitly to improve the computational efficiency. Numerical results demonstrate that the RNN closure can significantly improve the accuracy and efficiency of the POD-Galerkin reduced-order model of nonlinear problems. The POD-Galerkin reduced-order model with the RNN closure is also shown to be capable of making accurate predictions, well beyond the time interval of the training data.

Short-Term Prediction of PM2.5 Pollution with Deep Learning Methods

<p>Particulate matter (PM), classified according to aerodynamic diameter, is one of the harmful pollutants causing health damaging effects. It is considered as cancerogenic by the World Health Organization (WHO) because of the substances found in the chemical composition of PM. In this study, short-term prediction of PM2.5 pollution at 1, 2 and 3 hours was modelled using deep learning methods. Three deep learning algorithms and the combination thereof were evaluated: Long-short term memory units (LSTM), recurrent neural networks (RNN) and gated recurrent unit (GRU). Air Quality Monitoring Stations of the Ministry of Environment and Urbanization of Turkey were utilized to obtain the data. Specifically, meteorological and air pollution data were obtained from a monitoring station located in Keçiören District of Ankara. Several trials were conducted using different combinations of RNN, GRU and LSTM models. Pollutant concentrations and meteorological factors were integrated into the model as input parameters to predict PM2.5 concentration for 1,2 and 3 hours. Best results with R2 of 0.83, 0.7 and 0.63 for 1-, 2-, and 3-hour predictions, respectively, were obtained by using a combination of GRU and RNN models. The results of this study are promising for explaining the effect of different deep learning models on prediction performance.</p>

Study of the influence of lexicon and language restrictions on computer assisted transcription of historical manuscripts

State-of-the-art Handwritten Text Recognition (HTR) systems allow transcribers to speed-up the transcription of handwritten text images. These systems provide transcribers an initial draft transcription that can be corrected with less effort than transcribing the handwritten text images from scratch. Currently, even the draft transcriptions offered by the most advanced HTR systems contain errors. Therefore, the supervision of this draft by a human transcriber is still necessary to obtain the correct transcription of the handwritten text images. This supervision can be eased by using interactive and assistive transcription systems, where the transcriber and the automatic system cooperate in the amending process.In this paper, the draft transcription is provided by an HTR system based on Convolutional and Recurrent Neural Networks with Bidirectional Long-Short Term Memory units, and the assistive system is fed by lattices generated by using Weighted Finite State Transducers. The influence of the lexicon and language restrictions on the performance of our computer assisted transcription system is evaluated on three historical manuscripts.The transcriptions offered by the proposed HTR system present very low error rates for the studied historical manuscripts. However, our assistive transcription system without lexicon or language restrictions is able to provide an additional reduction on the human effort required to correct the transcriptions in more than 50% over the transcriptions offered by the HTR system.

Change Detection in Multisource VHR Images via Deep Siamese Convolutional Multiple-Layers Recurrent Neural Network

With the rapid development of Earth observation technology, very-high-resolution (VHR) images from various satellite sensors are more available, which greatly enrich the data source of change detection (CD). Multisource multitemporal images can provide abundant information on observed landscapes with various physical and material views, and it is exigent to develop efficient techniques to utilize these multisource data for CD. In this article, we propose a novel and general deep siamese convolutional multiple-layers recurrent neural network (RNN) (SiamCRNN) for CD in multitemporal VHR images. Superior to most VHR image CD methods, SiamCRNN can be used for both homogeneous and heterogeneous images. Integrating the merits of both convolutional neural network (CNN) and RNN, SiamCRNN consists of three subnetworks: deep siamese convolutional neural network (DSCNN), multiple-layers RNN (MRNN), and fully connected (FC) layers. The DSCNN has a flexible structure for multisource image and is able to extract spatial-spectral features from homogeneous or heterogeneous VHR image patches. The MRNN stacked by long-short term memory (LSTM) units is responsible for mapping the spatial-spectral features extracted by DSCNN into a new latent feature space and mining the change information between them. In addition, FC, the last part of SiamCRNN, is adopted to predict change probability. The experimental results in two homogeneous data sets and one challenging heterogeneous VHR images data set demonstrate that the promising performances of the proposed network outperform several state-of-the-art approaches.

Traffic index prediction based on sequence to sequence learning

The traffic index prediction plays an important role in many intelligent transportation applications. The challenge mainly comes from the strong nonlinearities in the changing of traffic index, which is caused by transitions of various traffic states including smooth traffic, congestion, breakdown and recovery. Deep learning performs well in capturing and describing the nonlinear relationship among variables. This paper uses a Sequence to Sequence (Seq2Seq) deep learning framework to predict the traffic index. In the proposed method, we use Long Short Term Memory (LSTM) as the basic circulating unit. And the LSTM units are stacked to extract the time changing characteristic and the periodic characteristic of the input traffic index sequence. Besides, the extracted feature vectors are decoded by another single layer LSTM to predict the traffic index of each time after inputting sequence. Experiments are conducted on Beijing traffic index datasets. The proposed method outperformed Autoregressive Integrated Moving Average Model (ARIMA) and LSTM under some commonly used evaluation metrics.

Convolutional-Type Neural Networks for Fading Channel Forecasting

In this article, a series of convolutional-type predictive neural networks are proposed for the issue of fading channel forecasting for orthogonal frequency-division multiplexing (OFDM) transmission systems in a multiple-input and multiple-output (MIMO) mode via a noisy channel. The proposed neural networks all employ convolutional connections that operate in a translation-invariant manner in the frequency domain of the time-varying channel transfer function, which effectively tackles the essential challenges of high dimensionality and denoising. Each of the proposed convolutional-type neural networks is built on a specific overall network architecture and functions as an independent predictor that offers advantages regarding a specific aspect such as accuracy over a certain prediction span or computational effort. Comparative evaluations against common prediction methods such as the Kalman filtering scheme and the standard long-short term memory units (LSTMs) are provided on the basis of transmission simulations over dispersive fading channels with Rayleigh components according to the well-established 3GPP Long Term Evolution (LTE) standards.

Visual Scanpath Prediction Using IOR-ROI Recurrent Mixture Density Network.

A visual scanpath represents the human eye movements when scanning the visual field for acquiring and receiving visual information. Predicting visual scanpaths when a certain stimulus is presented plays an important role in modeling overt human visual attention and search behavior. In this paper, we presented an 'Inhibition of Return - Region of Interest' (IOR-ROI) recurrent mixture density network based framework learning to produce human-like visual scanpaths under task-free viewing conditions. The proposed model simultaneously predicts a sequence of ordered fixation positions and their corresponding fixation durations. Our model integrates bottom-up features and semantic features extracted by convolutional neural networks. Then the integrated feature maps are fed into the IOR-ROI Long Short-Term Memory (LSTM) which is the core component of the proposed model. The IOR-ROI LSTM is a dual LSTM unit, i.e., the IOR-LSTM and the ROI-LSTM, capturing IOR dynamics and gaze shift behavior simultaneously. IOR-LSTM simulates the visual working memory to adaptively maintain and update visual information regarding previously fixated regions. ROI-LSTM is responsible for predicting the next possible ROIs given the spatially inhibited image feature maps on the feature-wise basis. Fixation duration is predicted by a regression neural network given the viewing history and image feature maps corresponding to currently fixated ROI. Considering the eye movement pattern variations among subjects, a mixture density network is adopted to model the next fixation distribution as Gaussian mixtures and the fixation duration is also modeled using Gaussian distribution. Our model is evaluated on the OSIE and MIT low resolution eye-tracking datasets and experimental results indicate that the proposed method can achieve superior performance in predicting visual scanpaths. The code will be publicly available on URL: https://github.com/sunwj/scanpath.

Deep Learning-Based Firework Video Pattern Classification

The video analytics technology has been a rapidly improving discipline in the past decade. With the recent developments in computer vision, we now have the ability to mine massive video data to obtain a clear understanding of what is happening in the world. Because of the remarkable successes of deep learning, currently, we are able to improve video analysis performance significantly than traditional statistical approaches. This study focuses on classifying the patterns of firework videos with various deep learning techniques using spatial and temporal features, beyond common types of pattern classifications. Among successful artificial neural networks, Convolutional Neural Networks (CNN) have demonstrated superiority on modeling high-level visual concepts, while Long Short-term Memory (LSTM) and Gated Recurrent Unit (GRU) units have shown great talent in modeling temporal dynamics in video-based pattern classification. Our basic models consist of CNN, LSTM, and GRU and, we did experiments by fine-tuning the parameters of layers and using different dropout values with sequence LSTM and GRU models. Our experimental results demonstrated that the model with a sequence of LSTM units and double dropout layers—one for input and another for hidden layers—outperforms the other experimental models with the training accuracy of 83.05%.

AMRNN: attended multi-task recurrent neural networks for dynamic illness severity prediction

Illness severity prediction (ISP) is crucial for caregivers in the intensive care unit (ICU) while saving the life of patients. Existing ISP methods fail to provide sufficient evidence for the time-critical decision making in the dynamic changing environment. Moreover, the correlated temporal features in multivariate time-series are rarely be considered in existing machine learning-based ISP models. Therefore, in this paper, we propose a novel interpretable analysis framework which simultaneously analyses organ systems differentiated based on the pathological and physiological evidence to predict illness severity of patients in ICU. It not only timely but also intuitively reflects the critical conditions of patients for caregivers. In particular, we develop a deep interpretable learning model, namely AMRNN, which is based on the Multi-task RNNs and Attention Mechanism. Physiological features of each organ system in multivariate time series are learned by a single Long-Short Term Memory unit as a dedicated task. To utilize the functional and temporal relationships among organ systems, we use a shared LSTM task to exploit correlations between different learning tasks for further performance improvement. Real-world clinical datasets (MIMIC-III) are used for conducting extensive experiments, and our method is compared with the existing state-of-the-art methods. The experimental results demonstrated that our proposed approach outperforms those methods and suggests a promising way of evidence-based decision support.

An Empirical Test of GRUs and Deep Contextualized Word Representations on De-Identification.

De-identification aims to remove 18 categories of protected health information from electronic health records. Ideally, de-identification systems should be reliable and generalizable. Previous research has focused on improving performance but has not examined generalizability. This paper investigates both performance and generalizability. To improve current state-of-the-art performance based on long short-term memory (LSTM) units, we introduce a system that uses gated recurrent units (GRUs) and deep contextualized word representations, both of which have never been applied to de-identification. We measure performance and generalizability of each system using the 2014 i2b2/UTHealth and 2016 CEGS N-GRID de-identification datasets. We show that deep contextualized word representations improve state-of-the-art performance, while the benefit of switching LSTM units with GRUs is not significant. The generalizability of de-identification system significantly improved with deep contextualized word representations; in addition, LSTM units-based system is more generalizable than the GRUs-based system.

Dynamic temporal residual network for sequence modeling

The long short-term memory (LSTM) network with gating mechanism has been widely used in sequence modeling tasks including handwriting and speech recognition. As an LSTM network can be unfolded along the temporal dimension and its temporal depth is equal to the length of the input feature sequence, the introduction of gating might not be sufficient to completely model the dynamic temporal dependencies in sequential data. Inspired by the residual learning in ResNet, this paper proposes a dynamic temporal residual network (DTRN) by incorporating residual learning into an LSTM network along the temporal dimension. DTRN involves two networks: Its primary network consists of modified LSTM units with weighted shortcut connections for adjacent temporal outputs, while its secondary network generates dynamic weights for the shortcut connections. To validate the performance of DTRN, we conduct experiments on three commonly used public handwriting recognition datasets (IFN/ENIT, IAM and Rimes) and one speech recognition dataset (TIMIT). The experimental results show that the proposed DTRN has outperformed previously reported methods.

Cultural Heritage and the Intelligent Internet of Things

Fourth Industrial Revolution technologies, such as artificial intelligence, big data, the Internet of Things (IoT), and virtual reality, have disrupted legacy methods of operations and have led to progress in many industries worldwide. These technologies also affect the cultural and national heritage. IoT generates large volumes of streaming data; therefore, advanced data analytics using big data analytics and artificial neural networks is an important research topic. In this study, IoT sensor data was collected at the restored Woljeong Bridge, which was originally built in the eighth century, or AD 760, during the Silla Dynasty (57 BC--AD 935) in South Korea. We empirically evaluate a recurrent neural network with recurrent units, including a long short-term memory (LSTM) unit and a gated recurrent unit (GRU). Additionally, we evaluate hybrid deep-learning models (convolution neural networks [CNN]-LSTM and CNN-GRU) to build a prediction model, facilitating the preventive conservation of an invaluable cultural and national heritage site. The experimental results show that the LSTM unit is an effective and robust model. When comparing the hybrid models (i.e., the joint CNN-LSTM and CNN-GRU architectures), we found that the vanilla LSTM and GRU models had superior time-series prediction capabilities.

Fatigue driving recognition network: fatigue driving recognition via convolutional neural network and long short‐term memory units

Fatigue driving has become one of the major causes of traffic accidents. The authors propose an effective method capable of detecting fatigue state via the spatial-temporal feature of driver's eyes. In this work, the authors consider fatigue detection as image-based sequence recognition and an end-to-end trainable convolutional neural network with long short-term memory (LSTM) units is designed. First, the authors apply a deep cascaded multi-task framework to extract eye region from infrared videos. Then the spatial features are learned by deep convolutional layers and the relationships between adjacent frames are analysed via LSTM units. Finally, through authors' model, a sequence-level prediction for driving state is produced. The proposed method achieves superior accuracy over the state-of-the-art techniques on authors' own dataset. Experimental results demonstrate the feasibility of authors' method.

A deep network for tissue microstructure estimation using modified LSTM units.

Diffusion magnetic resonance imaging(dMRI) offers a unique tool for noninvasively assessing tissue microstructure. However, accurate estimation of tissue microstructure described by complicated signal models can be challenging when a reduced number of diffusion gradients are used. Deep learning based microstructure estimation has recently been developed and achieved promising results. In particular, optimization-based learning, where deep network structures are constructed by unfolding the iterative processes performed for solving optimization problems, has demonstrated great potential in accurate microstructure estimation with a reduced number of diffusion gradients. In this work, using the optimization-based learning strategy, we propose a deep network structure that is motivated by the use of historical information in iterative optimization for tissue microstructure estimation, and such incorporation of historical information has not been previously explored in the design of deep networks for microstructure estimation. We assume that (1) diffusion signals can be sparsely represented by a dictionary and its coefficients jointly in the spatial and angular domain, and (2) tissue microstructure can be computed from the sparse representation. Following these assumptions, our network comprises two cascaded stages. The first stage takes image patches as input and computes the spatial-angular sparse representation of the input with learned weights. Specifically, the network structure in the first stage is constructed by unfolding an iterative process for solving sparse reconstruction problems, where historical information is incorporated. The components in this network can be shown to correspond to modified long short-term memory(LSTM) units. In the second stage, fully connected layers are added to compute the mapping from the sparse representation to tissue microstructure. The weights in the two stages are learned jointly by minimizing the mean squared error of microstructure estimation. Experiments were performed on dMRI scans with a reduced number of diffusion gradients. For demonstration, we evaluated the estimation of tissue microstructure described by three signal models: the neurite orientation dispersion and density imaging(NODDI) model, the spherical mean technique(SMT) model, and the ensemble average propagator(EAP) model. The results indicate that the proposed approach outperforms competing methods.

Decoding Movements from Cortical Ensemble Activity Using a Long Short-Term Memory Recurrent Network.

Although many real-time neural decoding algorithms have been proposed for brain-machine interface (BMI) applications over the years, an optimal, consensual approach remains elusive. Recent advances in deep learning algorithms provide new opportunities for improving the design of BMI decoders, including the use of recurrent artificial neural networks to decode neuronal ensemble activity in real time. Here, we developed a long-short term memory (LSTM) decoder for extracting movement kinematics from the activity of large (N = 134-402) populations of neurons, sampled simultaneously from multiple cortical areas, in rhesus monkeys performing motor tasks. Recorded regions included primary motor, dorsal premotor, supplementary motor, and primary somatosensory cortical areas. The LSTM's capacity to retain information for extended periods of time enabled accurate decoding for tasks that required both movements and periods of immobility. Our LSTM algorithm significantly outperformed the state-of-the-art unscented Kalman filter when applied to three tasks: center-out arm reaching, bimanual reaching, and bipedal walking on a treadmill. Notably, LSTM units exhibited a variety of well-known physiological features of cortical neuronal activity, such as directional tuning and neuronal dynamics across task epochs. LSTM modeled several key physiological attributes of cortical circuits involved in motor tasks. These findings suggest that LSTM-based approaches could yield a better algorithm strategy for neuroprostheses that employ BMIs to restore movement in severely disabled patients.

Constructing a Chinese electronic medical record corpus for named entity recognition on resident admit notes

BackgroundElectronic Medical Records(EMRs) contain much medical information about patients. Medical named entity extracting from EMRs can provide value information to support doctors’ decision making. The research on information extraction of Chinese Electronic Medical Records is still behind that has done in English.MethodsThis paper proposed a practical annotation scheme for medical entity extraction on Resident Admit Notes (RANs), and a model which can automatic extract medical entity. Nine types of clinical entities, four types of clinical relationships were defined in our annotation scheme. An end-to-end deep neural network with convolution neural network and long-short term memory units was applied in our model for the medical named entity recognition(NER).ResultWe annotated RANs in three rounds. The overall F-score of annotation consistency was up to 97.73%. And our NER model on the 255 annotated RANs achieved the best F-score of 91.08%.ConclusionThe annotation scheme and the model for NER in this paper are effective to extract medical named entity from RANs and provide the basis for fully excavating the patient’s information.

Nonlinear Dynamic Soft Sensor Modeling With Supervised Long Short-Term Memory Network

Soft sensor has been extensively utilized in industrial processes for prediction of key quality variables. To build an accurate virtual sensor model, it is very significant to model the dynamic and nonlinear behaviors of process sequential data properly. Recently, a long short-term memory (LSTM) network has shown great modeling ability on various time series, in which basic LSTM units can handle data nonlinearities and dynamics with a dynamic latent variable structure. However, the hidden variables in the basic LSTM unit mainly focus on describing the dynamics of input variables, which lack representation for the quality data. In this paper, a supervised LSTM (SLSTM) network is proposed to learn quality-relevant hidden dynamics for soft sensor application, which is composed of basic SLSTM unit at each sampling instant. In the basic SLSTM unit, the quality and input variables are simultaneously utilized to learn the dynamic hidden states, which are more relevant and useful for quality prediction. The effectiveness of the proposed SLSTM network is demonstrated on a penicillin fermentation process and an industrial debutanizer column.