Parallel Long Short-term Memory Research Articles

Data-Driven Detection of Video Forgery for Secured Microcontrollers: Dataset and Method

The microcontrollers in a camera often capture videos at a low frame rate due to limited processing capability. To satisfy the requirement of high quality of service, low-frame-rate videos are often forged as the high-frame-rate ones by the Frame Rate Up-Conversion (FRUC) operation. Therefore, detecting the existence of FRUC has become a necessary job for secured microcontrollers. In this paper, we propose a data-driven detection to identify whether a video is forged by FRUC. The core of detection is the creation of a large-scale video dataset VifFRUC (Videos forged by FRUC). Various types of forged videos can continue to be added into VifFRUC, making the detection more universal and robust. To match with VifFRUC, we have also designed a neural network, which trains a number of Long Short Term Memory (LSTM) units in parallel to learn the data-driven detection. The parallel LSTM structure of network can continually adapt to the newly added FRUC methods in VifFRUC. Extensive experiments on VifFRUC demonstrate the effectiveness of data-driven detection for FRUC, resulting in the security improvement of microcontrollers.

Parallel LSTM-CNN Network With Radar Multispectrogram for Human Activity Recognition

Owing to the ability of automatic feature extraction, deep learning is extensively applied in human activity recognition (HAR) based on radar. In this article, a recognition method based on a multispectrogram and deep-learning model is proposed. The radar echo data are transformed into spectrograms with different feature expressions by performing three time–frequency analyses, including short-time Fourier transform (STFT), reduced interference distribution with Hanning kernel (RIDHK), and smoothed pseudo-Wigner–Ville distribution (SPWVD), and then the spectrograms are sent to deep-learning model to realize HAR. The model employs three long short-term memory (LSTM) networks to learn the temporal features of the spectrograms and uses a 3-D convolutional neural network (3-DCNN) to extract the independent spatial features of a single spectrogram and the spatial correlation features between different kinds of spectrograms, named parallel LSTM-CNN network (PLCN). The PLCN is capable of effectively improving the degree of feature utilization in the spectrogram and the accuracy of human activity classification by fusing both temporal and spatial features. The final experimental results show that the average recognition accuracy of the method based on the PLCN for eight human activities reaches 94.75%, which is significantly higher than that of the three parallel LSTMs and the single 3-DCNN.

Temporal prediction of algal parameters in Three Gorges Reservoir based on highly time-resolved monitoring and long short-term memory network

Many dammed rivers throughout the world have experienced frequent harmful algal blooms (HABs) in the context of climate change and anthropogenic activities. Accurate forecasting of algal parameters (i.e., algal cell density and microcystin concentration) has great practical significance for taking precautions against HABs risks. Long short-term memory (LSTM) networks have recently shown potential in predicting water quality parameters. However, there is still little known about the robustness of the LSTM in forecasting highly time-resolved measurement of algal parameters. This study developed a hybrid deep-learning architecture (XG-LSTM) composed of one XGBoost module and two parallel LSTM models to predict algal cell density and microcystin concentration in the Three Gorges Reservoir (TGR). The proposed model was validated by in situ multi-sensor-system monitoring data at four bloom-impacted tributaries in the TGR. Each modelling process utilized the antecedent information of the algal parameters and the corresponding environmental variables as inputs for forecasting the algal parameters for the coming hours and days. As expected, the presented model achieved better performance than those without special feature extraction procedures, providing that the use of selected environmental parameters can improve LSTM performance. In addition, the hybrid XG-LSTM model successfully captured the time-series patterns of both algal cell density and microcystin concentration compared with other data-driven models, further suggesting the reliable utilization of this model in early warnings of bloom toxicity. Thus, the results presented demonstrate the potential of deep learning technology for real-time prediction of algal parameters in the TGR, and possibly for rapid detection of developing HABs in other aquatic ecosystems.

Highly accurate energy consumption forecasting model based on parallel LSTM neural networks

The main challenges of the energy consumption forecasting problem are the concerns for reliability, stability, efficiency and accuracy of the forecasting methods. The existing forecasting models suffer from the volatility of the energy consumption data. It is desired for AI models that predict irregular sudden changes and capture long-term dependencies in the data. In this study, a novel hybrid AI empowered forecasting model that combines singular spectrum analysis (SSA) and parallel long short term memory (PLSTM) neural networks is proposed. The decomposition with the SSA enhanced the performance of the PLSTM network. According to the experimental results, the proposed model outperforms the state-of-the-art models at different time intervals in terms of both prediction accuracy and computational efficiency.

Continuous Human Activity Recognition through Parallelism LSTM with Multi-Frequency Spectrograms

According to the real-living environment, radar-based human activity recognition (HAR) is dedicated to recognizing and classifying a sequence of activities rather than individual activities, thereby drawing more attention in practical applications of security surveillance, health care and human–computer interactions. This paper proposes a parallelism long short-term memory (LSTM) framework with the input of multi-frequency spectrograms to implement continuous HAR. Specifically, frequency-division short-time Fourier transformation (STFT) is performed on the data stream of continuous activities collected by a stepped-frequency continuous-wave (SFCW) radar, generating spectrograms of multiple frequencies which introduce different scattering properties and frequency resolutions. In the designed parallelism LSTM framework, multiple parallel LSTM sub-networks are trained separately to extract different temporal features from the spectrogram of each frequency and produce corresponding classification probabilities. At the decision level, the probabilities of activity classification from these sub-networks are fused by addition as the recognition output. To validate the proposed method, an experimental data set is collected by using an SFCW radar to monitor 11 participants who continuously perform six activities in sequence with three different transitions and random durations. The validation results demonstrate that the average accuracies of the designed parallelism unidirectional LSTM (Uni-LSTM) and bidirectional LSTM (Bi-LSTM) based on five frequency spectrograms are 85.41% and 96.15%, respectively, outperforming traditional Uni-LSTM and Bi-LSTM networks with only a single-frequency spectrogram by 5.35% and 6.33% at least. Additionally, the recognition accuracy of the parallelism LSTM network reveals an upward trend as the number of multi-frequency spectrograms (namely the number of LSTM subnetworks) increases, and tends to be stable when the number reaches 4.

Prediction and reconstruction of ocean wave heights based on bathymetric data using LSTM neural networks

Since climate change impacts threaten the coastal regions of the North Sea, consistent sea state time series are essential for building coastal protection or offshore structures. Vast gaps in buoy data, caused by false measurements or maintenance periods, require reconstruction and forecasting of ocean waves. Morphodynamic changes of the Ebb-Tidal Delta (ETD) sandbanks exert a huge impact on the local wave climate off the island of Norderney, Germany. Therefore, the objective of this paper is to develop a machine learning model based on Long Short-Term Memory (LSTM) neural networks for reconstruction and short- and long-term prediction of nearshore significant wave height (SWH), integrating bathymetric data for the first time. Time series of sea state and weather data of adjacent buoys as well as bathymetric data of the ETD sandbanks from 2004 to 2017 were used and the networks were tuned with Bayesian hyperparameter optimization. Including the bathymetries improved the performance of the LSTM for SWH reconstruction, short-, and long-term predictions by 16.7%, 7.4–11.7%, and 8.8–9.1% in terms of RMSE, respectively. The LSTM outperformed deep feed-forward neural networks and other state-of-the-art machine learning algorithms. With an RMSE of 0.069 m, a parallel LSTM structure (P-LSTM) is the proposed method for SWH reconstruction.

Multi-resolution modulation-filtered cochleagram feature for LSTM-based dimensional emotion recognition from speech

Continuous dimensional emotion recognition from speech helps robots or virtual agents capture the temporal dynamics of a speaker’s emotional state in natural human–robot interactions. Temporal modulation cues obtained directly from the time-domain model of auditory perception can better reflect temporal dynamics than the acoustic features usually processed in the frequency domain. Feature extraction, which can reflect temporal dynamics of emotion from temporal modulation cues, is challenging because of the complexity and diversity of the auditory perception model. A recent neuroscientific study suggests that human brains derive multi-resolution representations through temporal modulation analysis. This study investigates multi-resolution representations of an auditory perception model and proposes a novel feature called multi-resolution modulation-filtered cochleagram (MMCG) for predicting valence and arousal values of emotional primitives. The MMCG is constructed by combining four modulation-filtered cochleagrams at different resolutions to capture various temporal and contextual modulation information. In addition, to model the multi-temporal dependencies of the MMCG, we designed a parallel long short-term memory (LSTM) architecture. The results of extensive experiments on the RECOLA and SEWA datasets demonstrate that MMCG provides the best recognition performance in both datasets among all evaluated features. The results also show that the parallel LSTM can build multi-temporal dependencies from the MMCG features, and the performance on valence and arousal prediction is better than that of a plain LSTM method.

Pressure prediction and abnormal working conditions detection of water supply network based on LSTM

Abstract In this study, a deep learning model based on LSTM (Long Short-Term Memory) is used to predict the state of a water supply network due to its highly complex nonlinearity. The inputs of the model include state information on the pressures at measuring points, as well as control information on the water supply pressure and flow at each entry point. In order to enhance the performance of the model in feature extraction and identification and improve prediction accuracy, a parallel LSTM tandem DNN deep neural network model (PLDNN) is proposed. The experimental results indicate that the model has better learning performance and accuracy compared with traditional prediction methods (artificial neural networks, support vector machines, etc.) and general LSTM models.

Video captioning using boosted and parallel Long Short-Term Memory networks

Video captioning and its integration with deep learning is one of the most challenging issues in the field of machine vision and artificial intelligence. In this paper, a new boosted and parallel architecture is proposed for video captioning using Long Short-Term Memory (LSTM) networks. The proposed architecture comprises two LSTM layers and a word selection module. The first LSTM layer has the responsibility of encoding frame features extracted by a pre-trained deep Convolutional Neural Network (CNN). In the second LSTM layer, a novel architecture is used for video captioning by leveraging several decoding LSTMs in a parallel and boosting architecture. This layer, which is called Boosted and Parallel LSTM (BP-LSTM) layer, is constructed by iteratively training LSTM networks using a special kind of AdaBoost algorithm during the training phase. During the testing phase, the outputs of BP-LSTMs are concurrently combined using the maximum probability criterion and word selection module. We tested the proposed algorithm with two well-known video captioning datasets and compared the results with state-of-the-art algorithms. The results show that the proposed architecture considerably improves the accuracy of the generated sentence.