Convolutional Long Short-term Memory Network Research Articles

A multi-flexible video summarization scheme using property-constraint decision tree

Video summarization (VS) technology presents a multi-tendency development. However, there are still certain challenges in shortening a long video into several diverse and concise versions, such as generating various versions and variable lengths of summaries depending on user requirements. In this context, it is critical to extract the spatial–temporal features of video frames. To address these issues, this paper proposes a multi-flexible video summarization scheme using a property-constraint decision.Due to the difficulty of directly transforming user requirements into summary algorithm factors, this paper employs property-constraints as a bridge between user requirements and the video summary algorithm. The specific implementation method is to aggregate and analyze existing research results on summary-oriented properties. It begins with the establishment of a property-constraints library, followed by the translating and mapping of user requirements into VS property-constraints, and ultimately constructing a property-constraints tree for flexible decision versions of VS. In addition, a hybrid cascade bidirectional network (CB-ConvLSTM) based on the Convolutional LSTM Network (ConvLSTM) is designed to extract the spatial–temporal features of the video. On this basis, the VS generator is configured. The goal of this scheme, which combines the VS property constraints and CB-ConvLSTM, is to “analyze once, satisfy multiple factors, generate multiple levels”. Verification experiments and a comparative analysis are conducted on benchmark datasets to evaluate the algorithm in this paper. The results indicate that the proposed algorithm is highly rational, effective, and usable.

Short-Term Demand Forecast of E-Commerce Platform Based on ConvLSTM Network.

Based on real sales data, this article constructed LGBM and LSTM sales prediction models to compare and verify the performance of the proposed models. In this article, we forecast the product sales of stores in the future T + 3 days and use MAPE as the evaluation index. The experiment shows that the product sales prediction model based on the convolutional LSTM (ConvLSTM) network has better prediction accuracy. From a store point of view, ConvLSTM prediction model MAPE was 0.42 lower than the long short-term memory (LSTM) network and 0.68 lower than LGBM. From the perspective of commodity categories, different commodity categories are suitable for different forecasting methods. Some categories are suitable for regression forecasting, while others are suitable for time-series forecasting. Among the categories suitable for time-series forecasting, the ConvLSTM model performs the best.

Annual dilated convolutional LSTM network for time charter rate forecasting

Time charter rates must be predicted accurately to assist sensible decisions in the global, highly volatile shipping market. Time charter rates are affected by multiple factors, such as second-hand ship prices, order book, Libor interest rate, etc. However, not all factors convey predictive features to anticipate the future of time charter rates. Therefore, extracting predictive features from multiple driving time series from the shipping market is crucial for forecasting purposes. Accordingly, this paper proposes a novel convolutional recurrent neural network for time charter rates forecasting under the multi-variate phenomenon. The proposed network first extracts features from the monthly time series using a novel convolutional filter, the annual dilated filter. The annual dilated convolutional filter can extract the predictive features effectively and impose a sparse input structure to prevent overfitting. Then, a recurrent neural network learns the temporal information from the convoluted features. An extensive comparison study with many baseline models, including the persistence (Naïve I), statistical models, and the state-of-art networks, is conducted on the time charter rates of six kinds of ships. The empirical results demonstrate the proposed model’s superiority in forecasting the time charter rates.

Video Anomaly Detection Based on Convolutional Recurrent AutoEncoder.

As an essential task in computer vision, video anomaly detection technology is used in video surveillance, scene understanding, road traffic analysis and other fields. However, the definition of anomaly, scene change and complex background present great challenges for video anomaly detection tasks. The insight that motivates this study is that the reconstruction error for normal samples would be lower since they are closer to the training data, while the anomalies could not be reconstructed well. In this paper, we proposed a Convolutional Recurrent AutoEncoder (CR-AE), which combines an attention-based Convolutional Long–Short-Term Memory (ConvLSTM) network and a Convolutional AutoEncoder. The ConvLSTM network and the Convolutional AutoEncoder could capture the irregularity of the temporal pattern and spatial irregularity, respectively. The attention mechanism was used to obtain the current output characteristics from the hidden state of each Covn-LSTM layer. Then, a convolutional decoder was utilized to reconstruct the input video clip and the testing video clip with higher reconstruction error, which were further judged to be anomalies. The proposed method was tested on two popular benchmarks (UCSD ped2 Dataset and Avenue Dataset), and the experimental results demonstrated that CR-AE achieved 95.6% and 73.1% frame-level AUC on two public datasets, respectively.

EMDLP: Ensemble multiscale deep learning model for RNA methylation site prediction.

BackgroundRecent research recommends that epi-transcriptome regulation through post-transcriptional RNA modifications is essential for all sorts of RNA. Exact identification of RNA modification is vital for understanding their purposes and regulatory mechanisms. However, traditional experimental methods of identifying RNA modification sites are relatively complicated, time-consuming, and laborious.Machine learning approaches have been applied in the procedures of RNA sequence features extraction and classification in a computational way, which may supplement experimental approaches more efficiently. Recently, convolutional neural network (CNN) and long short-term memory (LSTM) have been demonstrated achievements in modification site prediction on account of their powerful functions in representation learning. However, CNN can learn the local response from the spatial data but cannot learn sequential correlations. And LSTM is specialized for sequential modeling and can access both the contextual representation but lacks spatial data extraction compared with CNN. There is strong motivation to construct a prediction framework using natural language processing (NLP), deep learning (DL) for these reasons.ResultsThis study presents an ensemble multiscale deep learning predictor (EMDLP) to identify RNA methylation sites in an NLP and DL way. It organically combines the dilated convolution and Bidirectional LSTM (BiLSTM), which helps to take better advantage of the local and global information for site prediction.The first step of EMDLP is to represent the RNA sequences in an NLP way. Thus, three encodings, e.g., RNA word embedding, One-hot encoding, and RGloVe, which is an improved learning method of word vector representation based on GloVe, are adopted to decipher sites from the viewpoints of the local and global information. Then, a dilated convolutional Bidirectional LSTM network (DCB) model is constructed with the dilated convolutional neural network (DCNN) followed by BiLSTM to extract potential contributing features for methylation site prediction. Finally, these three encoding methods are integrated by a soft vote to obtain better predictive performance. Experiment results on m1A and m6A reveal that the area under the receiver operating characteristic(AUROC) of EMDLP obtains respectively 95.56%, 85.24%, and outperforms the state-of-the-art models. To maximize user convenience, a user-friendly webserver for EMDLP was publicly available at http://www.labiip.net/EMDLP/index.php (http://47.104.130.81/EMDLP/index.php).ConclusionsWe developed a predictor for m1A and m6A methylation sites.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12859-022-04756-1.

Comparison of Nitrogen Dioxide Predictions During a Pandemic and Non-pandemic Scenario in the City of Madrid using a Convolutional LSTM Network

Traditionally, machine learning technologies with the methods and capabilities available, combined with a geospatial dimension, can perform predictive analyzes of air quality with greater accuracy. However, air pollution is influenced by many external factors, one of which has recently been caused by the restrictions applied to curb the relentless advance of COVID-19. These sudden changes in air quality levels can negatively influence current forecasting models. This work compares air pollution forecasts during a pandemic and non-pandemic period under the same conditions. The ConvLSTM algorithm was applied to predict the concentration of nitrogen dioxide using data from the air quality and meteorological stations in Madrid. The proposed model was applied for two scenarios: pandemic (January–June 2020) and non-pandemic (January–June 2019), each with sub-scenarios based on time granularity (1-h, 12-h, 24-h and 48-h) and combination of features. The Root Mean Square Error was taken as the estimation metric, and the results showed that the proposed method outperformed a reference model, and the feature selection technique significantly improved the overall accuracy.

Using MLSTM and Multioutput Convolutional LSTM Algorithms for Detecting Anomalous Patterns in Streamed Data of Unmanned Aerial Vehicles

In this article, we present a comparative study of two existing deep learning tools that are used in a novel way to detect anomalies in the streamed data of the unmanned aerial vehicle (UAV). Detecting anomalies is very vital to predict potential faults that are caused by hardware and software faults and may prevent the UAV from hazardous accidents. Therefore, we suggest using multiple long short-term memory and multioutput convolutional long short-term memory (LSTM) to detect anomalies in UAV data. LSTM networks attracted many researchers in several domains, as it is a useful tool for learning dynamic temporal patterns and long-range dependencies in sequential data, which cannot be achieved using traditional neural networks. However, utilizing multiple LSTM networks would result in too much redundancy. The redundancy issue can be solved by incorporating one convolutional LSTM (ConvLSTM) network with multiple outputs. The ConvLSTM is suitable for analyzing multivariate temporal data; due to its convolutional architecture and the advantage of preserving the benefits of the LSTM networks. We evaluated and compared the two approaches using well-known indicators such as the detection rate, the false alarm rate, the precision, and the F.score indicators. The two methods exhibited promising results in predicting different types of faults, for instance (sensor-impulse and sensor-cut). However, the multioutput ConvLSTM was faster in training and testing phases, and its results were superior in predicting (sensor-stuck and sensor-drift) faults.

Employing the Convolutional LSTM Network in Crop Classification using NDVI Time Series

تغییرات رشد محصولات کشاورزی در فواصل زمانی نسبتاً کوتاه، ناهم‌زمانی کشت محصولات مشابه، شباهت طیفی میان محصولات گوناگون در برخی از زمان‌های دورة کشت و کمبود داده‌های زمینی، طبقه‌بندی محصولات زراعی در تصاویر ماهواره‌ای را به کاری چالش‌برانگیز مبدل می‌کند. تغییر مقدار درصد پوشش و سبزینگی، در طول فصل رشد، از برجسته‌ترین ویژگی پوشش‏‌های گیاهی ازجمله محصولات کشاورزی است که می‌توان بر آن نظارت کرد. این کار با استفاده از سری زمانی شاخص‌های گیاهی انجام می‌شود که اطلاعات بسیار مفیدی از توالی ویژگی‌های فنولوژیک محصولات کشاورزی در اختیار ما قرار می‌دهد. استفاده از روش‌های یادگیری عمیق با توانایی یادگیری اطلاعات متوالی حاصل از این سری‌ها می‌تواند، در طبقه‌بندی محصولات کشاورزی و کاهش وابستگی به داده‌های زمینی، مفید باشد. شبکةLong-Short Term Memory (LSTM) یکی از انواع شبکه‌های عصبی بازگشتی در تجزیه و تحلیل داده‌های متوالی است که توانایی یادگیری توالی‌های بلندمدت در سری زمانی را دارد؛ بنابراین در این مطالعه، پس از محاسبة شاخص NDVI از باندهای ماهوارة سنتینل‌ـ 2 در نُه تاریخ متفاوت و تشکیل سری زمانی آن شاخص برای ورود به شبکه، دو ناحیة متفاوت در دشت مغان در نظر گرفته شد که محصولات کشت‌شده در آنها، طی عملیات زمینی، برداشت شده بود. در ناحیة اول، شبکة کانولوشنی LSTM برای طبقه‌بندی محصولات آموزش دید و در ناحیة دیگر، کارآیی این شبکة آموزش‌دیده در طبقه‌بندی محصولات ارزیابی شد و دقت کلی 82% و ضریب کاپای 8/0 به‌دست آمد. افزایش تعداد نمونه‌های زمینی و انتخاب مرز دقیق محصولات، می‌تواند کارایی روش مورد استفاده را افزایش دهد.

Prediction of Node Importance of Power System Based on ConvLSTM

In power systems, the destruction of some important nodes may cause cascading faults. If the most important node in the power system can be found, the important node can be protected in advance, thereby avoiding a blackout accident. At present, the evaluation algorithm of node importance is calculated based on the power flow of the power grid, so the calculation results must be lagging behind, and it does not have the ability to provide early warning for the power grid to provide protection signals. Therefore, it is necessary to predict the importance of nodes in the power system. After using a reasonable prediction model to predict the importance of nodes, we can simulate the future state of power system operation and avoid accidents for the dispatching agency of the power grid company according to the prediction results. This paper proposes a prediction model based on convolutional long short-term memory (ConvLSTM) to predict the importance of nodes. This method has obvious advantages over the long short-term memory (LSTM) network. The convolution operation is used to replace the original full connection operation of the LSTM network, which not only utilizes the advantages of convolution to extract spatial features but also retains the ability of LSTM to process time-series features. The simulation results show that the prediction of node importance using the ConvLSTM network is much more accurate than LSTM. Using statistical indicators to compare and analyze the prediction results, it can be seen that ConvLSTM has higher prediction accuracy. Therefore, using the ConvLSTM model to predict node importance has certain significance for grid dispatching agencies to accurately simulate the future state of the power system and avoid risks in advance.

Skeleton-based human action recognition with sequential convolutional-LSTM networks and fusion strategies

Skeleton-based human action recognition with sequential convolutional-LSTM networks and fusion strategies

Calcium scoring in low-dose ungated chest CT scans using convolutional long-short term memory networks.

We aimed to develop a novel deep-learning based method for automatic coronary artery calcium (CAC) quantification in low-dose ungated computed tomography attenuation correction maps (CTAC). In this study, we used convolutional long-short -term memory deep neural network (conv-LSTM) to automatically derive coronary artery calcium score (CAC) from both standard CAC scans and low-dose ungated scans (CT-attenuation correction maps). We trained convLSTM to segment CAC using 9543 scans. A U-Net model was trained as a reference method. Both models were validated in the OrCaCs dataset (n=32) and in the held-out cohort (n=507) without prior coronary interventions who had CTAC standard CAC scan acquired contemporarily. Cohen's kappa coefficients and concordance matrices were used to assess agreement in four CAC score categories (very low: <10, low:10-100; moderate:101-400 and high >400). The median time to derive results on a central processing unit (CPU) was significantly shorter for the conv-LSTM model- 6.18s (inter quartile range [IQR]: 5.99, 6.3) than for UNet (10.1s, IQR: 9.82, 15.9s, p<0.0001). The memory consumption during training was much lower for our model (13.11Gb) in comparison with UNet (22.31 Gb). Conv-LSTM performed comparably to UNet in terms of agreement with expert annotations, but with significantly shorter inference times and lower memory consumption.

A deep learning framework with convolutional long short‐term memory for influenza‐like illness trend estimation

AbstractForecasting and modeling real‐world time series problems have been challenging for researchers due to nonlinear and complicated patterns. With the recent developments in deep learning, more accurate and efficient forecasting can be provided by learning the relationships between input and output data. This article has introduced a novel hybrid approach, called the GA‐ConvLSTM‐CNN, based on the genetic algorithm (GA), convolutional long short‐term memory (ConvLSTM) and convolutional neural network (CNN) to capture complex and sophisticated features from historical data. The GA‐ConvLSTM‐CNN approach is constructed with two‐stacked ConvLSTM networks, two‐stacked CNN, and a fully connected layer to alleviate the complexity of the real‐world time series problems with multivariate and multistep. In order to increase the accuracy of the GA‐ConvLSTM‐CNN, the genetic algorithm is adopted to simultaneously optimize the network's hyperparameters. With the aim of making comparisons, the weekly influenza‐like illness (ILI) and the weather data have been handled. The GA‐ConvLSTM‐CNN framework compared different techniques comprising machine learning and deep learning approaches. The results have indicated that the GA‐ConvLSTM‐CNN has the ability to accomplish more effective prediction accuracy and provide remarkable support for ILI outbreaks than the benchmark techniques.

ConvLSTM Network-Based Rainfall Nowcasting Method with Combined Reflectance and Radar-Retrieved Wind Field as Inputs

Strong convection nowcasting has been gaining importance in regional security, economic development, and water resource management. Rainfall nowcasting with strong timeliness needs to effectively forecast the intensity of rainfall in a local region in the short term. The forecast performance of traditional methods is limited. In this paper, a rainfall nowcasting model based on the Convolutional Long Short-Term Memory (ConvLSTM) is proposed. Combined reflectance (CR) and the retrieved wind field are selected as the key factors for prediction. The model considers the influence of water vapor content, transport, and change on rainfall by measuring CR and the retrieved wind field. Forecast results are compared to different models and different input data schemes. The analysis shows that the CSI scores of this method can reach about 0.8, which is 28% higher than that of the optical flow method. The ConvLSTM structure with spatial analysis and time memory can greatly enhance the predictive ability of the model, and the addition of wind field data also improves the forecasting performance of the model. Therefore, the idea of forecasting rainfall on the basis of water vapor content and water vapor transport is practical and effective, and the accuracy of a forecast can be improved by using ConvLSTM network to extract spatiotemporal features.

Deep learning for predicting respiratory rate from biosignals

In the past decade, deep learning models have been applied to bio-sensors used in a body sensor network for prediction. Given recent innovations in this field, the prediction accuracy of novel models needs to be evaluated for bio-signals. In this paper, we evaluate the performance of deep learning models for respiratory rate prediction. We consider three datasets from bio-sensors which include electrocardiogram (ECG), photoplethysmogram (PPG) data, and surface electromyogram (sEMG) data. The deep learning models include Long short-term memory (LSTM) networks, Bidirectional LSTM (Bi-LSTM), attention-based variants of LSTM, CNN-LSTM and Convolutional-LSTM networks. The deep learning models are evaluated for two separate windows which are 32 s and 64 s window. The models’ performance is evaluated using mean absolute error (MAE). The 64 s window has more accurate prediction compared to the 32 s window. Our results indicate Bi-LSTM with Bahdanu Attention has the best performance for the bio-signals. LSTM performs best with one of the datasets, yielding an MAE of 0.70 ± 0.02. Bi-LSTM with Bahdanau attention showed best results with two of the three datasets with MAE of 0.51 ± 0.03 for sEMG based data and MAE of 0.24 ± 0.03 with PPG and ECG based data.

Convolutional LSTM: a deep learning approach to predict shoulder joint reaction forces

We developed a Convolutional LSTM (ConvLSTM) network to predict shoulder joint reaction forces using 3D shoulder kinematics data containing 30 different shoulder activities from eight human subjects. We considered simulation outcomes from the AnyBody musculoskeletal model as the baseline force dataset to validate ConvLSTM model predictions. Results showed a good correlation (>80% accuracy, r ≥ 0.82) between ConvLSTM predicted and AnyBody estimated force values, the generalization of the developed model for novel task type (p-value = 0.07 ∼ 0.33), and a better prediction accuracy for the ConvLSTM model than conventional CNN and LSTM models.

Hybrid Deep Learning Approach for Traffic Speed Prediction.

Traffic speed prediction plays a fundamental role in traffic management and driving route planning. However, timely accurate traffic speed prediction is challenging as it is affected by complex spatial and temporal correlations. Most existing works cannot simultaneously model spatial and temporal correlations in traffic data, resulting in unsatisfactory prediction performance. In this article, we propose a novel hybrid deep learning approach, named HDL4TSP, to predict traffic speed in each region of a city, which consists of an input layer, a spatial layer, a temporal layer, a fusion layer, and an output layer. Specifically, first, the spatial layer employs graph convolutional networks to capture spatial near dependencies and spatial distant dependencies in the spatial dimension. Second, the temporal layer employs convolutional long short-term memory (ConvLSTM) networks to model closeness, daily periodicity, and weekly periodicity in the temporal dimension. Third, the fusion layer designs a fusion component to merge the outputs of ConvLSTM networks. Finally, we conduct extensive experiments and experimental results to show that HDL4TSP outperforms four baselines on two real-world data sets.

Ovarian cancer detection using optical coherence tomography and convolutional neural networks.

Ovarian cancer has the sixth-largest fatality rate in the United States among all cancers. A non-surgical assay capable of detecting ovarian cancer with acceptable sensitivity and specificity has yet to be developed. However, such a discovery would profoundly impact the pace of the treatment and improvement to patients’ quality of life. Achieving such a solution requires high-quality imaging, image processing, and machine learning to support an acceptably robust automated diagnosis. In this work, we propose an automated framework that learns to identify ovarian cancer in transgenic mice from optical coherence tomography (OCT) recordings. Classification is accomplished using a neural network that perceives spatially ordered sequences of tomograms. We present three neural network-based approaches, namely a VGG-supported feed-forward network, a 3D convolutional neural network, and a convolutional LSTM (Long Short-Term Memory) network. Our experimental results show that our models achieve a favorable performance with no manual tuning or feature crafting, despite the challenging noise inherent in OCT images. Specifically, our best performing model, the convolutional LSTM-based neural network, achieves a mean AUC (± standard error) of 0.81 ± 0.037. To the best of the authors’ knowledge, no application of machine learning to analyze depth-resolved OCT images of whole ovaries has been documented in the literature. A significant broader impact of this research is the potential transferability of the proposed diagnostic system from transgenic mice to human organs, which would enable medical intervention from early detection of an extremely deadly affliction.

D2TNet: A ConvLSTM Network With Dual-Direction Transfer for Pan-Sharpening

In this article, we propose an efficient convolutional long short-term memory (ConvLSTM) network with dual-direction transfer for pan-sharpening, termed D2TNet. We design a specially structured ConvLSTM network that allows for dual-directional communication, including multiscale information and multilevel information. On the one hand, due to the sensitivity of spatial information to scales and the sensitivity of spectral information to levels, multiscale and multilevel information is extracted to facilitate the fuller use of source images. On the other hand, ConvLSTM is employed to capture the strong dependencies between multiscale information and multilevel information. Besides, we introduce a multiscale loss to enable different scales contributing to each other to generate high-resolution multispectral images that are closer to the ground truth. Extensive experiments, including qualitative evaluation, quantitative evaluation, and efficiency comparison, are implemented to verify that our D2TNet outperforms state-of-the-art methods indeed.

Modulation Signal Classification Algorithm Based on Denoising Residual Convolutional Neural Network

Traditional denoising algorithms are easy to lose signal details, resulting in low recognition accuracy of modulated signals. A algorithm for modulation signal classification based on denoising residual Convolutional Neural Network (DRCNet) is proposed. DRCNet inserts a soft threshold function as a nonlinear transformation layer into the deep architecture to build a soft threshold learning network (STLNet). STLNet obtains an appropriate threshold according to the signal-to-noise ratio of the input signal samples, which is used to remove useless noise features, thereby effectively denoising the signal.This paper builds a serial network (RCTLNet) connected by Convolutional Neural Networks and Long-Short-Term Memory Networks. RCTLNet uses the convolutional neural network to extract the spatial features of the signal samples, and uses the long and short-term memory network to extract the temporal features of the signal samples. And then realize the classification and identification of the modulated signal. Experiments show that the recognition accuracy of the proposed modulated signal classification model is 92%, which is 7% higher than that of the Convolutional Long-Short Term Memory Network (CNN-LSTM).

Imaginary Control of a Mobile Vehicle Using Deep Learning Algorithm: A Brain Computer Interface Study

Controlling a remote mobile vehicle using electroencephalograph (EEG) signals is still a challenge specially achieving a high degree of accuracy and precision. In the present study, the focus is on implementing an efficient feature space in a deep-based learning (DL) algorithm for a single trial application. More specifically, a boosting feature algorithm by means of long short-term memory (LSTM) networks are implemented in a deep auto-encoder (DAE) algorithm for producing an effective feature space tp identify event related desynchronization/event related synchronization (ERD/ERS) patters in EEG signals. For this purpose, three different DL-based algorithms are implemented that the models are based on a convolutional neural network (CNN), DAE, and LSTM networks to extract and boost the main features. In addition, our previous improved support vector machine (SVM)-based algorithm is employed to consider the potential of SVM and implemented DL-based algorithms for a two classes identification. To consider the efficiency of our implemented methods, algorithms are employed for control of a remote mobile vehicle in an imaginary right-hand opening and making a right-hand fist task. In our experiment, eleven subjects participated in an imaginary movement task. In the experiment, the displayed movement pictures were colored in yellow and red colors for stimulating brain to generate stronger ERD/ERS patterns. Results showed that the proposed algorithm by using the boosting technique significantly increased the accuracy with a higher precision of 73.31&#x0025; &#x00B1; 0.03. The proposed method enabled the DL algorithm to be used in single trial experiments.