Convolutional Long Short-term Memory Research Articles

Memory-Net: Coupling feature maps extraction and hierarchical feature maps reuse for efficient and effective PET/CT multi-modality image-based tumor segmentation

Dense connection (DC), which densely reuses shallow feature maps to boost feature propagation, has been widely used in medical image segmentation models. However, feature aggregation of DC can be redundant, resulting in an expensive computational cost for resource-constrained platforms. To address this problem, we proposed a gated convolutional unit (GCU)-based feature forwarding method. GCU is a modified convolutional long short-term memory (Conv-LSTM) unit. First, we present a gated convolutional module (GCM) by hierarchically stacking GCUs and combining it with UNet architecture to create Memory-Net, wherein the hidden states of GCUs serve as the feature maps and the cell memories throughout the GCM form an information highway, enabling the model to efficiently and selectively aggregate useful information to boost feature propagation. We further combine the GCU with hyperdense connection (HDC) to propose a hyper-gated convolutional unit (HGCU) and develop a novel GCU and HGCU based multi-branch encoder (GCU-HGCU-Encoder), wherein the cell memory of the encoding branch for a specific input modality is not only used within the branch but also across all encoding branches, resulting in more efficient and effective multi-modality information fusion for Memory-Net. For improved segmentation, we introduced a recurrent-dense-Siamese decoder (RDS-Decoder) to create Memory-HDRDS-UNet, which is the final proposal of Memory-Net, by combining a GCU-HGCU-Encoder, a simple RDS-Decoder, and skip layers. We validated the superiority of Memory-Net on PET/CT volumes with lymphomas, and it achieved an average Dice score of 0.8690 and an average sensitivity of 0.9616, outperforming the state-of-the-art methods with a much lower computational cost.

Deep-Learning-Based Scenario Recognition With GNSS Measurements on Smartphones

Smartphones are in everyone’s hands for applications including navigation- and localization-based services, and scenario recognition is critical for seamless indoor and outdoor navigation. How to use smartphone sensing data to recognize different scenarios is a meaningful but challenging problem. To address this issue, we propose a structured grid-based deep-learning scenario recognition technique that uses smartphone global navigation satellite system (GNSS) measurements (satellite position, pseudorange, Doppler shift, and C/N0). In this work, the scenarios are grouped into four categories: deep indoors, shallow indoors, semioutdoors, and open outdoors. The proposed approach utilizes Voronoi tessellations to obtain structured-grid representations from satellite positions and performs computations using convolutional neural networks (CNNs) and convolutional long short-term memory (ConvLSTM) networks. With only spatial information being considered, the CNN model is used to extract the features of Voronoi tessellations for scenario recognition, achieving a high accuracy of 98.82%. Then, to enhance the robustness of the algorithm, the ConvLSTM network is adopted, which treats the measurements as spatiotemporal sequences, improving the accuracy to 99.92%. Compared with existing methods, the proposed algorithm is simple and efficient, using only GNSS measurements without the need for additional sensors. Furthermore, the latencies of the CNN and ConvLSTM models on a CPU are only 16.82 and 27.94 ms, respectively. Therefore, the proposed algorithm has the potential for real-time applications.

Learning topology optimization process via convolutional long‐short‐term memory autoencoder‐decoder

AbstractThis article proposed an autoencoder‐decoder architecture with convolutional long‐short‐term memory (ConvLSTM) cell for the purpose of learning topology optimization iterations. The overall topology optimization process is treated as time‐series data, with each iteration as a single step. The first few steps are fed into the encoder to generate encoder embedding, which is fed into the decoder. The decoder uses the encoder embedding as input and generates the result at each future step until the end of iteration. To train the proposed neural network, a large dataset is generated by a conventional topology optimization method, that is, solid isotropic material with penalization for intermediate densities, with randomly picked boundary conditions, load conditions, and volume constraints. Unlike other deep learning models introduced before, the proposed method can learn each topology optimization step iteratively and present the full optimization path. Furthermore, the proposed method can be extended to give solutions to unseen boundary and load conditions with a significant reduction in computation cost in a little sacrifice on the performance of the optimum design.

Distance adaptive graph convolutional gated network-based smart air quality monitoring and health risk prediction in sensor-devoid urban areas

Robust air quality early warning systems for sustainable development of cities have garnered significant attention. These systems enable health risk warnings and regulatory plans when harmful pollutant levels are forecasted. However, with current frameworks, forecasting is only possible at locations with sufficient sensor data and cannot be predicted at new areas. Furthermore, long-term sensor failure also limits their real-time monitoring and forecasting applications. We propose a distance adaptive graph convolutional gated network that provides simultaneous forecasts of primary air pollutants at multiple temporal horizons and locations of a mega-city via spatio-temporal sensor fusion. The framework also solves critical problems of early warning systems related to long-term sensor failure and prediction at a new location of city. The results suggest that for 12 h ahead PM2.5 forecast, the proposed model reduces prediction errors by 43.89% and 52.59% compared to the timeseries-Transformer and convolutional long-short term memory network, respectively. For long-term sensor failure imputation, the mean absolute error for CO, NO2, O3, PM10, and PM2.5 ranged between 0.081–0.160, 0.004–0.007, 0.005–0.0155, 9.41–13.5, and 5.75–8.12, respectively. Whereas the remotely forecasted concentrations at sensor-less locations showed a close similarity to the actual air quality distribution in the city area.

Effect of hyper-parameters on the performance of ConvLSTM based deep neural network in crop classification.

Deep learning based data driven methods with multi-sensors spectro-temporal data are widely used for pattern identification and land-cover classification in remote sensing domain. However, adjusting the right tuning for the deep learning models is extremely important as different parameter setting can alter the performance of the model. In our research work, we have evaluated the performance of Convolutional Long Short-Term Memory (ConvLSTM) and deep learning techniques, over various hyper-parameters setting for an imbalanced dataset and the one with highest performance is utilized for land-cover classification. The parameters that are considered for experimentation are; Batch size, Number of Layers in ConvLSTM model, and No of filters in each layer of the ConvLSTM are the parameters that will be considered for our experimentation. Experiments also have been conducted on LSTM model for comparison using the same hyper-parameters. It has been found that the two layered ConvLSTM model having 16-filters and a batch size of 128 outperforms other setting scenarios, with an overall validation accuracy of 97.71%. The accuracy achieved for the LSTM is 93.9% for training and 92.7% for testing.

Predicting Household Electric Power Consumption Using Multi-step Time Series with Convolutional LSTM

Predicting Household Electric Power Consumption Using Multi-step Time Series with Convolutional LSTM

A novel hybrid model for lithium-ion batteries lifespan prediction with high accuracy and interpretability

Machine learning can accurately predict the remaining useful life (RUL) of lithium-ion batteries because of its strong learning ability, efficient computing efficiency, and high accuracy. However, the prediction behavior and principle of many data-driven models as black box functions are unknown, and the potential of high accurate prediction requires further investigation. In view of these research gaps, this study proposes a novel hybrid model based on adaptive feature separable convolution (AFSC) and convolutional long short-term memory (ConvLSTM) network to improve the accuracy of RUL prediction and the interpretability of the model. The model extracts aging features from charging process data and can be applied to both early prediction and RUL prediction. Validation based on 124 commercial lithium iron phosphate battery aging data shows that the mean absolute error (MAE) of the early prediction results using the first 20 cycles is only 7 cycles, while the MAE of the RUL prediction is 0.12 cycles, both demonstrating excellent performance. In addition, the feature processing and prediction process of the model is analyzed through the visualization of upsampling and attention weights.

Construction of an Integrated Drought Monitoring Model Based on Deep Learning Algorithms

Drought is one of the major global natural disasters, and appropriate monitoring systems are essential to reveal drought trends. In this regard, deep learning is a very promising approach for characterizing the non-linear nature of drought factors. We used multi-source remote sensing data such as the Moderate Resolution Imaging Spectroradiometer (MODIS) and Climate Hazards Group Infrared Precipitation with Station (CHIRPS) data to integrate drought impact factors such as precipitation, vegetation, temperature, and soil moisture. The application of convolutional long short-term memory (ConvLSTM) to construct an integrated drought monitoring model was proposed and tested, using the Xinjiang Uygur Autonomous Region as an example. To better compare the monitoring performance of ConvLSTM models, three other classical deep learning models and three classical machine learning models were also used for comparison. The results show that the composite drought index (CDI) output by the ConvLSTM model had a consistent high correlation with the drought rating of the multi-scale standardized precipitation evapotranspiration index (SPEI). The correlation coefficients between the CDI and the multi-scale standardized precipitation index (SPI) were all above 0.5 (p < 0.01), which was highly significant, and the correlation coefficient between CDI-1 and the monthly soil relative humidity at a 10 cm depth was above 0.45 (p < 0.01), which was well correlated. In addition, the spatial distribution of the CDI-6 simulated by the model was highly correlated with the degree of drought expressed by the SPEI-6 observations at the stations. This study provides a new approach for integrated regional drought monitoring.

Multi-level feature fusion for multimodal human activity recognition in Internet of Healthcare Things

Human Activity Recognition (HAR) has become a crucial element for smart healthcare applications due to the fast adoption of wearable sensors and mobile technologies. Most of the existing human activity recognition frameworks deal with a single modality of data that degrades the reliability and recognition accuracy of the system for heterogeneous data sources. In this article, we propose a multi-level feature fusion technique for multimodal human activity recognition using multi-head Convolutional Neural Network (CNN) with Convolution Block Attention Module (CBAM) to process the visual data and Convolutional Long Short Term Memory (ConvLSTM) for dealing with the time-sensitive multi-source sensor information. The architecture is developed to be able to analyze and retrieve channel and spatial dimension features through the use of three branches of CNN along with CBAM for visual information. The ConvLSTM network is designed to capture temporal features from the multiple sensors’ time-series data for efficient activity recognition. An open-access multimodal HAR dataset named UP-Fall detection dataset is utilized in experiments and evaluations to measure the performance of the developed fusion architecture. Finally, we deployed an Internet of Things (IoT) system to test the proposed fusion network in real-world smart healthcare application scenarios. The findings from the experimental results reveal that the developed multimodal HAR framework surpasses the existing state-of-the-art methods in terms of multiple performance metrics.

Forecasting Short-Term Passenger Flow of Subway Stations Based on the Temporal Pattern Attention Mechanism and the Long Short-Term Memory Network

Rational use of urban underground space (UUS) and public transportation transfer underground can solve urban traffic problems. Accurate short-term prediction of passenger flow can ensure the efficient, safe, and comfortable operation of subway stations. However, complex and nonlinear interdependencies between time steps and time series complicate such predictions. This study considered temporal patterns across multiple time steps and selected relevant information on short-term passenger flow for prediction. A hybrid model based on the temporal pattern attention (TPA) mechanism and the long short-term memory (LSTM) network was developed (i.e., TPA-LSTM) for predicting the future number of passengers in subway stations. The TPA mechanism focuses on the hidden layer output values of different time steps in history and of the current time as well as correlates these output values to improve the accuracy of the model. The card swiping data from the Hangzhou Metro automatic fare collection system in China were used for verification and analysis. This model was compared with a convolutional neural network (CNN), LSTM, and CNN-LSTM. The results showed that the TPA-LSTM outperformed the other models with good applicability and accuracy. This study provides a theoretical basis for the pre-allocation of subway resources to avoid subway station crowding and stampede accidents.

Deep spatial-temporal graph modeling for efficient NDVI forecasting

Spatio-temporal graph modelling is a new, prominent predictive tool to use on datasets with complex spatial and temporal relationships. Normalized Difference Vegetation Index (NDVI) is a remote measure offering these complex relationships, used by agricultural producers and researchers due to its strong correlation with crop growth. Accurate periodic field-level NDVI forecasting helps project crop yield, crucial for planning agricultural production. This NDVI forecasting problem was previously studied, with best results obtained by Convolutional Long Short-Term Memory (ConvLSTM) architecture. We modify the ConvLSTM architecture, improving over the original paper. Additionally, we propose a new architecture based on Graph WaveNet (GWNN). GWNN captures spatial relationships in the non-tabular data with an adaptive dependency matrix and long-range temporal relationships with stacked spatial-temporal layers. We test each model (original ConvLSTM, new ConvLSTM, and GWNN) over the same geographical points. Under Root Mean Square Error metric, GWNN outperforms original ConvLSTM by 31% and our new one by 15%. Moreover, the GWNN is more than 170 times faster at training. We compare these models on other NDVI datasets, up to 50 times larger than the original set. The consistent results show the GWNN is most efficient in both quality and runtime for the NDVI forecasting problem.

Grey Wolf Optimization Based Breast Cancer Detection using 1D Convolution LSTM Classifier

Wydawnictwo SIGMA-NOT wydaje czasopisma fachowe informujące swoich czytelników o najnowszych osiągnięciach naukowych i nowoczesnych rozwiązaniach technicznych w Polsce i na świecie, popularyzuje problemy techniczne oraz poszerza wiedzę i kulturę techniczną.

Novel Deep Learning Network for Gait Recognition Using Multimodal Inertial Sensors.

Some recent studies use a convolutional neural network (CNN) or long short-term memory (LSTM) to extract gait features, but the methods based on the CNN and LSTM have a high loss rate of time-series and spatial information, respectively. Since gait has obvious time-series characteristics, while CNN only collects waveform characteristics, and only uses CNN for gait recognition, this leads to a certain lack of time-series characteristics. LSTM can collect time-series characteristics, but LSTM results in performance degradation when processing long sequences. However, using CNN can compress the length of feature vectors. In this paper, a sequential convolution LSTM network for gait recognition using multimodal wearable inertial sensors is proposed, which is called SConvLSTM. Based on 1D-CNN and a bidirectional LSTM network, the method can automatically extract features from the raw acceleration and gyroscope signals without a manual feature design. 1D-CNN is first used to extract the high-dimensional features of the inertial sensor signals. While retaining the time-series features of the data, the dimension of the features is expanded, and the length of the feature vectors is compressed. Then, the bidirectional LSTM network is used to extract the time-series features of the data. The proposed method uses fixed-length data frames as the input and does not require gait cycle detection, which avoids the impact of cycle detection errors on the recognition accuracy. We performed experiments on three public benchmark datasets: UCI-HAR, HuGaDB, and WISDM. The results show that SConvLSTM performs better than most of those reporting the best performance methods, at present, on the three datasets.

Adapting a deep convolutional RNN model with imbalanced regression loss for improved spatio-temporal forecasting of extreme wind speed events in the short to medium range

Abstract. The number of wind farms and amount of wind power production in Europe, both on- and offshore, have increased rapidly in the past years. To ensure grid stability and on-time (re)scheduling of maintenance tasks and to mitigate fees in energy trading, accurate predictions of wind speed and wind power are needed. Particularly, accurate predictions of extreme wind speed events are of high importance to wind farm operators as timely knowledge of these can both prevent damages and offer economic preparedness. This work explores the possibility of adapting a deep convolutional recurrent neural network (RNN)-based regression model to the spatio-temporal prediction of extreme wind speed events in the short to medium range (12 h lead time in 1 h intervals) through the manipulation of the loss function. To this end, a multi-layered convolutional long short-term memory (ConvLSTM) network is adapted with a variety of imbalanced regression loss functions that have been proposed in the literature: inversely weighted, linearly weighted and squared error-relevance area (SERA) loss. Forecast performance is investigated for various intensity thresholds of extreme events, and a comparison is made with the commonly used mean squared error (MSE) and mean absolute error (MAE) loss. The results indicate the inverse weighting method to most effectively shift the forecast distribution towards the extreme tail, thereby increasing the number of forecasted events in the extreme ranges, considerably boosting the hit rate and reducing the root-mean-squared error (RMSE) in those ranges. The results also show, however, that such improvements are invariably accompanied by a pay-off in terms of increased overcasting and false alarm ratio, which increase both with lead time and intensity threshold. The inverse weighting method most effectively balances this trade-off, with the weighted MAE loss scoring slightly better than the weighted MSE loss. It is concluded that the inversely weighted loss provides an effective way to adapt deep learning to the task of imbalanced spatio-temporal regression and its application to the forecasting of extreme wind speed events in the short to medium range.

Hybrid Multi-Scale Convolutional Long Short-Term Memory Network for Remaining Useful Life Prediction and Offset Analysis

Abstract Prognostic and health management (PHM) has become increasingly popular due to the requirement of improved maintenance techniques in the industry. Remaining useful life (RUL) estimation is an important parameter through which PHM can be utilized to implement timely and cost-effective maintenance. Due to recent advancements in sensor-based and other Industry 4.0 related technologies, data-driven methods for RUL estimation have become more prevalent and effective. In this paper, a novel data-driven method for sensor-based RUL estimation using a combination of multi-scale convolutional neural network (MS-CNN) and long short-term memory (LSTM) is proposed. The proposed hybrid multi-scale convolutional LSTM (HMCL) model is capable of extracting both spatial features of various scales and temporal features from the input data to provide accurate RUL predictions. L2 regularization and dropout techniques are used to reduce overfitting. The performance of the proposed model is evaluated using the C-MAPSS dataset. It achieves excellent performance as compared to other state-of-the-art methods making it a promising approach for sensor-based RUL prediction. Additionally, to discern the cause for occurrence of offsets, i.e., deviations in the model’s predictions with the true RUL value, an offset analysis is carried out. Through the analysis, an estimate on the location and cause of offsets is established and based on the sensory input data, offsets are identified using an SVM classification model. Despite being a simple classification model, it is able to achieve a decent performance in classifying the offsets.

Hybrid attention-based temporal convolutional bidirectional LSTM approach for wind speed interval prediction.

Precise wind speed prediction is crucial for the management of the wind power generation systems. However, the stochastic nature of the wind speed makes optimal interval prediction very complicated. In this paper, a hybrid approach consisting of improved complete ensemble empirical mode decomposition with adaptive noise (ICEEMDAN), temporal convolutional network with attention mechanism (ATCN), and bidirectional long short-term memory network (Bi-LSTM) is proposed for wind speed interval prediction (WSIP). First, ICEEMDAN is used to pre-process the raw data by decomposing the wind signal to several intrinsic mode functions. ATCN is used to reduce the uncertainty from the denoised data and extract the important temporal and spatial characteristics. Then, Bi-LSTM is used to forecast the high-quality intervals for the wind speed. Existing approaches observe a decline in the forecasting performance when the time ahead increases. As a result, the hybrid approach is evaluated using 5-min, 10-min, and 30-min ahead WSIP. To evaluate the novelty of the proposed approach, an experiment is conducted utilising wind speed data from the Garden City, Manhattan wind farm. The experimental results demonstrate that the proposed framework outperformed the comparison models with percentage improvements of 36%, 47%, and 17% for 5-min, 10-min, and 30-min ahead WSIP.

Differential diagnosis of prostate cancer and benign prostatic hyperplasia based on DCE-MRI using bi-directional CLSTM deep learning and radiomics.

Dynamic contrast-enhanced MRI (DCE-MRI) is routinely included in the prostate MRI protocol for a long time; its role has been questioned. It provides rich spatial and temporal information. However, the contained information cannot be fully extracted in radiologists' visual evaluation. More sophisticated computer algorithms are needed to extract the higher-order information. The purpose of this study was to apply a new deep learning algorithm, the bi-directional convolutional long short-term memory (CLSTM) network, and the radiomics analysis for differential diagnosis of PCa and benign prostatic hyperplasia (BPH). To systematically investigate the optimal amount of peritumoral tissue for improving diagnosis, a total of 9 ROIs were delineated by using 3 different methods. The results showed that bi-directional CLSTM with ± 20% region growing peritumoral ROI achieved the mean AUC of 0.89, better than the mean AUC of 0.84 by using the tumor alone without any peritumoral tissue (p = 0.25, not significant). For all 9 ROIs, deep learning had higher AUC than radiomics, but only reaching the significant difference for ± 20% region growing peritumoral ROI (0.89 vs. 0.79, p = 0.04). In conclusion, the kinetic information extracted from DCE-MRI using bi-directional CLSTM may provide helpful supplementary information for diagnosis of PCa.

MLA-LSTM: A Local and Global Location Attention LSTM Learning Model for Scoring Figure Skating

Video-based scoring using neural networks is a very important means for evaluating many sports, especially figure skating. Although many methods for evaluating action quality have been proposed, there is no uniform conclusion on the best feature extractor and clip length for the existing methods. Furthermore, during the feature aggregation stage, these methods cannot accurately locate the target information. To address these tasks, firstly, we systematically compare the effects of the figure skating model with three different feature extractors (C3D, I3D, R3D) and four different segment lengths (5, 8, 16, 32). Secondly, we propose a Multi-Scale Location Attention Module (MS-LAM) to capture the location information of athletes in different video frames. Finally, we present a novel Multi-scale Location Attentive Long Short-Term Memory (MLA-LSTM), which can efficiently learn local and global sequence information in each video. In addition, our proposed model has been validated on the Fis-V and MIT-Skate datasets. The experimental results show that I3D and 32 frames per second are the best feature extractor and clip length for video scoring tasks. In addition, our model outperforms the current state-of-the-art method hybrid dynAmic-statiC conText-aware attentION NETwork (ACTION-NET), especially on MIT-Skate (by 0.069 on Spearman’s rank correlation). In addition, it achieves average improvements of 0.059 on Fis-V compared with Multi-scale convolutional skip Self-attentive LSTM Module (MS-LSTM). It demonstrates the effectiveness of our models in learning to score figure skating videos.

A ConvLSTM Neural Network Model for Spatiotemporal Prediction of Mining Area Surface Deformation Based on SBAS-InSAR Monitoring Data

The surface deformation caused by underground mining leads to damage to surface buildings and brings potential safety hazards and property losses. The demand for reliable prediction methods of surface deformation in mining areas is becoming increasingly significant. At present, most prediction methods are based on sampling points; however, these methods neglect to consider local and overall spatial features, and this oversight affects the spatial accuracy of prediction results. The data form of the prediction output is often discontinuous and not intuitive. In order to solve this problem, the spatiotemporal prediction method of surface deformation in mining areas is a very effective proposal. However, few scholars have proposed a solution based on this idea. In this study, a convolution long short-term memory (ConvLSTM) neural network for surface deformation spatiotemporal prediction based on time-series interferometric synthetic aperture radar (InSAR) is proposed to directly predict the overall spatial deformation of the surface in the mining area. First, based on Sentinel-1A images of the Jinchuan Mining Area, Jinchang, Gansu province, China, the time-series InSAR surface deformation data of the study area from January 2018 to October 2020 (81 scenes) are obtained using small baseline subset InSAR (SBAS-InSAR) technology. Because of the large value scale of surface deformation in the mining area, we propose a method to fit the maximum and minimum values of time-series deformation, respectively, and carry out piecewise numerical compression. Then, based on the ConvLSTM neural network layer, construct the spatiotemporal prediction model of time-series InSAR surface deformation. Support vector regression (SVR), multilayer perceptron (MLP) regression, and the gray model [GM (1,1)] are used as benchmark methods. The prediction results of our models are compared with the three benchmark methods. The comparison results show that the prediction effect of the ConvLSTM model with optimal input time steps is significantly better than the benchmark methods in the comprehensive performance of various evaluation metrics, especially for the metrics used to evaluate the error. This shows that the ConvLSTM model has relatively fine spatiotemporal prediction performance for time-series InSAR surface deformation. Based on the InSAR time-series deformation monitoring results of the Jinchuan Mining Area, we carry out the spatiotemporal prediction of surface deformation in the subsequent 50 time steps (600 days). The results agree with the natural deformation development law in band collection statistics and 3-D representation of deformation; moreover, the reliability of numerical spatial distribution is relatively high. This result can be used to intuitively evaluate the overall surface deformation of the mining area in the monitoring range, find potential hazards in time, and take measures to address these hazards quickly. At the same time, this research process also provides a new concept design for such problems.

Personalized Breath-Based Biometric Authentication With Wearable Multimodality

Breath sounds have been shown as a potential biometric in personal identification and verification. In this article, we show that by combining signals captured by motion sensors on the chest with audio features, we can further improve the performance of these tasks. Our work is composed of three main contributions: the design of a newly created Internet-of-Things (IoT) device, the publication of a novel dataset, and newly proposed multimodal models. Specifically, we first design new hardware that consists of an acoustic sensor to collect audio features from the nose, as well as an accelerometer and gyroscope to collect movement on the chest as a result of an individual’s breathing. Using this hardware, we collect and publish a dataset from a number of sessions with different volunteers, where each session includes three common gestures: normal, deep, and strong breathing. Finally, we experiment with two multimodal models based on convolutional long short term memory (CNN-LSTM) and temporal convolutional networks (TCNs) architectures. The results demonstrate the suitability of our new hardware for both verification and identification tasks.