Residual Long Short-term Memory Research Articles

Multimodal data-based human motion intention prediction using adaptive hybrid deep learning network for movement challenged person.

Recently, social demands for a good quality of life have increased among the elderly and disabled people. So, biomedical engineers and robotic researchers aimed to fuse these techniques in a novel rehabilitation system. Moreover, these models utilized the biomedical signals acquired from the human body's particular organ, cells, or tissues. The human motion intention prediction mechanism plays an essential role in various applications, such as assistive and rehabilitation robots, that execute specific tasks among elders and physically impaired individuals. However, more complications are increased in the human-machine-based interaction techniques, creating more scope for personalized assistance for the human motion intention prediction system. Therefore, in this paper, an Adaptive Hybrid Network (AHN) is implemented for effective human motion intention prediction. Initially, multimodal data like electroencephalogram (EEG)/Electromyography (EMG) signals and sensor measures data are collected from the available data resource. The gathered EEG/EMG signals are then converted into spectrogram images and sent to AH-CNN-LSTM, which is the integration of an Adaptive Hybrid Convolution Neural Network (AH-CNN) with a Long Short-Term Memory (LSTM) network. Similarly, the data details of sensor measures are directly subjected to AH-CNN-Res-LSTM, which is the combination of Adaptive Hybrid CNN with Residual Network and LSTM (Res-LSTM) to get the predictive result. Further, to enhance the prediction, the parameters in both the AH-CNN-LSTM and AH-CNN-Res-LSTM techniques are optimized using the Improved Yellow Saddle Goatfish Algorithm (IYSGA). The efficiency of the implemented model is computed by conducting the comparison experiment of the proposed technique with other standard models. The performance outcome of the developed method outperformed the other traditional methods.

Comparative Analysis of LSTM Architectures for Wind Speed Forecasting: A Case Study in Muş, Turkey

This study assesses the effectiveness of five distinct Long Short-Term Memory (LSTM) architectures for forecasting wind speed in Muş, Turkey. The models include Vanilla LSTM, Stacked LSTM, Bidirectional LSTM, Attention LSTM, and Residual LSTM. The data, obtained from the Muş Meteorological Office, underwent preprocessing to handle missing values by averaging the same day and month values between 1969 and 2023. The dataset, containing 20,088 daily wind speed measurements, was split into training and test sets, with 80% allocated for training and 20% for testing. Each model was trained over 100 epochs with a batch size of 32, and performance was assessed using Mean Squared Error (MSE), Mean Absolute Error (MAE), and Mean Absolute Percentage Error (MAPE). The Vanilla LSTM model showed the lowest MSE and MAE values, indicating superior overall performance, while the Attention LSTM model achieved the lowest MAPE, demonstrating better percentage accuracy. These findings indicate that the Vanilla and Attention LSTM models are the most effective for wind speed forecasting, with the choice between them depending on the prioritization of total error versus percentage error.

A Cognitive IoT Learning Models for Agro Climatic Estimation Aiding Farmers in Decision making

Abstract climate change continues to be an impact for every nation’s agricultural system, forecasting it is regarded as one of the most significant economic factors. For farmers to survive the increasing frequency of extreme weather events that have a detrimental effect on agricultural production, climate data and services are essential. Weather forecasts are essential for agricultural resource management because they help farmers prepare ahead of time and safeguard their crops from natural calamities. Furthermore, climate data has been fuelled by global warming, resulting in unexpected hurricanes that have even harmed agriculture’s production roots. These days, the daily forecasting of weather variables, such as rainfall, maximum temperature, and humidity, is primarily done using artificial intelligence, machine learning, and deep learning approaches. The current climate condition models require more innovation in terms of high performance and computational complexity. This study suggests Harris Hawk Optimised deep learning network and ensemble residual Long Short-term memory (R-LSTM) for climatic condition prediction that supports an improvement in crop-yield output. The climate parameter is used to train the proposed model, which is then assessed using the several state-of-the-art learning techniques and performance metrics like accuracy, precision, recall, specificity, and F1-score. The results show that the suggested model has a 97.3% accuracy rate, a 96.9% precision rate, a 96.6% recall rate, and a 97.4% F1-score. The results of the current study show that the suggested model is a very good choice for predicting climate change. By increasing crop output productivity, this in turn significantly contributes to raising farmers’ standard of living.

Residual LSTM-based short duration forecasting of polarization current for effective assessment of transformers insulation

The empirical application of polarization and depolarization current (PDC) measurement of transformers facilitates the extraction of critical insulation-sensitive parameters. This technique, rooted in time-domain dielectric response analysis, forms the bedrock for parameterization and insulation modeling. However, the inherently time-consuming nature of polarization current measurements renders them susceptible to data corruption. This article explores deep-learning-based short-duration techniques for forecasting polarization current to address this limitation. By incorporating spatial shortcuts, the residual long short-term memory (LSTM) network facilitates the seamless propagation of spatial and temporal gradients. Furthermore, the relative forecasting assessment of the proposed residual LSTM model’s performance is made against traditional LSTM, attention LSTM, gated recurrent units (GRU), and convolutional neural network (CNN) models. Thus, optimal model selection strategies are evaluated based on their capability to capture extended dependencies and short-term information present in the data. In addition, the Monte Carlo dropout prediction is employed to estimate uncertainty in polarization current forecasts. The findings demonstrate that the proposed residual LSTM network model for polarization current forecasting yields the lowest error metrics and maintains prediction consistency over the testing duration. Thus, the proposed approach significantly reduces PDC measurement time, providing an effective means to develop proactive maintenance strategies for evaluating the insulation condition of transformers.

A deep learning-based interferometric synthetic aperture radar framework for abnormal displacement deformation prediction of bridges

This paper first proposes a novel framework for combining deep learning approach and Synthetic Aperture Radar (SAR) technique to evaluate and predict the condition of bridge displacement . The Long short-term memory (LSTM) neural network and the Small Baseline Subsets InSAR (SBAS-InSAR) are used to predict the longitudinal deformation of the bridge. Firstly, the proposed framework based on LSTM is established to obtain the relationship between the longitudinal deformation of the bridge and the influence parameters (such as temperature, temporal baseline, and others). In particular, the residual connection (RC) module is adapted to form the residual long short-term memory (Res-LSTM) neural network to improve prediction accuracy and robustness. A case study in Nanjing is analyzed to verify the performance of the proposed framework. The extracted datasets obtained from Sentienl-1 are divided into the training, validation, and testing dataset, which can satisfy the requirements in the engineering field. The R 2 and MSE of the testing datasets utilizing the proposed model are 93.17% and 15.62. Furthermore, the results indicated that the proposed framework based on Res-LSTM could predict and warn the abnormal structural deformation, extend the lifespan of structures, and minimize structural damage due to excessive deformation.

ICaps-ResLSTM: Improved capsule network and residual LSTM for EEG emotion recognition

Electroencephalography (EEG) emotion recognition is an important task for brain–computer interfaces. The time, frequency, and spatial domains of EEG signals have been widely studied. However, these methods often ignore the spatial and temporal correlations in dual modules, resulting in insufficient emotional representations. In this paper, a dual module EEG emotion recognition method based on an improved capsule network and residual Long-Short Term Memory (ResLSTM) is proposed. Using an improved capsule network as the spatial module is more advantageous in learning specific EEG spatial representations. The ResLSTM of the temporal module inherits the information flow from the upper spatial module and conducts complementary learning of the spatiotemporal dual module features through residual connections, thus obtaining more discriminative EEG features and ultimately boosting the classification capabilities of the model. The average accuracy of arousal, valence, and dominance on the DEAP dataset reached 98.06%, 97.94%, and 98.15%, respectively. The DREAMER dataset’s average accuracy of arousal, valence, and dominance reached 94.97%, 94.71%, and 94.96%, respectively. The results of our experiments indicate that our method outperforms state-of-the-art approaches.

Mul-DesLSTM: An integrative multi-time granularity deep learning prediction method for urban rail transit short-term passenger flow

It is critical for the management and control of urban rail transit (URT) to be able to predict passenger flow accurately and in real time. Considering that the high-resolution data aggregated by the automatic fare collection (AFC) system is wasted, this paper analyzes the problem of applying a multi-time granularity passenger flow data fusion forecasting process. First, we examine the challenge of constructing a dataset of passenger flow data with different time granularities. Thus, an algorithm is proposed for selecting passenger flow datasets with multi-time granularity. Furthermore, a multi-time granularity dense residual network (Mul-DesLSTM) with a dense residual structure and LSTM (long short-term memory) as the predictor is constructed, inspired by a residual network. Using Mul-DesLSTM, finer-grained passenger flow features can be fused layer by layer while maintaining the accuracy of traditional single-granularity passenger flow predictions. Lastly, Mul-DesLSTM is applied to the URT system of Shanghai, China, and compared with baselines. As a result, the proposed Mul-DesLSTM outperforms the baselines with LSTM as a predictor and state-of-the-art model. When the predicted time granularity is 30 min, compared to the single-time granularity LSTM network, the mean absolute error, root mean square error, and symmetric mean absolute percentage error can be reduced by 51%, 63%, and 15%, respectively. The results can serve as a reference and basis for the operation and management of URT systems.

Salinity Modeling Using Deep Learning with Data Augmentation and Transfer Learning

Salinity management in estuarine systems is crucial for developing effective water-management strategies to maintain compliance and understand the impact of salt intrusion on water quality and availability. Understanding the temporal and spatial variations of salinity is a keystone of salinity-management practices. Process-based numerical models have been traditionally used to estimate the variations in salinity in estuarine environments. Advances in data-driven models (e.g., deep learning models) make them effective and efficient alternatives to process-based models. However, a discernible research gap exists in applying these advanced techniques to salinity modeling. The current study seeks to address this gap by exploring the innovative use of deep learning with data augmentation and transfer learning in salinity modeling, exemplified at 23 key salinity locations in the Sacramento–San Joaquin Delta which is the hub of the water-supply system of California. Historical, simulated (via a hydrodynamics and water quality model), and perturbed (to create a range of hydroclimatic and operational scenarios for data-augmentation purposes) flow, and salinity data are used to train a baseline multi-layer perceptron (MLP) and a deep learning Residual Long-Short-Term Memory (Res-LSTM) network. Four other deep learning models including LSTM, Residual Network (ResNet), Gated Recurrent Unit (GRU), and Residual GRU (Res-GRU) are also examined. Results indicate that models pre-trained using augmented data demonstrate improved performance over models trained from scratch using only historical data (e.g., median Nash–Sutcliffe efficiency increased from around 0.5 to above 0.9). Moreover, the five deep learning models further boost the salinity estimation performance in comparison with the baseline MLP model, though the performance of the latter is acceptable. The models trained using augmented data are then (a) used to develop a web-based Salinity Dashboard (Dashboard) tool that allows the users (including those with no machine learning background) to quickly screen multiple management scenarios by altering inputs and visualizing the resulting salinity simulations interactively, and (b) transferred and adapted to estimate observed salinity. The study shows that transfer learning results more accurately replicate the observations compared to their counterparts from models trained from scratch without knowledge learned and transferred from augmented data (e.g., median Nash–Sutcliffe efficiency increased from around 0.4 to above 0.9). Overall, the study illustrates that deep learning models, particularly when pre-trained using augmented data, are promising supplements to existing process-based models in estuarine salinity modeling, while the Dashboard enables user engagement with those pre-trained models to inform decision-making efficiently and effectively.

Residual LSTM based short-term load forecasting

As the modern energy systems is becoming more complex and flexible, accurate load forecasting has been the key to scheduling power to meet customers’ needs, load switching, and infrastructure development. In this paper, we propose a neural network framework based on a modified deep residual network (DRN) and a long short-term memory (LSTM) recurrent neural network (RNN) for addressing the short-term load forecasting (STLF) problem. The proposed model not only inherits the DRN’s excellent characteristic to avoid vanishing gradient for training deeper neural networks, but also continues the LSTM’s strong ability to capture nonlinear patterns for time series forecasting. Moreover, through the dimension weighted units based on attention mechanism, the dimension-wise feature response is adaptively recalibrated by explicitly modeling the interdependencies between dimensions, so that we can jointly improve the performance of the model from three aspects: depth, time and feature dimension. The snapshot ensemble method has also been applied to improve the accuracy and robustness of the proposed model. By implementing multiple sets of experiments on two public datasets, we demonstrate that the proposed model has high accuracy, robustness and generalization capability, and can perform STLF better than the existing mainstream models.

A novel multi-step ahead forecasting model for flood based on time residual LSTM

Climate change has significantly impacted hydrology, including extreme precipitation, and changing precipitation patterns that could lead to an increase in flooding. Life and property benefit from accurate and reliable multi-step flood forecasting. Recently, Recurrent Neural Network (RNN) have become increasingly popular among hydrology researchers for their ability to capture historical dependencies, simplify computations by ignoring intermediate hydrological processes, and provide higher prediction accuracy than traditional models. However, RNN-based flood prediction models face two significant challenges. Firstly, due to their strict time-serial, RNN suffer from gradient issues such as vanishing and exploding gradients, which can make training RNN models difficult. To address this issue, we propose a Residual Long Short-Term Memory (ResLSTM) model that incorporates time residual connections into the time connections of LSTM. Secondly, most flood prediction models output a deterministic value, but the natural hydrological characteristics of the basin are a nonlinear and complex system with many influencing factors that have some randomness. This requires the use of probabilistic methods to modeling. Thus, we introduce the probabilistic forecasting model Autoregressive Recurrent Networks (DeepAR) into our flood prediction model, which outputs a prediction interval rather than a deterministic value. Then, we build four flood probability prediction models by combining DeepAR and four enhanced RNN, including ResLSTM (ours), Long Short-Term Memory (LSTM), Gate Recurrent Unit (GRU), and Time Feedforward Connections Simple Gate Recurrent Unit (TFC-SGRU). The performance of these models is evaluated by the long-term hydrologic data of the Passaic and Ramapo River basins in the United States. The results demonstrate that the prediction interval of the four models is more adaptive to flood uncertainties. And the accuracy of peak flow prediction is nearly 100% within a 90% prediction probability interval. The temporal residual-based model is more accurate and robust than the original LSTM and GRU. We believe this study fills a research gap in multi-step-ahead flood probability prediction and improves the accuracy and reliability of flood prediction models.

A Novel Joint Time-Frequency Spectrum Resources Sustainable Risk Prediction Algorithm Based on TFBRL Network for the Electromagnetic Environment

To protect the electromagnetic environment and understand its current state in a timely manner, monitoring the electromagnetic environment has great practical significance, while massive amounts of data are generated. It is crucial to utilize data mining technology to extract valuable information from these massive amounts of data for effective spectrum management. Traditional spectrum prediction methods do not integrate the prior information of spectrum resource occupancy, so that the prediction of the channel state of a single frequency point is of limited significance. To address these issues, the paper describes a dynamic threshold algorithm which mines bottom noise and spectrum resource occupancy from massive electromagnetic environment data. Moreover, the paper describes a joint time-frequency spectrum resource prediction algorithm based on the time-frequency block residual LSTM (TFBRL) network, which utilizes hourly time closeness, daily period, and annual trend as prior knowledge of spectrum resources. The TFBRL network comprises three main parts: (1) a residual convolution network with a squeeze-and-excitation (SE) attention mechanism, (2) a long short term memory (LSTM) model with memory ability to capture sequence latent information, and (3) a feature fusion module based on a matrix to combine time closeness, daily period, and annual trend feature components. Experimental results demonstrate that the TFBRL network outperforms the baseline networks, improving by 31.37%, 16.00% and 13.06% compared with the best baseline for MSE, RMSE and MAE, respectively. Thus, the TFBRL network has good risk prediction performance and lays the foundation for subsequent frequency scheduling.

Daily Peak-Electricity-Demand Forecasting Based on Residual Long Short-Term Network

Forecasting the electricity demand of buildings is a key step in preventing a high concentration of electricity demand and optimizing the operation of national power systems. Recently, the overall performance of electricity-demand forecasting has been improved through the application of long short-term memory (LSTM) networks, which are well-suited to processing time-series data. However, previous studies have focused on improving the accuracy in forecasting only overall electricity demand, but not peak demand. Therefore, this study proposes adding residual learning to the LSTM approach to improve the forecast accuracy of both peak and total electricity demand. Using a residual block, the residual LSTM proposed in this study can map the residual function, which is the difference between the hypothesis and the observed value, and subsequently learn a pattern for the residual load. The proposed model delivered root mean square errors (RMSE) of 10.5 and 6.91 for the peak and next-day electricity demand forecasts, respectively, outperforming the benchmark models evaluated. In conclusion, the proposed model provides highly accurate forecasting information, which can help consumers achieve an even distribution of load concentration and countries achieve the stable operation of the national power system.

Novel Salinity Modeling Using Deep Learning for the Sacramento–San Joaquin Delta of California

Water resources management in estuarine environments for water supply and environmental protection typically requires estimates of salinity for various flow and operational conditions. This study develops and applies two novel deep learning (DL) models, a residual long short-term memory (Res-LSTM) network, and a residual gated recurrent unit (Res-GRU) model, in estimating the spatial and temporal variations of salinity. Four other machine learning (ML) models, previously developed and reported, consisting of multi-layer perceptron (MLP), residual network (ResNet), LSTM, and GRU are utilized as the baseline models to benchmark the performance of the two novel models. All six models are applied at 23 study locations in the Sacramento–San Joaquin Delta (Delta), the hub of California’s water supply system. Model input features include observed or calculated tidal stage (water level), flow and salinity at model upstream boundaries, salinity control gate operations, crop consumptive use, and pumping for the period of 2001–2019. Meanwhile, field observations of salinity at the study locations during the same period are also utilized for the development of the predictive use of the models. Results indicate that the proposed DL models generally outperform the baseline models in simulating and predicting salinity on both daily and hourly scales at the study locations. The absolute bias is generally less than 5%. The correlation coefficients and Nash–Sutcliffe efficiency values are close to 1. Particularly, Res-LSTM has slightly superior performance over Res-GRU. Moreover, the study investigates the overfitting issues of both the DL and baseline models. The investigation indicates that overfitting is not notable. Finally, the study compares the performance of Res-LSTM against that of an operational process-based salinity model. It is shown Res-LSTM outperforms the process-based model consistently across all study locations. Overall, the study demonstrates the feasibility of DL-based models in supplementing the existing operational models in providing accurate and real-time estimates of salinity to inform water management decision making.

Recognising Cattle Behaviour with Deep Residual Bidirectional LSTM Model Using a Wearable Movement Monitoring Collar

Cattle behaviour is a significant indicator of cattle welfare. With the advancements in electronic equipment, monitoring and classifying multiple cattle behaviour patterns is becoming increasingly important in precision livestock management. The aim of this study was to detect important cattle physiological states using a neural network model and wearable electronic sensors. A novel long short-term memory (LSTM) recurrent neural network model that uses two-way information was developed to accurately classify cattle behaviour and compared with baseline LSTM. Deep residual bidirectional LSTM and baseline LSTM were used to classify six behavioural patterns of cows with window sizes of 64, 128 and 256 (6.4 s, 12.8 s and 25.6 s, respectively). The results showed that when using deep residual bidirectional LSTM with window size 128, four classification performance indicators, namely, accuracy, precision, recall, and F1-score, achieved the best results of 94.9%, 95.1%, 94.9%, and 94.9%, respectively. The results showed that the deep residual bidirectional LSTM model can be used to classify time-series data collected from twelve cows using inertial measurement unit collars. Six aim cattle behaviour patterns can be classified with high accuracy. This method can be used to quickly detect whether a cow is suffering from bovine dermatomycosis. Furthermore, this method can be used to implement automated and precise cattle behaviour classification techniques for precision livestock farming.

A Residual LSTM and Seq2Seq Neural Network Based on GPT for Chinese Rice-Related Question and Answer System

Rice has a wide planting area as one of the essential food crops in China. The problem of diseases and pests in rice production has always been one of the main factors affecting its quality and yield. It is essential to provide treatment methods and means for rice diseases and pests quickly and accurately in the production process. Therefore, we used the rice question-and-answer (Q&A) community as an example. This paper aimed at the critical technical problems faced by the agricultural Q&A community: the accuracy of the existing agricultural Q&A model is low, which is challenging to meet users’ requirements to obtain answers in real-time in the production process. A network based on Attention-ResLSTM-seq2seq was used to realize the construction of the rice question and answer model. Firstly, the text presentation of rice question-and-answer pairs was obtained using the GPT pre-training model based on a 12-layer transformer. Then, ResLSTM(Residual Long Short-Term Memory) was used to extract text features in the encoder and decoder, and the output project matrix and output gate of LSTM were used to control the spatial information flow. When the network contacts the optimal state, the network only retains the constant mapping value of the input vector, which effectually reduces the network parameters and increases the network performance. Next, the attention mechanism was connected between the encoder and the decoder, which can effectually strengthen the weight of the keyword feature information of the question. The results showed that the BLEU and ROUGE of the Attention-ResLSTM-Seq2seq model reached the highest scores, 35.3% and 37.8%, compared with the other six rice-related generative question answering models.

GAN-rcLSTM: A Deep Learning Model for Radar Echo Extrapolation

The target of radar echo extrapolation is to predict the motion and development of radar echo in the future based on historical radar observation data. For such spatiotemporal prediction problems, a deep learning method based on Long Short-Term Memory (LSTM) networks has been widely used in recent years, although such models generally suffer from weak and blurry prediction. This paper proposes two models called Residual Convolution LSTM (rcLSTM) and Generative Adversarial Networks-rcLSTM (GAN-rcLSTM): The former introduces the residual module, and the latter introduces the discriminator. We use the historical data of 2017 and 2018 in the Jiangsu region as training and test sets. Experiments show that in long sequence forecasts, our model can provide more stable and clear images, while achieving higher CSI scores.

Multibranch Attentive Gated ResNet for Short-Term Spatio-Temporal Solar Irradiance Forecasting

The increasing penetration of solar generation into power grids has promoted the need for accurate and reliable short-term solar irradiance forecasting. This article introduces a novel multibranch attentive gated recurrent residual network (ResAttGRU) consisting of multiple branches of residual networks, gated recurrent units (GRUs), and the attention mechanism. The proposed multibranch ResAttGRU is capable of modeling data at various resolutions, extracting hierarchical features, and capturing short- and long-term dependencies. Moreover, this network also presents a strong multitimescale representative within the proposed architecture, while GRUs can exploit temporal information at less computational cost than the popular long short-term memory (LSTM). The novelty of the proposed architecture is to employ multiple convolutional-based branches with different filter lengths to learn multitimescale features jointly, accelerate the learning process, and reduce overfitting by leveraging shared representations as the auxiliary information. This study also compares the multibranch ResAttGRU networks with state-of-the-art deep learning methods using 18 years of NSRDB data at 12 solar sites. Finally, the proposed multibranch ResAttGRU requires 7.1% fewer parameters than multibranch residual LSTM while achieving similar average RMSE, MAE, and R-squared values.

Residual Convolution Long Short-Term Memory Network for Machines Remaining Useful Life Prediction and Uncertainty Quantification

Recently, deep learning is widely used in the field of remaining useful life (RUL) prediction. Among various deep learning technologies, recurrent neural network (RNN) and its variant, e.g., long short-term memory (LSTM) network, are gaining more attention because of their capability of capturing temporal dependence. Although the existing RNN-based approaches have demonstrated their RUL prediction effectiveness, they still suffer from the following two limitations: 1) it is difficult for RNN to extract directly degradation features from original monitoring data, and 2) most of the RNN-based prognostics methods are unable to quantify the uncertainty of prediction results. To address the above limitations, this paper proposes a new method named Residual convolution LSTM (RC-LSTM) network. In RC-LSTM, a new ResNet-based convolution LSTM (Res-ConvLSTM) layer is stacked with convolution LSTM (ConvLSTM) layer to extract degradation representations from monitoring data. Then, predicated on the RUL following a normal distribution, an appropriate output layer is constructed to quantify the uncertainty of the forecast result. Finally, the effectiveness and superiority of RC-LSTM is verified using monitoring data from accelerated degradation tests of rolling element bearings.

An Efficient Anomaly Recognition Framework Using an Attention Residual LSTM in Surveillance Videos.

Video anomaly recognition in smart cities is an important computer vision task that plays a vital role in smart surveillance and public safety but is challenging due to its diverse, complex, and infrequent occurrence in real-time surveillance environments. Various deep learning models use significant amounts of training data without generalization abilities and with huge time complexity. To overcome these problems, in the current work, we present an efficient light-weight convolutional neural network (CNN)-based anomaly recognition framework that is functional in a surveillance environment with reduced time complexity. We extract spatial CNN features from a series of video frames and feed them to the proposed residual attention-based long short-term memory (LSTM) network, which can precisely recognize anomalous activity in surveillance videos. The representative CNN features with the residual blocks concept in LSTM for sequence learning prove to be effective for anomaly detection and recognition, validating our model’s effective usage in smart cities video surveillance. Extensive experiments on the real-world benchmark UCF-Crime dataset validate the effectiveness of the proposed model within complex surveillance environments and demonstrate that our proposed model outperforms state-of-the-art models with a 1.77%, 0.76%, and 8.62% increase in accuracy on the UCF-Crime, UMN and Avenue datasets, respectively.

Host-Parasite: Graph LSTM-in-LSTM for Group Activity Recognition.

This article aims to tackle the problem of group activity recognition in the multiple-person scene. To model the group activity with multiple persons, most long short-term memory (LSTM)-based methods first learn the person-level action representations by several LSTMs and then integrate all the person-level action representations into the following LSTM to learn the group-level activity representation. This type of solution is a two-stage strategy, which neglects the "host-parasite" relationship between the group-level activity ("host") and person-level actions ("parasite") in spatiotemporal space. To this end, we propose a novel graph LSTM-in-LSTM (GLIL) for group activity recognition by modeling the person-level actions and the group-level activity simultaneously. GLIL is a "host-parasite" architecture, which can be seen as several person LSTMs (P-LSTMs) in the local view or a graph LSTM (G-LSTM) in the global view. Specifically, P-LSTMs model the person-level actions based on the interactions among persons. Meanwhile, G-LSTM models the group-level activity, where the person-level motion information in multiple P-LSTMs is selectively integrated and stored into G-LSTM based on their contributions to the inference of the group activity class. Furthermore, to use the person-level temporal features instead of the person-level static features as the input of GLIL, we introduce a residual LSTM with the residual connection to learn the person-level residual features, consisting of temporal features and static features. Experimental results on two public data sets illustrate the effectiveness of the proposed GLIL compared with state-of-the-art methods.