Attentive Recurrent Neural Network Research Articles

A long-term prediction method for PM2.5 concentration based on spatiotemporal graph attention recurrent neural network and grey wolf optimization algorithm

With the rapid advancement of global industrialization and urbanization, the problem of air pollution, primarily attributed to PM2.5, as the main component has become increasingly serious, directly affecting people's health. Therefore, it is necessary to forecast the long-term PM2.5 levels. However, most existing prediction methods for PM2.5 concentration lack sufficient extraction of urban spatial characteristics. Therefore, this paper proposes a novel method based on spatiotemporal graph attention recurrent neural network and grey wolf optimization (GWO-GART). Firstly, a long-term prediction model of PM2.5, the Graph Attention Cycle Network (GART), is constructed by using Graph Attention network (GAT), Graph Neural Network (GNN) and Rated Recurrent Unit (GRU) networks. In the GART model, a novel multi-GAT is proposed for the first time, which can extract the spatial features of cities circularly and obtain more implicit associations between cities. Secondly, we utilize grey wolf optimization (GWO) to select the best set of hyperparameters for the model. This automated approach aids in designing the network structure of GART. We compared GWO-GART with three single models and two hybrid models using a dataset collected from the real world. The results show that, compared with the state-of-the-art (SOTA) models, GWO-GART had an average reduction of 2.13% in RMSE, 2.47% in MAE, and an average increase of 1.55% in CSI on all datasets. This demonstrates that GWO-GART provides an efficient approach to precisely forecast long-term PM2.5 levels in urban regions.

Enhancing Dimensional Emotion Recognition from Speech through Modulation-Filtered Cochleagram and Parallel Attention Recurrent Network

Dimensional emotion can better describe rich and fine-grained emotional states than categorical emotion. In the realm of human–robot interaction, the ability to continuously recognize dimensional emotions from speech empowers robots to capture the temporal dynamics of a speaker’s emotional state and adjust their interaction strategies in real-time. In this study, we present an approach to enhance dimensional emotion recognition through modulation-filtered cochleagram and parallel attention recurrent neural network (PA-net). Firstly, the multi-resolution modulation-filtered cochleagram is derived from speech signals through auditory signal processing. Subsequently, the PA-net is employed to establish multi-temporal dependencies from diverse scales of features, enabling the tracking of the dynamic variations in dimensional emotion within auditory modulation sequences. The results obtained from experiments conducted on the RECOLA dataset demonstrate that, at the feature level, the modulation-filtered cochleagram surpasses other assessed features in its efficacy to forecast valence and arousal. Particularly noteworthy is its pronounced superiority in scenarios characterized by a high signal-to-noise ratio. At the model level, the PA-net attains the highest predictive performance for both valence and arousal, clearly outperforming alternative regression models. Furthermore, the experiments carried out on the SEWA dataset demonstrate the substantial enhancements brought about by the proposed method in valence and arousal prediction. These results collectively highlight the potency and effectiveness of our approach in advancing the field of dimensional speech emotion recognition.

Cloud-load forecasting via decomposition-aided attention recurrent neural network tuned by modified particle swarm optimization

Recent improvements in networking technologies have led to a significant shift towards distributed cloud-based services. However, adequate management of computation resources by providers is vital to maintain the costs of operations and quality of services. A robust system is needed to forecast demand and prevent excessive resource allocations. Extensive literature review suggests that the potential of recurrent neural networks with attention mechanisms is not sufficiently explored and applied to cloud computing. To address this gap, this work proposes a methodology for forecasting load of cloud resources based on recurrent neural networks with and without attention layers. Utilized deep learning models are further optimized through hyperparameter tuning using a modified particle swarm optimization metaheuristic, which is also introduced in this work. To help models deal with complex non-stationary data sequences, the variational mode decomposition for decomposing complex series has also been utilized. The performance of this approach is compared to several state-of-the-art algorithms on a real-world cloud-load dataset. Captured performance metrics (R2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$R^2$$\\end{document}, mean square error, root mean square error, and index of agreement) strongly indicate that the proposed method has great potential for accurately forecasting cloud load. Further, models optimized by the introduced metaheuristic outperformed competing approaches, which was confirmed by conducted statistical validation. In addition, the best-performing forecasting model has been subjected to SHapley Additive exPlanations analysis to determine the impact each feature has on model forecasts, which could potentially be a very useful tool for cloud providers when making decisions.

Audio-visual speech recognition based on joint training with audio-visual speech enhancement for robust speech recognition

Visual features are attractive cues that can be used for robust automatic speech recognition (ASR). In particular, speech recognition performance can be improved by combining audio with visual information obtained from the speaker’s face rather than using only audio in acoustically unfavorable environments. For this reason, recently, studies on various audio-visual speech recognition (AVSR) models have been actively conducted. However, from the experimental results of the AVSR models, important information for speech recognition is mainly concentrated on audio signals, and visual information plays a role in enhancing the robustness of recognition when the audio signal is corrupted in noisy environments. Therefore, there is a limit to the improvement of the recognition performance of conventional AVSR models in noisy environments. Unlike the conventional AVSR models that directly use input audio-visual information as it is, in this paper, we propose an AVSR model that first performs AVSE to enhance target speech based on audio-visual information and then uses both audio information enhanced by the AVSE and visual information such as the speaker’s lips or face. In particular, we propose a deep AVSR model that performs end-to-end training as one model by integrating an AVSR model based on the conformer with hybrid decoding and an AVSE model based on the U-net with recurrent neural network (RNN) attention (RA). Experimental results on the LRS2-BBC and LRS3-TED datasets demonstrate that the AVSE model effectively suppresses corrupting noise and the AVSR model successfully achieves noise robustness. Especially, the proposed jointly trained model integrating the AVSE and AVSR stages into one model showed better recognition performance than the other compared methods.

InfluencerRank: Discovering Effective Influencers via Graph Convolutional Attentive Recurrent Neural Networks

As influencers play considerable roles in social media marketing, companies increase the budget for influencer marketing. Hiring effective influencers is crucial in social influencer marketing, but it is challenging to find the right influencers among hundreds of millions of social media users. In this paper, we propose InfluencerRank that ranks influencers by their effectiveness based on their posting behaviors and social relations over time. To represent the posting behaviors and social relations, the graph convolutional neural networks are applied to model influencers with heterogeneous networks during different historical periods. By learning the network structure with the embedded node features, InfluencerRank can derive informative representations for influencers at each period. An attentive recurrent neural network finally distinguishes highly effective influencers from other influencers by capturing the knowledge of the dynamics of influencer representations over time. Extensive experiments have been conducted on an Instagram dataset that consists of 18,397 influencers with their 2,952,075 posts published within 12 months. The experimental results demonstrate that InfluencerRank outperforms existing baseline methods. An in-depth analysis further reveals that all of our proposed features and model components are beneficial to discover effective influencers.

Identifying gas mixtures based on acoustic relaxation spectroscopy and attention recurrent neural network

Qualitative and quantitative detection of the composition of gaseous mixtures remains challenging in acoustic gas sensing. Traditional feature-based methods attempt to extract local features from complex numerical models, but tend to ignore crucial information owing to the lack of a global view. Deep learning can achieve competitive performance in acquiring information. This study proposes an attention recurrent gas detection (ARGD) neural network for gas sensing based on acoustic relaxation spectroscopy. This model combines a bidirectional gated recurrent unit to extract gas information and an attention module to determine the crucial features. The processed features were input into a double-task network for simultaneous prediction of the types of gas and their respective concentrations. Samples of natural gas involving CO2, CH4, and N2 were identified with six acoustic sensor pairs. The simulated and experimental results demonstrate the effectiveness and robustness of the proposed model. A classification accuracy of up to 99% was achieved, with the root mean squared error declining to 0.67%. The combination of deep learning and acoustic relaxation spectroscopy provides new ideas for further research of acoustic gas detection.

MTMC-AUR2CNet: Multi-textural multi-class attention recurrent residual convolutional neural network for COVID-19 classification using chest X-ray images

Coronavirus disease (COVID-19) has infected over 603 million confirmed cases as of September 2022, and its rapid spread has raised concerns worldwide. More than 6.4 million fatalities in confirmed patients have been reported. According to reports, the COVID-19 virus causes lung damage and rapidly mutates before the patient receives any diagnosis-specific medicine. Daily increasing COVID-19 cases and the limited number of diagnosis tool kits encourage the use of deep learning (DL) models to assist health care practitioners using chest X-ray (CXR) images. The CXR is a low radiation radiography tool available in hospitals to diagnose COVID-19 and combat this spread. We propose a Multi-Textural Multi-Class (MTMC) UNet-based Recurrent Residual Convolutional Neural Network (MTMC-UR2CNet) and MTMC-UR2CNet with attention mechanism (MTMC-AUR2CNet) for multi-class lung lobe segmentation of CXR images. The lung lobe segmentation output of MTMC-UR2CNet and MTMC-AUR2CNet are mapped individually with their input CXRs to generate the region of interest (ROI). The multi-textural features are extracted from the ROI of each proposed MTMC network. The extracted multi-textural features from ROI are fused and are trained to the Whale optimization algorithm (WOA) based DeepCNN classifier on classifying the CXR images into normal (healthy), COVID-19, viral pneumonia, and lung opacity. The experimental result shows that the MTMC-AUR2CNet has superior performance in multi-class lung lobe segmentation of CXR images with an accuracy of 99.47%, followed by MTMC-UR2CNet with an accuracy of 98.39%. Also, MTMC-AUR2CNet improves the multi-textural multi-class classification accuracy of the WOA-based DeepCNN classifier to 97.60% compared to MTMC-UR2CNet.

Bidirectional spatial–temporal traffic data imputation via graph attention recurrent neural network

Spatiotemporal traffic data is increasingly important in transportation services with the development of intelligent transportation system (ITS). However, due to various unpredictable disruptions in the data collection and storage process, traffic data is often incomplete which will seriously hinder downstream tasks if not handled properly. Most existing methods for traffic data imputation either impose too strong assumptions on the data distribution or almost ignore the interdependencies across time steps and the information expressed by missingness. In this article, we propose a graph attention recurrent neural network (GARNN) for traffic data imputation. In our model, we impute data from both temporal and spatial perspectives. First, we model the observations and missingness separately via two LSTMs to treat the missingness of data as another special information distinct from observations. Then, a decay mechanism and graph attention network (GAT) are applied to learn the interdependencies across time steps and capture the spatial correlations respectively to generate temporal estimation and spatial estimation. Finally, those two estimations are integrated into the ultimate imputation. The whole process is in a bidirection. The proposed method is evaluated on two public datasets under three different missing scenarios. Experimental results show the effectiveness of the proposed model compared with other baselines.

A novel total nitrogen prediction method based on recurrent neural networks utilizing cross-coupling attention and selective attention

Considering the critical role the total nitrogen (TN) plays in the stable operation of wastewater treatment plants (WWTPs), it is necessary to predict its future variation accurately. However, limited by current detection techniques, only a few kinds of water quality parameters (WQPs) with finite historical data can be obtained in the WWTPs. The small sample size of WQPs tremendously hinders the precision of TN prediction. In this study, a novel cross-coupling attention recurrent neural network (CCA-RNN) is proposed to overcome this problem. First, cross-coupling attention (CCA) is designed to enable the topological structures in the historical input data to be directly and effectively extracted with fewer training samples. Then, selective attention is introduced to dynamically select the useful topological relationships and the corresponding variables sent by the high-speed channel. Compared with the other state-of-the-art methods, CCA-RNN achieves the best performance on both the public and practical wastewater datasets, proving its superiority and great potential to be deployed in similar problems.

Prediction of chlorophyll‐a data based on triple‐stage attention recurrent neural network

AbstractMarine Internet of Things (IOT) is the use of Internet technology to connect various sensing devices at sea, so as to integrate maritime information and realize the monitoring and systematic management of complex data at sea. The marine environment is complex and changeable, and marine disasters occur frequently, such as red tides. Due to the sudden and destructive nature of red tide, it plays a pivotal role to monitor the occurrence of the red tide for the marine IoT, where machine learning has been widely used to predict red tides. However, they were rarely able to catch the sudden change of chlorophyll‐a, which has important practical significance for predicting the occurrence of red tide. In order to deal with the above problems, this paper proposes the triple‐stage attention‐based recurrent neural network, which can enhance the representation ability of input sequences, selectively capture dynamic spatial correlations between input multi‐channel observations in the input sequence, meanwhile adaptively capturing dynamic temporal correlations between different time intervals in the input sequence. The results show that this method outperforms the state‐of‐art baseline methods here.

Multimodal EEG Emotion Recognition Based on the Attention Recurrent Graph Convolutional Network

EEG-based emotion recognition has become an important part of human–computer interaction. To solve the problem that single-modal features are not complete enough, in this paper, we propose a multimodal emotion recognition method based on the attention recurrent graph convolutional neural network, which is represented by Mul-AT-RGCN. The method explores the relationship between multiple-modal feature channels of EEG and peripheral physiological signals, converts one-dimensional sequence features into two-dimensional map features for modeling, and then extracts spatiotemporal and frequency–space features from the obtained multimodal features. These two types of features are input into a recurrent graph convolutional network with a convolutional block attention module for deep semantic feature extraction and sentiment classification. To reduce the differences between subjects, a domain adaptation module is also introduced to the cross-subject experimental verification. This proposed method performs feature learning in three dimensions of time, space, and frequency by excavating the complementary relationship of different modal data so that the learned deep emotion-related features are more discriminative. The proposed method was tested on the DEAP, a multimodal dataset, and the average classification accuracies of valence and arousal within subjects reached 93.19% and 91.82%, respectively, which were improved by 5.1% and 4.69%, respectively, compared with the only EEG modality and were also superior to the most-current methods. The cross-subject experiment also obtained better classification accuracies, which verifies the effectiveness of the proposed method in multimodal EEG emotion recognition.

Continual learning with attentive recurrent neural networks for temporal data classification

Continual learning is an emerging research branch of deep learning, which aims to learn a model for a series of tasks continually without forgetting knowledge obtained from previous tasks. Despite receiving a lot of attention in the research community, temporal-based continual learning techniques are still underutilized. In this paper, we address the problem of temporal-based continual learning by allowing a model to continuously learn on temporal data. To solve the catastrophic forgetting problem of learning temporal data in task incremental scenarios, in this research, we propose a novel method based on attentive recurrent neural networks, called Temporal Teacher Distillation (TTD). TTD solves the catastrophic forgetting problem in an attentive recurrent neural network based on three hypotheses, namely Rotation Hypothesis, Redundant Hypothesis, and Recover Hypothesis. Rotation Hypothesis and Redundant hypotheses could cause the attention shift phenomenon, which degrades the model performance on the learned tasks. Moreover, not considering the Recover Hypothesis increases extra memory usage in continuously training different tasks. Therefore, the proposed TTD based on the above hypotheses complements the inadequacy of the existing methods for temporal-based continual learning. For evaluating the performance of our proposed method in task incremental setting, we use a public dataset, WIreless Sensor Data Mining (WISDM), and a synthetic dataset, Split-QuickDraw-100. According to experimental results, the proposed TTD significantly outperforms state-of-the-art methods by up to 14.6% and 45.1% in terms of accuracy and forgetting measures, respectively. To the best of our knowledge, this is the first work that studies continual learning in real-world incremental categories for temporal data classification with attentive recurrent neural networks and provides the proper application-oriented scenario.

Gated Attention Recurrent Neural Network: A Deeping Learning Approach for Radar-Based Precipitation Nowcasting

Precipitation nowcasting predicts the future rainfall intensity in local areas in a brief time that impacts directly on human life. In this paper, we express the precipitation nowcasting as a spatiotemporal sequence prediction problem. Predictive learning for a spatiotemporal sequence aims to construct a model of natural spatiotemporal processes to predict the future frames based on historical frames. The spatiotemporal process is an abstraction of some of the spatial things in nature that change with time, and they usually do not change very dramatically. To simplify the model and facilitate the training, we considered that the spatiotemporal process satisfies the generalized Markov properties. The natural spatiotemporal processes are nonlinear and non-stationary in many aspects. The processes are not satisfied with the first-order Markov properties when making predictions, such as the nonlinear movement, expansion, dissipation, and intensity enhancement of echoes. To describe such complex spatiotemporal variations, higher-order Markov models need to be used for the modeling. However, many of the previous models for spatiotemporal prediction constructed were based on first-order Markov properties, losing information on the higher-order variations. Thus, we propose a recurrent neural network which satisfies the multi-order Markov properties to create more accurate spatiotemporal predictions. In this network, the core component is the memory cell structure of the gated attention mechanism, which combines the current input information, extracts the historical state that best matches the existing input from the historical multi-period memory information, and then predicts the future. Through this principle of the gated attention, we could extract the historical state information that is richer and deeper to predict the future and more accurately describe the changing characteristics of motion. The experiments show that our GARNN network captures the spatiotemporal characteristics better and obtains excellent results in the precipitation forecasting with radar echoes.

Intelligent machinery health prognostics under variable operation conditions with limited and variable-length data

High accuracy health prognostics are significant to machinery intelligent operation and maintenance. Current data-driven prognostics achieve great success that benefits from amply learning samples. In fact, data scarcity challenge widely exists in machinery prognostics and health management, especially for high-end equipment. This study aims to solve this dilemma and proposes a novel meta learning algorithm reconstructed by classic variable-length prediction mode and attention mechanism, namely meta attention recurrent neural network (MARNN). Specifically, we first develop the encoder-decoder with attention mechanism (EDA) cell to perform episodic learning for the subtask-level upgrade. Then multiple subtasks with EDA as prediction models are aggregated to accomplish meta-level upgrade, thus mining the general degradation knowledge from historical datasets. Finally, cross-domain prognostics tasks can be easily realized through fine-tuning tricks, and three rotating machinery run-to-failed experiments are conducted to prove the generalizations of MARNN, which can obtain desired results even when the on-site adaptation data is reduced to one-twentieth.

Federated Learning for Privacy-Aware Human Mobility Modeling.

Human mobility modeling is a complex yet essential subject of study related to modeling important spatiotemporal events, including traffic, disease spreading, and customized directions and recommendations. While spatiotemporal data can be collected easily via smartphones, current state-of-the-art deep learning methods require vast amounts of such privacy-sensitive data to generate useful models. This work investigates the creation of spatiotemporal models using a Federated Learning (FL) approach—a machine learning technique that avoids sharing personal data with centralized servers. More specifically, we examine three centralized models for next-place prediction: a simple Gated Recurrent Unit (GRU) model, as well as two state-of-the-art centralized approaches, Flashback and DeepMove. Flashback is a Recurrent Neural Network (RNN) that utilizes historical hidden states with similar context as the current spatiotemporal context to improve performance. DeepMove is an attentional RNN that aims to capture human mobility's regularity while coping with data sparsity. We then implemented models based on FL for the two best-performing centralized models. We compared the performance of all models using two large public datasets: Foursquare (9,450 million check-ins, February 2009 to October 2010) and Gowalla (3,300 million check-ins, April 2012 to January 2014). We first replicated the performance of both Flashback and DeepMove, as reported in the original studies, and compared them to the simple GRU model. Flashback and GRU proved to be the best performing centralized models, so we further explored both in FL scenarios, including several parameters such as the number of clients, rounds, and epochs. Our results indicated that the training process of the federated models was less stable, i.e., the FL versions of both Flashback and GRU tended to have higher variability in the loss curves. The higher variability led to a slower convergence and thus a poorer performance when compared to the corresponding centralized models. Model performance was also highly influenced by the number of federated clients and the sparsity of the evaluation dataset. We additionally provide insights into the technical challenges of applying FL to state-of-the-art deep learning methods for human mobility.

Short-Term Nationwide Airport Throughput Prediction With Graph Attention Recurrent Neural Network

With the dynamic air traffic demand and the constrained capacity resources, accurately predicting airport throughput is essential to ensure the efficiency and resilience of air traffic operations. Many research efforts have been made to predict traffic throughputs or flight delays at an airport or over a network. However, it is still a challenging problem due to the complex spatiotemporal dynamics of the highly interacted air transportation systems. To address this challenge, we propose a novel deep learning model, graph attention neural network stacking with a Long short-term memory unit (GAT-LSTM), to predict the short-term airport throughput over a national air traffic network. LSTM layers are included to extract the temporal correlations in the data, while the graph attention mechanism is used to capture the spatial dependencies. For the graph attention mechanism, two graph modeling methods, airport-based graph and OD-pair graph are explored in this study. We tested the proposed model using real-world air traffic data involving 65 major airports in China over 3 months in 2017 and compared its performance with other state-of-the-art models. Results showed that the temporal pattern was the dominate factor, compared to the spatial pattern, in predicting airport throughputs over an air traffic network. Among the prediction models that we compared, both the proposed model and LSTM performed well on prediction accuracy over the entire network. Better performance of the proposed model was observed when focusing on airports with larger throughputs. We also conducted an analysis on model interpretability. We found that spatiotemporal correlations in the data were learned and shown via the model parameters, which helped us to gain insights into the topology and the dynamics of the air traffic network.

Deep recurrent extreme learning machine for behind-the-meter photovoltaic disaggregation

In recent years, sustainable sources of energies attract significant interest due to the serious environmental issues of fossil fuels. Rooftop photovoltaic (PV) panels are among the important power generation technologies exploited in many modern countries. Most PV panels are installed behind-the-meter (BTM), resulting in a lack of observability of actual load and PV power generation in a power distribution system. This paper proposes a novel supervised spatiotemporal approach to accurately disaggregate the net-load data of a set of neighboring residential units. To this end, spatiotemporal correlations of a group of neighboring residential units are modeled using a weighted undirected graph where the nodes store the temporal features. The edges reflect the spatial correlation between neighboring residential units and are determined by the information analysis technique. Afterward, a generative graph attention recurrent neural network (RNN) is devised for capturing highly nonlinear patterns of input graphs using an RNN encoder. In addition, to boost the generalization capacity and robustness of the proposed model, we reconstruct the input graphs using sparse contractive decoders. Finally, an extreme learning machine (ELM) neural network is employed to disaggregate the input net-load time-series of a set of residential units using the extracted complex spatiotemporal patterns. Experimental results on the real-world Pecan Street dataset demonstrate the superiority of the proposed RNN-ELM method over the recent BTM disaggregation techniques.

Graph correlated attention recurrent neural network for multivariate time series forecasting

Multivariate time series(MTS) forecasting is an urgent problem for numerous valuable applications. At present, attention-based methods can relieve recurrent neural networks’ limitations in MTS forecasting that are hard to focus on key information and capture long-term dependencies, but they fail to learn the time-varying pattern based on the reliable interaction. To reinforce the memory ability of key features across time, we propose a Graph Correlated Attention Recurrent Neural Network(GCAR). GCAR first nests Feature-level attention in the graph attention module to complement external feature representations on the extraction of multi-head temporal correlations. Then Multi-level attention is designed to add target factors’ impact on the selection of external correlation and achieve a fine-grained distinction of external features’ contribution. To better capture different series’ continuous dynamic changes, two parallel LSTMs are respectively applied to learn historical target series and external feature representations’ temporal dependencies. Finally, a fusion gate is employed to balance their information conflicts. The performance of GCAR model is tested on 4 datasets, and results show GCAR model performs the most stable and greatest predictive accuracy as the increasing of predicted horizons compared with state-of-the-art models even if the multivariate time series present strong volatility and randomness.

Prediction of Time Series Gene Expression and Structural Analysis of Gene Regulatory Networks Using Recurrent Neural Networks.

Methods for time series prediction and classification of gene regulatory networks (GRNs) from gene expression data have been treated separately so far. The recent emergence of attention-based recurrent neural network (RNN) models boosted the interpretability of RNN parameters, making them appealing for the understanding of gene interactions. In this work, we generated synthetic time series gene expression data from a range of archetypal GRNs and we relied on a dual attention RNN to predict the gene temporal dynamics. We show that the prediction is extremely accurate for GRNs with different architectures. Next, we focused on the attention mechanism of the RNN and, using tools from graph theory, we found that its graph properties allow one to hierarchically distinguish different architectures of the GRN. We show that the GRN responded differently to the addition of noise in the prediction by the RNN and we related the noise response to the analysis of the attention mechanism. In conclusion, this work provides a way to understand and exploit the attention mechanism of RNNs and it paves the way to RNN-based methods for time series prediction and inference of GRNs from gene expression data.

A Dual-Staged Attention Based Conversion-Gated Long Short Term Memory for Multivariable Time Series Prediction

In multivariate time series modeling, it is necessary to capture short-term mutation and long-term dependence information simultaneously. However, mechanism which can capture short-term change is difficult to be used to grasp long-term dependence information, and vice versa. In order to capture both short-term mutation and long-term dependence information in the same model, this paper proposed a dual-staged attention mechanism based on conversion-gated Long Short Term Memory network(DA-CG-LSTM). Hyperbolic tangent function is introduced into the input-gate and the forget-gate of Long Short Term Memory network(LSTM), which improves the ability of the network to extract the short-term mutation information. Further, dual-staged attention mechanism is added in the network, which includes input attention and temporal attention. Input attention adaptively extracts the feature relations of exogenous sequences, and temporal attention selects the relevant hidden layer states across all the time steps. Experiments on air quality and traffic flow time series data show that the proposed network has lower average absolute error, average absolute percentage error and root mean square error by more than 50% compared with Dual-staged Attention Recurrent Neural Network(DA-RNN) and Transformation-gated LSTM(TG-LSTM).