Spatial-temporal Attention Mechanism Research Articles

Analysis of slope stochastic fields using a novel deep learning model with attention mechanism

This paper proposes a novel deep learning model incorporating attention mechanisms for the analysis of slope stochastic fields. Initially, a deep learning model is designed to digitally image the stochastic field features of soil strength variability. This is achieved by discretizing the slope soil stochastic field using the Karhunen-Loeve expansion method and transforming the discrete results into digital images. These images are then used to establish a Convolutional Neural Network (CNN) surrogate model that maps the implicit relationship between stochastic field images and slope functional function values, thus calculating the probability of slope failure. The precision of the CNN surrogate model is enhanced through Bayesian optimization and five-fold cross-validation. Moreover, to overcome the limitations of existing data-driven landslide stability prediction models, this study also introduces a Spatial-Temporal Attention (STA) mechanism. By combining the CNN with Long Short-Term Memory (LSTM) networks, the model can accurately approximate the actual results of slope stability calculations in scenarios of high-dimensional representation imaging of stochastic fields and low-probability slope instability. Consequently, this significantly improves the computational efficiency of slope reliability analysis considering stochastic field simulations.

A transportation Revitalization index prediction model based on Spatial-Temporal attention mechanism

The global outbreak of COVID-19 has had a substantial impact on various sectors worldwide, including the economy, healthcare, entertainment, policy formulation, and international relations, with the transportation industry being particularly hard-hit. To curb the widespread transmission of the virus, many regions globally have implemented policies and measures to restrict transportation. These actions not only directly affect the transportation industry but also further impose a severe impact on the economy and societal development of various areas. In this context, the Transportation Revitalization Index (TRI) becomes particularly important. It can evaluate the degree of recovery of city traffic conditions after the pandemic, and accurate prediction of TRI can help governments and decision-makers respond more precisely to the challenges that the pandemic brings to the transportation industry. However, existing research primarily focuses on the direct correlation between TRI change data and COVID-19 pandemic data, without fully considering the dynamic spatial correlation features and time dependency features that affect the nonlinear changes of TRI. In light of the above situation, this study proposes a Deep Spatial-Temporal prediction model based on the Attention Mechanism (DeepST-AM). The DeepST-AM deeply integrates historical TRI data with multivariate pandemic information and uses a spatial–temporal attention mechanism to capture the deep and complex spatial–temporal information of urban data. To more accurately capture the long-term complex features of TRI data, this paper designs a Gaussian temporal convolution model dedicated to TRI data. To validate the effectiveness of DeepST-AM, researchers used real data from 29 core cities in China as samples and compared the performance of DeepST-AM with existing multiple methods on TRI prediction tasks. The experimental results showed that compared to other methods, the DeepST-AM proposed in this paper has a significant advantage in the long-term prediction tasks of TRI in terms of performance evaluation, indicator prediction, etc. In summary, this research provides a more accurate and comprehensive prediction model for the traffic recovery status after the pandemic, hoping to provide strong support for future decisions.

Spatial–temporal hypergraph based on dual-stage attention network for multi-view data lightweight action recognition

For the problems of irrelevant frames and high model complexity in action recognition, we propose a Spatial–Temporal Hypergraph based on Dual-Stage Attention Network (STHG-DAN) for multi-view data lightweight action recognition. It includes two stages: Temporal Attention Mechanism based on Trainable Threshold (TAM-TT) and Hypergraph Convolution based on Dynamic Spatial–Temporal Attention Mechanism (HG-DSTAM). In the first stage, TAM-TT uses a learning threshold to extract keyframes from multi-view videos, with the multi-view data serving as a guarantee for providing more comprehensive information subsequently; In the second stage, HG-DSTAM divides the human joints into three parts: trunk, hand and leg to build spatial–temporal hypergraphs, extracts high-order features from spatial–temporal hypergraphs constructed of multi-view human body joints, inputs them into the dynamic spatial–temporal attention mechanism, and learns the intra frame correlation of multi-view data between the joint features of body parts, which can obtain the significant areas of action; We use multi-scale convolution operation and depth separable network, which can realize efficient action recognition with a few trainable parameters. We experiment on the NTU-RGB+D, NTU-RGB+D 120 and the imitating traffic police gesture dataset. The performance and accuracy of the model are better than the existing algorithms, effectively improving the machine and human body language interaction cognitive ability.

Attention Mechanism with Spatial-Temporal Joint Deep Learning Model for the Forecasting of Short-Term Passenger Flow Distribution at the Railway Station

Accurate understanding of passenger flow distribution is crucial for effective station crowd management. However, due to the complexity and randomness of passenger flow and the unclear spatial-temporal correlation between functional areas within the station, predicting the spatiotemporal distribution dynamics of inflow and future short-term distribution trends is challenging. Emerging deep learning models offer valuable insights for accurately predicting passenger flow distribution. Thus, we propose a deep learning architecture, named “ST-Bi-LSTM,” which combines a bidirectional long short-term memory network with a spatial-temporal attention mechanism. Initially, we outline the methodologies of Bi-LSTM, the DeepWalk-based spatial attention mechanism, and the temporal attention mechanism. The spatial attention mechanism is employed to extract station spatial network topology information and enhance the representation of passenger flow characteristics in highly correlated areas during the forecasting process. Simultaneously, the temporal attention Bi-LSTM is utilized for capturing temporal correlations. The architecture comprises four branches dedicated to station real-time video monitoring data, spatial network topology, function area attributes, and train timetables. Subsequently, leveraging in-station CCTV data, passenger travel behavior data, and train timetables, we apply the architecture to the Tianjin West High-Speed Railway Station. We conduct a comparative analysis of the prediction performance and time complexity of the proposed architecture against existing baseline models, demonstrating superior performance and robustness exhibited by the ST-Bi-LSTM model (achieving a reduction in RMSE of over 10%). This study facilitates the transition of station management from passive response to active prediction of station passenger flow dynamics.

Velocity-aware spatial-temporal attention LSTM model for inverse dynamic model learning of manipulators.

An accurate inverse dynamics model of manipulators can be effectively learned using neural networks. However, further research is required to investigate the impact of spatiotemporal variations in manipulator motion sequences on network learning. In this work, the Velocity Aware Spatial-Temporal Attention Residual LSTM neural network (VA-STA-ResLSTM) is proposed to learn a more accurate inverse dynamics model, which uses a velocity-aware spatial-temporal attention mechanism to extract dynamic spatiotemporal features selectively from the motion sequence of the serial manipulator. The multi-layer perception (MLP) attention mechanism is adopted to capture the correlation between joint position and velocity in the motion sequence, and the state correlation between hidden units in the LSTM network to reduce the weight of invalid features. A velocity-aware state fusion approach of LSTM network hidden units' states is proposed, which utilizes variation in joint velocity to adapt to the temporal characteristics of the manipulator dynamic motion, improving the generalization and accuracy of the neural network. Comparative experiments have been conducted on two open datasets and a self-built dataset. Specifically, the proposed method achieved an average accuracy improvement of 61.88% and 43.93% on the two different open datasets and 71.13% on the self-built dataset compared to the LSTM network. These results demonstrate a significant advancement in accuracy for the proposed method. Compared with the state-of-the-art inverse dynamics model learning methods of manipulators, the modeling accuracy of the proposed method in this paper is higher by an average of 10%. Finally, by visualizing attention weights to explain the training procedure, it was found that dynamic modeling only relies on partial features, which is meaningful for future optimization of inverse dynamic model learning methods.

STAVOS: A Medaka Larval Cardiac Video Segmentation Method Based on Deep Learning

Medaka (Oryzias latipes), as a crucial model organism in biomedical research, holds significant importance in fields such as cardiovascular diseases. Currently, the analysis of the medaka ventricle relies primarily on visual observation under a microscope, involving labor-intensive manual operations and visual assessments that are cumbersome and inefficient for biologists. Despite attempts by some scholars to employ machine learning methods, limited datasets and challenges posed by the blurred edges of the medaka ventricle have constrained research to relatively simple tasks such as ventricle localization and heart rate statistics, lacking precise segmentation of the medaka ventricle edges. To address these issues, we initially constructed a video object segmentation dataset comprising over 7000 microscopic images of medaka ventricles. Subsequently, we proposed a semi-supervised video object segmentation model named STAVOS, incorporating a spatial-temporal attention mechanism. Additionally, we developed an automated system capable of calculating various parameters and visualizing results for a medaka ventricle using the provided video. The experimental results demonstrate that STAVOS has successfully achieved precise segmentation of medaka ventricle contours. In comparison to the conventional U-Net model, where a mean accuracy improvement of 0.392 was achieved, our model demonstrates significant progress. Furthermore, when compared to the state-of-the-art Tackling Background Distraction (TBD) model, there is an additional enhancement of 0.038.

Synergetic gait prediction and compliant control of SEA-driven knee exoskeleton for gait rehabilitation.

In recent years, lower limb exoskeletons have achieved satisfactory clinical curative effects in rehabilitating stroke patients. Furthermore, generating individualized trajectories for each patient and avoiding secondary injury in rehabilitation training are important issues. This paper explores the utilization of series elastic actuator (SEA) to deliver compliant force and enhance impact resistance in human-robot interaction, and we present the design of novel knee exoskeleton driven by SEA. Subsequently, the novel gait trajectory prediction method and compliant control method are proposed. The attention-based CNN-LSTM model is established to generate personalized gait trajectories for affected limbs, in which the spatial-temporal attention mechanism is adopted to improve the prediction accuracy. The compliant control strategy is proposed to nonlinearly and adaptively tune impedance parameters based on artificial potential field (APF) method, and active rehabilitation training is carried out in the coordination space to guarantee patient safety. The experimental results based on four healthy subjects demonstrated that synergetic gait prediction model could satisfactorily characterize the coordination movement with higher accuracy. The compliant control could limit the patient's movement in the safe coordination tunnel while considering personalization and flexibility.

Interpretable spatial-temporal attention convolutional network for rainfall forecasting

The interpretability of rainfall forecasting models is a major challenge in the field of artificial intelligence. Its importance is equal to the evaluation of model accuracy. Owing to the uncertainty and nonlinearity of the rainfall forecasting process, existing hydrological rainfall forecasting models often have low prediction robustness, and the machine learning method ignores the influence of physical factors in the surrounding areas; thus, the physical interpretability of the model is inferior. This study proposes an interpretable spatial-temporal attention convolution network for rainfall interval forecasting (IDSTA-TCN). To analyze and mine the uncertainty of input data more effectively, we enhanced a variational mode decomposition (EBO-VMD) algorithm based on Bayesian optimization to decompose the input data. To improve the performance of the original temporal convolution network (TCN) unit in rainfall forecasting and to effectively use the spatial-temporal information of rainfall forecasting data, we improved the spatial-temporal attention mechanism, dynamically combined the geographical information of the meteorological stations in the surrounding area to assist the network unit in better mining the nonlinear impact of historical input information on current rainfall forecasting, and improved the prediction performance of the IDSTA-TCN model. Finally, we designed a more reasonable error factor adjustment interval output strategy to better quantify the impact of prediction errors on the rainfall interval prediction results. Moreover, we conducted a series of experiments to compare and analyze the accuracy of each model according to the most commonly used evaluation indicators in the field of rainfall forecasting. The results demonstrate that the proposed IDSTA-TCN model is superior to the baseline models in most cases. We preliminarily analyzed the spatial-temporal attention weights, which explained the physical rationality of our model.

A spatial–temporal deep learning-based warning system against flooding hazards with an empirical study in Taiwan

With climate change intensifying, increasingly extreme rainfall events which lead to flooding are occurring at a higher frequency. As such, it is crucial to develop and implement useful flood forecasting models which can provide early knowledge of the severity of flood risks and allow decision makers to take preventive and response actions which reduce human casualties. This study, carried out in collaboration with the National Science and Technology Center for Disaster Reduction (NCDR) of Taiwan, utilizes deep learning techniques combined with attention mechanisms to establish a data-driven flood forecasting framework which can serve as an early warning system for flooding hazards. The overall framework, which utilizes historical flood depth data from an array of flood sensors while engineering additional features from KRID (estimation Rainfall with Integrated radar-gauge Data) observed rainfall data consists of two phases. The first phase is a flood event detection model built using a classification-based random forest algorithm. The second phase is a flood depth prediction model which utilizes a spatial–temporal attention mechanism combined with a transformation-gated long short-term memory (LSTM).This novel model, known as the Spatial–Temporal Attention Transformation-Gated LSTM (STA-TGLSTM), is capable of predicting flood levels over the next three hours. In an empirical study based in Changhua County, Taiwan, STA-TGLSTM achieves the lowest RMSE losses for predicting the next three hours of flood levels compared to baseline machine learning models. Notably, the accuracy of predicting the flood sensor depth for the first hour (T+1) reaches 76% when STA-TGLSTM is applied. In comparison to traditional hydrological models, this data-driven model offers greater computational efficiency, providing future flood depth information within seconds. Moreover, as demonstrated in the empirical study, by visualizing the weight values of the spatial–temporal attention mechanism, valuable insights into the relationship between each flood sensor and the spatiotemporal variables can be readily gained.

PPG-based Continuous BP Waveform Estimation Using Polarized Attention-guided Conditional Adversarial Learning Model.

The blood pressure (BP) waveform is a vital source of physiological and pathological information concerning the cardiovascular system. This study proposes a novel attention-guided conditional generative adversarial network (cGAN), named PPG2BP-cGAN, to estimate BP waveforms based on photoplethysmography (PPG) signals. The proposed model comprises a generator and a discriminator. Specifically, the UNet3+-based generator integrates a full-scale skip connection structure with a modified polarized self-attention module based on a spatial-temporal attention mechanism. Additionally, its discriminator comprises PatchGAN, which augments the discriminative power of the generated BP waveform by increasing the perceptual field through fully convolutional layers. We demonstrate the superior BP waveform prediction performance of our proposed method compared to state-of-the-art (SOTA) techniques on two independent datasets. Our approach first pre-trained on a dataset containing 683 subjects and then tested on a public dataset. Experimental results from the Multi-parameter Intelligent Monitoring in Intensive Care dataset show that the proposed method achieves a root mean square error of 3.54, mean absolute error of 2.86, and Pearson coefficient of 0.99 for BP waveform estimation. Furthermore, the estimation errors (mean error ± standard deviation error) for systolic BP and diastolic BP are 0.72 ± 4.34 mmHg and 0.41 ± 2.48 mmHg, respectively, meeting the American Association for the Advancement of Medical Instrumentation standard. Our approach exhibits significant superiority over SOTA techniques on independent datasets, thus highlighting its potential for future applications in continuous cuffless BP waveform measurement.

Action Recognition and Benchmark Using Event Cameras.

Recent years have witnessed remarkable achievements in video-based action recognition. Apart from traditional frame-based cameras, event cameras are bio-inspired vision sensors that only record pixel-wise brightness changes rather than the brightness value. However, little effort has been made in event-based action recognition, and large-scale public datasets are also nearly unavailable. In this paper,we present an event-based action recognition framework called EV-ACT. The Learnable Multi-Fused Representation (LMFR) is first proposed to integrate multiple event information in a learnable manner. The LMFR with dual temporal granularity is fed into the event-based slow-fast network for the fusion of appearance and motion features. A spatial-temporal attention mechanism is introduced to further enhance the learning capability of action recognition. To prompt research in this direction, we have collected the largest event-based action recognition benchmark named THUE-ACT-50 and the accompanying THUE-ACT-50-CHL dataset under challenging environments, including a total of over 12,830 recordings from 50 action categories, which is over 4 times the size of the previous largest dataset. Experimental results show that our proposed framework could achieve improvements of over 14.5%, 7.6%, 11.2%, and 7.4% compared to previous works on four benchmarks. We have also deployed our proposed EV-ACT framework on a mobile platform to validate its practicality and efficiency.

Prediction of Pollutant Concentration Based on Spatial-Temporal Attention, ResNet and ConvLSTM.

Accurate and reliable prediction of air pollutant concentrations is important for rational avoidance of air pollution events and government policy responses. However, due to the mobility and dynamics of pollution sources, meteorological conditions, and transformation processes, pollutant concentration predictions are characterized by great uncertainty and instability, making it difficult for existing prediction models to effectively extract spatial and temporal correlations. In this paper, a powerful pollutant prediction model (STA-ResConvLSTM) is proposed to achieve accurate prediction of pollutant concentrations. The model consists of a deep learning network model based on a residual neural network (ResNet), a spatial-temporal attention mechanism, and a convolutional long short-term memory neural network (ConvLSTM). The spatial-temporal attention mechanism is embedded in each residual unit of the ResNet to form a new residual neural network with the spatial-temporal attention mechanism (STA-ResNet). Deep extraction of spatial-temporal distribution features of pollutant concentrations and meteorological data from several cities is carried out using STA-ResNet. Its output is used as an input to the ConvLSTM, which is further analyzed to extract preliminary spatial-temporal distribution features extracted from the STA-ResNet. The model realizes the spatial-temporal correlation of the extracted feature sequences to accurately predict pollutant concentrations in the future. In addition, experimental studies on urban agglomerations around Long Beijing show that the prediction model outperforms various popular baseline models in terms of accuracy and stability. For the single-step prediction task, the proposed pollutant concentration prediction model performs well, exhibiting a root-mean-square error (RMSE) of 9.82. Furthermore, even for the pollutant prediction task of 1 to 48 h, we performed a multi-step prediction and achieved a satisfactory performance, being able to achieve an average RMSE value of 13.49.

A lightweight neural network for lung nodule detection based on improved ghost module.

Computer tomography images are the preferred method of preoperative evaluation for lung disease. However, it remains difficult to detect and recognize nodules accurately and efficiently due to poor data imaging quality, heavy reliance on physician experience and the need for more human-computer interaction for diagnosis. Currently, image nodule detection based on deep convolutional neural networks has gained much momentum. To alleviate doctors' tremendous labor in the diagnosis procedure, and improve the accuracy of intelligent detection of lung nodules, we improved GhostNet and proposed a lightweight neural network for object detection for lung nodule image detection. Firstly, the bneck structure in the backbone feature extraction network is adopted and improved from the structure of MobileNetV3. The weights are adjusted by changing the initial channel attention mechanism and introducing a spatial-temporal attention mechanism. Then, in the enhanced feature extraction part, we mainly use depth-separable convolution blocks to replace the 3×3 convolution of the original network for the purpose of reducing the model parameters, and make more improvements based on the network structure to enhance the applicability of the network. Diagnostic precision, recall, F1-score, mAP and parameter count were calculated. According to our lightweight neural network, F1-score, precision, and recall were 0.87, 86.34%, and 86.69%, respectively. Based on our dataset, the Yolov4-GNet network proposed in this research outperforms the current neural networks on both precision and recall as well as F1. The lung nodule detection method proposed in this research not only simplifies the processing of images, but also outperforms comparable methods in nodule detection rate and positioning accuracy, providing a new way for lung nodule detection.

A multi-sensor mapping Bi-LSTM model of bridge monitoring data based on spatial-temporal attention mechanism

The premise of intelligent structural health monitoring is to build a digital benchmark model representing the multi-sensor mapping relationship. Previously, researchers have used machine learning methods such as Long-and-Short-Term-Memory (LSTM) networks for modeling multi-sensor mapping relationships. However, the typical LSTM networks treat multi-dimensional input data equally, ignoring the possible correlations of different periods and sensor placements. Consequently, they can hardly obtain an accurate model for long-time series and multidimensional datasets. To address the issues, by introducing a two-layer attention mechanism in the temporal and spatial dimensions, an improved Bi-direction LSTM neural network model embedding the attention mechanism is proposed. The model focuses on grasping the non-stationary response process and the spatial correlation of multi-sensors. The accuracy and efficiency of the proposed approach are verified through two case studies. In addition, based on the visualization of attention weight assignments, the spatial–temporal attention distribution is studied to give an engineering interpretation.

Remaining useful life prediction of lithium-ion batteries using a spatial temporal network model based on capacity self-recovery effect

The accurate prediction of the remaining useful life (RUL) of lithium-ion batteries (LIBs) is crucial to ensure the safe operation of electric vehicles (EVs). However, predicting RUL accurately is challenging due to the capacity self-recovery effect, which has been seldom addressed in existing literature, and the complex degradation mechanism. Here, in order to solve the above issues, we extract the health factors from measurable data to reveal the characteristics of battery degradation. Additionally, to capture spatial correlation of multi-series data and address the capacity self-recovery effect, the spatial-temporal remain useful life forecasting (STRULF) model is designed, which combines the spatial LSTM and temporal attention mechanism (TAM). Furthermore, we embed an online updating (OL) strategy with AutoDropout to enhance the model accuracy and generalization under different operating conditions. We evaluate the performance of our proposed model using four lithium-ion battery datasets. The experimental results reveal that the OL-STRULF model outperforms other methods in predicting RUL accurately.

Attention-based spatial–temporal multi-graph convolutional networks for casualty prediction of terrorist attacks

At present, terrorism has become an important factor affecting world peace and development. As the time series data of terrorist attacks usually show a high degree of spatial–temporal correlation, the spatial–temporal prediction of casualties in terrorist attacks is still a significant challenge in the field of counter-terrorism. Most of the existing terrorist attack prediction methods lack the ability to model the spatial–temporal dynamic correlation of the time series data of terrorist attacks, so they cannot yield satisfactory prediction results. In this paper, we propose a novel Attention-based spatial–temporal multi-graph convolutional network (AST-MGCN) for casualty prediction of terrorist attacks. Specifically, we construct the spatial adjacency graph and spatial diffusion graph based on the different social-spatial dynamic relationships of terrorist attacks and determine the multi-scale period of time series data of terrorist attacks by using wavelet transform to model the temporal trend, period and closeness properties of terrorist attacks. The AST-MGCN mainly consists of spatial multi-graph convolution for extracting social-spatial features in multi-views and temporal convolution for capturing the transition rules. In addition, we also use the spatial–temporal attention mechanism to effectively capture the most relevant spatial–temporal dynamic information. Experiments on public datasets demonstrate that the proposed model outperforms the state-of-the-art baselines.

Spatial–temporal information model-based load current interval prediction for transmission lines

Load current interval prediction (LCIP) plays an increasingly significant role in transmission lines load demand uncertainty assessment and becomes necessary for power system dispatch and management. However, the stochastic, volatility and non-stationary characteristics of load demand bring challenges to high-quality prediction intervals (PIs) estimation. In this article, a hybrid LCIP model based on Spatial–Temporal Attention integrating Temporal Convolutional Network (STATCN) and residual estimation is proposed. First, to diminish the uncertainty in the prediction process, a spatial-partitioned method and a temporal-phased method are proposed for decoupling the spatial and temporal correlations among transmission lines. Then, a spatial–temporal attention mechanism is fused in temporal convolutional networks to enhance the sensitivity of ensemble model for important information from the temporal and spatial feature dimensions. Finally, in the STATCN interval prediction model, a residual estimation optimization strategy is designed to eliminate quantile crossings and ensure the validity of the prediction interval results. The performance of the proposed prediction framework is verified by actual numerical simulation. The results demonstrate that our proposed method has higher quality PIs compared to the existing state-of-the-art interval prediction methods.

Spatial correlation and temporal attention-based LSTM for remaining useful life prediction of turbofan engine

Remaining useful life (RUL) prediction has always been a core task of prognostics and health management technology, which is crucial to the reliable and safe operation of mechanical equipment. In recent years, data-driven methods have played an increasingly important role in RUL prediction. However, most methods have no effective mechanism to measure the importance of different variables at different times, and lack information extraction in the temporal dimension, which seriously affects the prediction accuracy of RUL. To solve the negative impact of these problems, this paper proposes a long short-term memory framework based on spatial correlation and temporal attention mechanism (SCTA-LSTM). Firstly, spatial correlation attention fully considers the relationship between variables, and adaptively measures the importance of different variables in input data at different times. Then, temporal attention enhances further the information extraction ability of LSTM in the temporal dimension, and finally, the fully connected network is used to predict RUL. To verify the effectiveness of the proposed SCTA-LSTM, two different turbofan engine simulation datasets from the Prognostics Center of Excellence at NASA Ams Research Center are used for modeling and testing. The experimental results show that this method outperforms other existing methods.

Forecasting of water consumption by integrating spatial and temporal characteristics of short-term water use in cities

Urban water consumption prediction is an important basis for urban water resource management and optimal scheduling of water supply. Most existing prediction methods disregard the spatial–temporal characteristics of water consumption data and utilize incomplete data mining, resulting in prediction models with low accuracy. To accurately predict short-term water demand, a data-cleaning algorithm based on statistical distributions and velocity constraints is proposed in this study. The long- and short-term time-series network (LSTNet), automated spatial–temporal graph prediction (AutoSTG), and attention-based spatial–temporal graphical convolutional neural network (ASTGCN) deep learning models were investigated for automatically capturing the spatial–temporal dynamic features. Additionally, the short-term predictions for 1, 3, 6, 12, and 24 h forecasting periods were investigated based on the water consumption data of water plants in Tianjin. The results revealed that the proposed data-cleaning algorithm can effectively manage single-point anomalies and continuous outliers, thus improving prediction accuracy. The LSTNet, AutoSTG, and ASTGCN deep learning models, particularly the ASTGCN model based on the spatial–temporal attention mechanism, significantly improved the prediction accuracy and adaptive spatial–temporal feature extraction. The final prediction model that combines data-cleaning algorithms with the ASTGCN deep learning method achieved root mean square error, mean absolute error, accuracy, and relative squared error values of 566.4 m3, 377.7 m3, 0.964, and 0.112, respectively. The proposed forecasting framework can improve short-term water use forecasting at multiple sites in the city and provide support for fine-grained water allocation.

Toward attention-based learning to predict the risk of brain degeneration with multimodal medical data.

Brain degeneration is commonly caused by some chronic diseases, such as Alzheimer's disease (AD) and diabetes mellitus (DM). The risk prediction of brain degeneration aims to forecast the situation of disease progression of patients in the near future based on their historical health records. It is beneficial for patients to make an accurate clinical diagnosis and early prevention of disease. Current risk predictions of brain degeneration mainly rely on single-modality medical data, such as Electronic Health Records (EHR) or magnetic resonance imaging (MRI). However, only leveraging EHR or MRI data for the pertinent and accurate prediction is insufficient because of single-modality information (e.g., pixel or volume information of image data or clinical context information of non-image data). Several deep learning-based methods have used multimodal data to predict the risks of specified diseases. However, most of them simply integrate different modalities in an early, intermediate, or late fusion structure and do not care about the intra-modal and intermodal dependencies. A lack of these dependencies would lead to sub-optimal prediction performance. Thus, we propose an encoder-decoder framework for better risk prediction of brain degeneration by using MRI and EHR. An encoder module is one of the key components and mainly focuses on feature extraction of input data. Specifically, we introduce an encoder module, which integrates intra-modal and inter-modal dependencies with the spatial-temporal attention and cross-attention mechanism. The corresponding decoder module is another key component and mainly parses the features from the encoder. In the decoder module, a disease-oriented module is used to extract the most relevant disease representation features. We take advantage of a multi-head attention module followed by a fully connected layer to produce the predicted results. As different types of AD and DM influence the nature and severity of brain degeneration, we evaluate the proposed method for three-class prediction of AD and three-class prediction of DM. Our results show that the proposed method with integrated MRI and EHR data achieves an accuracy of 0.859 and 0.899 for the risk prediction of AD and DM, respectively. The prediction performance is significantly better than the benchmarks, including MRI-only, EHR-only, and state-of-the-art multimodal fusion methods.