Temporal Fusion Research Articles

Data-centric multi-task surgical phase estimation with sparse scene segmentation

PurposeSurgical workflow estimation techniques aim to divide a surgical video into temporal segments based on predefined surgical actions or objectives, which can be of different granularity such as steps or phases. Potential applications range from real-time intra-operative feedback to automatic post-operative reports and analysis. A common approach in the literature for performing automatic surgical phase estimation is to decouple the problem into two stages: feature extraction from a single frame and temporal feature fusion. This approach is performed in two stages due to computational restrictions when processing large spatio-temporal sequences.MethodsThe majority of existing works focus on pushing the performance solely through temporal model development. Differently, we follow a data-centric approach and propose a training pipeline that enables models to maximise the usage of existing datasets, which are generally used in isolation. Specifically, we use dense phase annotations available in Cholec80, and sparse scene (i.e., instrument and anatomy) segmentation annotation available in CholecSeg8k in less than 5% of the overlapping frames. We propose a simple multi-task encoder that effectively fuses both streams, when available, based on their importance and jointly optimise them for performing accurate phase prediction.Results and conclusionWe show that with a small fraction of scene segmentation annotations, a relatively simple model can obtain comparable results than previous state-of-the-art and more complex architectures when evaluated in similar settings. We hope that this data-centric approach can encourage new research directions where data, and how to use it, plays an important role along with model development.

Interpretable wind speed prediction with multivariate time series and temporal fusion transformers

Wind power has been utilized well in power systems, so steady and successful wind speed forecasting is crucial to security management power grid market economy. To date, most researchers have often discounted the interpretability of prediction models, leading to obscure forecasts. This study puts forward a unique forecasting methodology that incorporates notable decomposition techniques, multifactor interpretable forecasting models, and optimization algorithms. In the proposed model, variational mode decomposition is employed to break down the raw wind speed sequence into a set of intrinsic mode functions. Adaptive differential evolution is then used for optimizing several parameters of temporal fusion transformers (TFT) to achieve satisfactory forecasting performance. TFT is a new attention-based deep learning model that puts together high-performance multi-horizon prediction and interpretable insights into temporal dynamics. Empirical studies using eight real-world 1-h wind speed data sets in Albert, Canada, and Five Points, USA demonstrate that the system using the proposed model outperforms those employing other comparable models in nearly all performance metrics. Examples of TFT's interpretable outputs are the importance ranking of the decomposed wind speed sub-sequences and meteorological data and attention analysis of different step lengths. The findings signify substantial progress for wind speed prediction and aid policymakers.

Research on Wastewater Treatment Monitoring Algorithms Based on Deep Convolutional Neural Networks

Around the problems of data loss, noise, and different temporal and spatial scale features of urban wastewater treatment process data, a method of monitoring and predicting wastewater treatment process data based on deep convolutional neural networks is proposed in the paper. Firstly, to address the problem that urban wastewater treatment process data has multiple spatial and temporal scale characteristics, which makes it difficult for the data to be used effectively, a spatial and temporal data fusion model based on fuzzy neural network (FNN) is proposed. Fuzzy neural networks have strong generalization ability and robustness. Secondly, to enable accurate and real-time monitoring of the content of the monitored components in the effluent of the urban wastewater treatment process, an intelligent prediction model based on SDF-FNN is established for the effluent. Finally, in order to verify the effectiveness of this intelligent prediction model, the model is tested using data collected from actual municipal wastewater treatment plants. The experimental results show that the wastewater treatment intelligent monitoring model is effective and can predict the content of the effluent monitoring index with high accuracy.

Water Quality Chl-a Inversion Based on Spatio-Temporal Fusion and Convolutional Neural Network

The combination of remote sensing technology and traditional field sampling provides a convenient way to monitor inland water. However, limited by the resolution of remote sensing images and cloud contamination, the current water quality inversion products do not provide both high temporal resolution and high spatial resolution. By using the spatio-temporal fusion (STF) method, high spatial resolution and temporal fusion images were generated with Landsat, Sentinel-2, and GaoFen-2 data. Then, a Chl-a inversion model was designed based on a convolutional neural network (CNN) with the structure of 4-(136-236-340)-1-1. Finally, the results of the Chl-a concentrations were corrected using a pixel correction algorithm. The images generated from STF can maintain the spectral characteristics of the low-resolution images with the R2 between 0.7 and 0.9. The Chl-a inversion results based on the spatio-temporal fused images and CNN were verified with measured data (R2 = 0.803), and then the results were improved (R2 = 0.879) after further combining them with the pixel correction algorithm. The correlation R2 between the Chl-a results of GF2-like and Sentinel-2 were both greater than 0.8. The differences in the spatial distribution of Chl-a concentrations in the BYD lake gradually increased from July to August. Remote sensing water quality inversion based on STF and CNN can effectively achieve high frequency in time and fine resolution in space, which provide a stronger scientific basis for rapid diagnosis of eutrophication in inland lakes.

A deep LSTM network for the Spanish electricity consumption forecasting

Nowadays, electricity is a basic commodity necessary for the well-being of any modern society. Due to the growth in electricity consumption in recent years, mainly in large cities, electricity forecasting is key to the management of an efficient, sustainable and safe smart grid for the consumer. In this work, a deep neural network is proposed to address the electricity consumption forecasting in the short-term, namely, a long short-term memory (LSTM) network due to its ability to deal with sequential data such as time-series data. First, the optimal values for certain hyper-parameters have been obtained by a random search and a metaheuristic, called coronavirus optimization algorithm (CVOA), based on the propagation of the SARS-Cov-2 virus. Then, the optimal LSTM has been applied to predict the electricity demand with 4-h forecast horizon. Results using Spanish electricity data during nine years and half measured with 10-min frequency are presented and discussed. Finally, the performance of the proposed LSTM using random search and the LSTM using CVOA is compared, on the one hand, with that of recently published deep neural networks (such as a deep feed-forward neural network optimized with a grid search) and temporal fusion transformers optimized with a sampling algorithm, and, on the other hand, with traditional machine learning techniques, such as a linear regression, decision trees and tree-based ensemble techniques (gradient-boosted trees and random forest), achieving the smallest prediction error below 1.5%.

COVID 19 Pandemic, Socio-Economic Behaviour and Infection Characteristics: An Inter-Country Predictive Study Using Deep Learning.

This work aims to develop a data driven multi-horizon incidence forecasting model considering the inter-country variability in static socio-economic factors. The specific objectives of this study are to predict the future country-wise COVID 19 incidences, to locate the influences of individual socio-economic factors on the predictions, to analyze the clusters of countries on the basis of influential explanatory variables and thus to search for intra-cluster and inter-cluster characteristics. To that respect this study has used the deep neural network based temporal fusion transformer for the predictions, Pearson correlation to understand the influence of socio-economic variables on incidence and hierarchical clustering for cluster-analysis. The findings conclude that the inter-country infection related predictions vary widely over spatio-temporal variability and different socio-economic variables have different influences over this inter-country variability. It is observed that greater the population size, stronger the global connectedness, larger the social cohesion, higher the population density and meaningful the gender based discrimination higher will be the future spread. On the other hand greater the development level, higher the nutritional status, greater the access to quality health services, greater the urban population and greater the material poverty lesser will be the future spread. Definite spatial pattern of influence of the explanatory variables emerged from cluster analysis. To minimize the vulnerability towards unforeseen biological calamities modern and sustainable development policies are needed; affluence may not guarantee less infection. But these policies should vary between economies due to the variation in socio-economic status of the countries worldwide.

A Higher-Order Motif-Based Spatiotemporal Graph Imputation Approach for Transportation Networks

Due to the incomplete coverage and failure of traffic data collectors during the collection, traffic data usually suffers from information missing. Achieving accurate imputation is critical to the operation of transportation networks. Existing approaches usually focus on the characteristic analysis of temporal variation and adjacent spatial representation, and the consideration of higher-order spatial correlations and continuous data missing attracts more attentions from the academia and industry. In this paper, by leveraging motif-based graph aggregation, we propose a spatiotemporal imputation approach to address the issue of traffic data missing. First, through motif discovery, the higher-order graph aggregation model was presented in traffic networks. It utilized graph convolution network (GCN) to polymerize the correlated segment attributes of the missing data segments. Then, the multitime dimension imputation model based on bidirectional long short-term memory (Bi-LSTM) incorporated the recent, daily-periodic, and weekly-periodic dependencies of the historical data. Finally, the spatial aggregated values and the temporal fusion values were integrated to obtain the results. We conducted comprehensive experiments based on the real-world dataset and discussed the case of random and continuous data missing by different time intervals, and the results showed that the proposed approach was feasible and accurate.

Transformer for High-Speed Train Wheel Wear Prediction With Multiplex Local–Global Temporal Fusion

The wheel wear status of high-speed trains (HSTs) is an essential indicator for their safety and reliability. When the wheel wear exceeds the warning value without timely maintenance, it will seriously affect the dynamic performance of the HST and even cause a derailment accident. With HSTs and sensor technology development, massive operation data can be obtained, which provides a new possibility for developing a data-driven algorithm for wheel wear prediction. To this end, this article proposes a novel transformer-based framework with multiplex local–global temporal fusion (LGF-Trans), which can be used for wheel wear prediction via vibration signals. First, a multiplex local temporal fusion architecture is proposed, composed of multiple local temporal attention networks (LTA-Networks). It can encode the local temporal correlation of the signal and improve the detail perception ability of the model. Subsequently, the transformer architecture is introduced, which uses the multi-head attention mechanism to fully encode the global temporal correlation features of vibration signals, thereby modeling the internal relationship between the input signal and the wheel wear status. LGF-Trans fully integrates the advantages of convolutional network and transformer architecture in local feature learning and global feature learning, thereby effectively extracting valuable features from massive noisy operating data. Experiments on the real operation dataset of CRH1A HST show that LGF-Trans can accurately predict wheel wear curves, and it has a better performance than the state-of-the-art deep learning methods. This confirms that LGF-Trans is expected to be a powerful tool for wheel wear prediction.

Two-Step Precipitable Water Vapor Fusion Method

Precipitable water vapor (PWV) is one of the key parameters in the evolution of extreme weather and climate change. However, current data fusion methods (such as Gaussian processes, spherical cap harmonics, and polynomial fitting) can hardly obtain simultaneously the PWV map with high precision and high spatiotemporal resolution. To solve this problem, a two-step-based PWV fusion (TPF) method is proposed, in which a hybrid PWV fusion model (HPFM) and a spatial and temporal fusion model (STFM) are introduced separately. In the first step, HPFM is established by combining the global pressure and temperature 2 wet (GPT2w) model, spherical harmonic functions, and polynomial fitting to obtain the PWV value with high precision at an arbitrary location in the study area. In the second step, STFM is proposed to generate the PWV map with high temporal resolution taking advantage of site-based global navigation satellite system (GNSS)-derived PWV. To validate the performance of the proposed method, GNSS observations, ERA-Interim, and ERA5 reanalysis products are selected in Yunnan Province, China, to carry out the experiment. Statistical results show that: 1) HPFM has the ability to obtain atmospheric water vapor with a root mean square (rms) of less than 3 mm in an arbitrary location of the PWV map and 2) STFM can generate PWV maps with the same temporal resolution as GNSS observations, and the accuracy of the obtained PWV values can be guaranteed. Therefore, the proposed TPF method is proven to have the ability to simultaneously retrieve PWV maps with high accuracy and spatiotemporal resolution.

Short-Term Electricity Load Forecasting Based on Temporal Fusion Transformer Model

Electricity load forecasting plays an important role in the operation of power systems. Inaccurate forecast would reduce the safety of power supply and affect the economic and social activities as well as national defense and security. In addition, the forecast results also support decision-making on electricity generation and market transactions. Traditional methods such as AR, ARIMA, SARIMA have been widely used to forecast short term electricity load. Recently, load forecasting based on artificial and deep neural networks have shown significant accuracy improvement over traditional statistical models. In this research, a novel recurrent neural network named temporal fusion transformer (TFT) is used to forecast short-term electricity load of Hanoi city. The TFT is a newly developed model and it combines the advantages of several other RNN models such as LSTM and the self-attention mechanism. In addition to historical load data, we use temperature and humidity features, and time features such as calendar month, lunar month, days of the week, hours of the day and holidays. The forecast results of TFT are compared with traditional statistical models as well as well-known RNN models. The compared results show that the proposed method is better than other methods in both MAE and MAPE criteria.

Spatio–Temporal–Spectral Collaborative Learning for Spatio–Temporal Fusion with Land Cover Changes

Spatio-temporal fusion by combining the complementary spatial and temporal advantages of multi-source remote sensing images to obtain time-series high spatial resolution images is highly desirable in monitoring surface dynamics. Currently, deep learning (DL)-based fusion methods have received extensive attention. However, existing DL-based spatio-temporal fusion methods are generally limited in fusing the images with land cover changes. In this paper, we propose a spatio-temporal-spectral collaborative learning framework for spatio-temporal fusion to alleviate this problem. Specifically, the proposed method integrates the convolutional neural network and recurrent neural network into a unified framework, consisting of three sub-networks: multi-scale siamese convolutional neural network, multi-layer convolutional recurrent neural network, and adaptive weighting fusion network. The multi-scale siamese convolutional neural network has a flexible weight-sharing network to extract multi-scale spatial-spectral features from multi-source remote sensing images. The multi-layer convolutional recurrent neural network is constructed on the convolutional long-short term memory units to comprehensively learn the land cover changes by spatial, spectral, and temporal joint features. The adaptive weighting fusion network with a spatio-temporal-spectral change loss is proposed to further improve the interpretability and robustness. The experiments were performed on the publicly available benchmark datasets featured by phenology and land cover type changes, respectively. The experimental results demonstrated the competitive performance of the proposed method than other state-of-the-art fusion methods.

Learning interlaced sparse Sinkhorn matching network for video super-resolution

How to effectively fuse inter- and intra-frame spatio-temporal information plays a key role in video super-resolution (VSR). Most existing works rely heavily on the accuracy of motion estimation and compensation for spatio-temporal feature alignment. However, they cannot perform well when suffering from large-scale and complex motions. To this end, this paper introduces an efficient and effective Interlaced Sparse Sinkhorn Matching (ISSM) network for VSR, which aligns supporting frames with the reference one in the feature space by learning optimal matching between image regions across frames. Specifically, the ISSM divides the input dense affinity matrix into two sparse block matrixes: one can match long-distance regions while the other can match short-distance regions, and then we leverage an efficient Sinkhorn method on each block to learn optimal matching. Moreover, we insert a residual atrous spatial pyramid pooling module before the ISSM, which can flexibly generate multi-scale features frame by frame to capture the multi-scale context information in images. The aligned features of each adjacent frame are then fed to a bidirectional temporal fusion module to capture the rich temporal information. Finally, the fused features are sent into a frame-wise dynamic reconstruction network to produce an HR frame. Extensive evaluations on three benchmark datasets demonstrate the superiority of our method over the state-of-the-art methods in terms of PSNR and SSIM.

Information Hiding in 60 Hz Video by Use of Temporal Color Fusion of Human Visual System

Information Hiding in 60 Hz Video by Use of Temporal Color Fusion of Human Visual System

Spatial–Temporal Fusion Based Path Planning for Source Seeking in Wireless Sensor Network

Spatial–Temporal Fusion Based Path Planning for Source Seeking in Wireless Sensor Network

Temporal consistency two-stream CNN for human motion prediction

Fusion is critical for a two-stream network. In this paper, we propose a novel temporal fusion (TF) module to fuse the two-stream joints’ information to predict human motion, including a temporal concatenation and a reinforcement trajectory spatial-temporal (TST) block that specifically designed to keep prediction temporal consistency. In particular, the temporal concatenation keeps the temporal consistency of preliminary predictions from two streams. Meanwhile, the TST block improves the spatial-temporal feature coupling. However, the TF module can increase the temporal continuities between the first predicted pose and the given poses and between each predicted pose. The fusion is based on a two-stream network that consists of a dynamic velocity stream (V-Stream) and a static position stream (P-Stream) because we found that the joints’ velocity information improves our short-term prediction, while the joints’ position information is better at long-term prediction, and they are complementary in motion prediction. Finally, our approach achieves impressive results on three benchmark datasets, including H3.6M, CMU-Mocap, and 3DPW in both short-term and long-term predictions, confirming its effectiveness and efficiency.

An Improved Stereo Matching Algorithm for Vehicle Speed Measurement System Based on Spatial and Temporal Image Fusion.

This paper proposes an improved stereo matching algorithm for vehicle speed measurement system based on spatial and temporal image fusion (STIF). Firstly, the matching point pairs in the license plate area with obviously abnormal distance to the camera are roughly removed according to the characteristic of license plate specification. Secondly, more mismatching point pairs are finely removed according to local neighborhood consistency constraint (LNCC). Thirdly, the optimum speed measurement point pairs are selected for successive stereo frame pairs by STIF of binocular stereo video, so that the 3D points corresponding to the matching point pairs for speed measurement in the successive stereo frame pairs are in the same position on the real vehicle, which can significantly improve the vehicle speed measurement accuracy. LNCC and STIF can be used not only for license plate, but also for vehicle logo, light, mirror etc. Experimental results demonstrate that the vehicle speed measurement system with the proposed LNCC+STIF stereo matching algorithm can significantly outperform the state-of-the-art system in accuracy.

Millimeter-Wave Radar and Video Fusion Vehicle Detection based on Adaptive Kalman Filtering

Road traffic safety has always been the focus of social concern, and the actual road traffic scenes, where only a single sensor is applied cannot cope with the interference brought by complex external factors, which makes vehicle detection extremely challenging. This paper focuses on a vehicle detection algorithm for the fusion of millimeter-wave radar sensor and monocular camera sensor, including the calibration of millimeter-wave radar and camera, the establishment of a temporal fusion model of the two sensors. Finally, the target information obtained from the two sensors is fused with the data using the adaptive Kalman filter fusion algorithm, which can reduce data ambiguity and increase the reliability and validity of the data. Experiments show that the method can overcome the shortcomings of single sensor in target detection, and the obtained target information is more comprehensive compared with the monocular camera detection results.

Temporal Fusion Transformers for interpretable multi-horizon time series forecasting

Multi-horizon forecasting often contains a complex mix of inputs – including static (i.e. time-invariant) covariates, known future inputs, and other exogenous time series that are only observed in the past – without any prior information on how they interact with the target. Several deep learning methods have been proposed, but they are typically ‘black-box’ models that do not shed light on how they use the full range of inputs present in practical scenarios. In this paper, we introduce the Temporal Fusion Transformer (TFT) – a novel attention-based architecture that combines high-performance multi-horizon forecasting with interpretable insights into temporal dynamics. To learn temporal relationships at different scales, TFT uses recurrent layers for local processing and interpretable self-attention layers for long-term dependencies. TFT utilizes specialized components to select relevant features and a series of gating layers to suppress unnecessary components, enabling high performance in a wide range of scenarios. On a variety of real-world datasets, we demonstrate significant performance improvements over existing benchmarks, and highlight three practical interpretability use cases of TFT.

Improving the spatiotemporal fusion accuracy of fractional vegetation cover in agricultural regions by combining vegetation growth models

Spatiotemporal fusion has provided a feasible way to generate fractional vegetation cover (FVC) data with high spatial and temporal resolution. However, when the currently available spatiotemporal fusion methods are applied over agricultural regions, they usually underestimate high FVC values at the peak vegetation growth stage with medium FVC values as base data. This mainly results from inconsistencies in the temporal variations between fine- and coarse-resolution data if substantial temporal changes occur in vegetation. Therefore, a Spatial and Temporal Fusion method combining with Vegetation Growth Models (STF-VGM) was proposed to address this problem in this study, which incorporates vegetation growth models into the fusion process. Unlike other spatiotemporal fusion methods that mainly rely on changes in coarse-resolution data for prediction, STF-VGM fully utilizes available coarse- and fine-resolution time series data, including uncontaminated information in cloud/cloud shadow contaminated images. By establishing vegetation growth models with time series data, a conversion relationship between coarse- and fine-resolution FVC that changes along with the nonlinear vegetation change process can be extracted. STF-VGM makes prediction based on this variable relationship. A typical agricultural region located in the North China Plain was selected as the study area. The validation results indicated that the prediction accuracy for high FVC values was significantly improved using STF-VGM compared to the commonly used Enhanced Spatial and Temporal Adaptive Reflectance Fusion Model (ESTARFM) and Flexible Spatiotemporal DAta Fusion (FSDAF) methods (STF-VGM: coefficient of determination (R2) = 0.9491, root mean square error (RMSE) = 0.0650, average difference (AD) = -0.0092; ESTARFM: R2 = 0.9341, RMSE = 0.1127, AD = -0.0631; FSDAF: R2 = 0.9224, RMSE = 0.1110, AD = -0.0599). The satisfactory performance of STF-VGM was also achieved in predicting FVC values at other vegetation growth stages (early growth stage: R2 = 0.8277, RMSE = 0.0440, AD = 0.0027; rapid growth stage: R2 = 0.9183, RMSE = 0.0844, AD = 0.0500). In addition, STF-VGM also has the potential to improve the spatiotemporal fusion accuracy of other vegetation parameters and vegetation indices, which will be evaluated in the future.

Spatio-temporal graph convolutional network for diagnosis and treatment response prediction of major depressive disorder from functional connectivity.

The pathophysiology of major depressive disorder (MDD) has been explored to be highly associated with the dysfunctional integration of brain networks. It is therefore imperative to explore neuroimaging biomarkers to aid diagnosis and treatment. In this study, we developed a spatiotemporal graph convolutional network (STGCN) framework to learn discriminative features from functional connectivity for automatic diagnosis and treatment response prediction of MDD. Briefly, dynamic functional networks were first obtained from the resting‐state fMRI with the sliding temporal window method. Secondly, a novel STGCN approach was proposed by introducing the modules of spatial graph attention convolution (SGAC) and temporal fusion. A novel SGAC was proposed to improve the feature learning ability and special anatomy prior guided pooling was developed to enable the feature dimension reduction. A temporal fusion module was proposed to capture the dynamic features of functional connectivity between adjacent sliding windows. Finally, the STGCN proposed approach was utilized to the tasks of diagnosis and antidepressant treatment response prediction for MDD. Performances of the framework were comprehensively examined with large cohorts of clinical data, which demonstrated its effectiveness in classifying MDD patients and predicting the treatment response. The sound performance suggests the potential of the STGCN for the clinical use in diagnosis and treatment prediction.