Fusion Of Temporal Information Research Articles

Distance Regression Enhanced with Temporal Information Fusion and Adversarial Training for Robot-Assisted Endomicroscopy.

Probe-based confocal laser endomicroscopy (pCLE) has a role in characterising tissue intraoperatively to guide tumour resection during surgery. To capture good quality pCLE data which is important for diagnosis, the probe-tissue contact needs to be maintained within a working range of micrometre scale. This can be achieved through micro-surgical robotic manipulation which requires the automatic estimation of the probe-tissue distance. In this paper, we propose a novel deep regression framework composed of the Deep Regression Generative Adversarial Network (DR-GAN) and a Sequence Attention (SA) module. The aim of DR-GAN is to train the network using an enhanced image-based supervision approach. It extents the standard generator by using a well-defined function for image generation, instead of a learnable decoder. Also, DR-GAN uses a novel learnable neural perceptual loss which combines for the first time spatial and frequency domain features. This effectively suppresses the adverse effects of noise in the pCLE data. To incorporate temporal information, we've designed the SA module which is a cross-attention module, enhanced with Radial Basis Function based encoding (SA-RBF). Furthermore, to train the regression framework, we designed a multi-step training mechanism. During inference, the trained network is used to generate data representations which are fused along time in the SA-RBF module to boost the regression stability. Our proposed network advances SOTA networks by addressing the challenge of excessive noise in the pCLE data and enhancing regression stability. It outperforms SOTA networks applied on the pCLE Regression dataset (PRD) in terms of accuracy, data quality and stability.

AMFNet: Attention-Guided Multi-Scale Fusion Network for Bi-Temporal Change Detection in Remote Sensing Images

Change detection is crucial for evaluating land use, land cover changes, and sustainable development, constituting a significant component of Earth observation tasks. The difficulty in extracting features from high-resolution images, coupled with the complexity of image content, poses challenges for traditional change detection algorithms in terms of accuracy and applicability. The recent emergence of deep learning methods has led to substantial progress in the field of change detection. However, existing frameworks often involve the simplistic integration of bi-temporal features in specific areas, lacking the fusion of temporal information and semantic details in the images. In this paper, we propose an attention-guided multi-scale fusion network (AMFNet), which effectively integrates bi-temporal image features and diverse semantics at both the encoding and decoding stages. AMFNet utilizes a unique attention-guided mechanism to dynamically adjust feature fusion, enhancing adaptability and accuracy in change detection tasks. Our method intelligently incorporates temporal information into the deep learning model, considering the temporal dependency inherent in these tasks. We decode based on an interactive feature map, which improves the model’s understanding of evolving patterns over time. Additionally, we introduce multi-level supervised training to facilitate the learning of fused features across multiple scales. In comparison with different algorithms, our proposed method achieves F1 values of 0.9079, 0.8225, and 0.8809 in the LEVIR-CD, GZ-CD, and SYSU-CD datasets, respectively. Our model outperforms the SOTA model, SAGNet, by 0.69% in terms of F1 and 1.15% in terms of IoU on the LEVIR-CD dataset, by 2.8% in terms of F1 and 1.79% in terms of IoU on the GZ-CD dataset, and by 0.54% in terms of F1 and 0.38% in terms of IoU on the SYSU-CD dataset. The method proposed in this study can be applied to various complex scenarios, establishing a change detection method with strong model generalization capabilities.

Attention-based convolutional neural network with multi-modal temporal information fusion for motor imagery EEG decoding

Convolutional neural network (CNN) has been widely applied in motor imagery (MI)-based brain computer interface (BCI) to decode electroencephalography (EEG) signals. However, due to the limited perceptual field of convolutional kernel, CNN only extracts features from local region without considering long-term dependencies for EEG decoding. Apart from long-term dependencies, multi-modal temporal information is equally important for EEG decoding because it can offer a more comprehensive understanding of the temporal dynamics of neural processes. In this paper, we propose a novel deep learning network that combines CNN with self-attention mechanism to encapsulate multi-modal temporal information and global dependencies. The network first extracts multi-modal temporal information from two distinct perspectives: average and variance. A shared self-attention module is then designed to capture global dependencies along these two feature dimensions. We further design a convolutional encoder to explore the relationship between average-pooled and variance-pooled features and fuse them into more discriminative features. Moreover, a data augmentation method called signal segmentation and recombination is proposed to improve the generalization capability of the proposed network. The experimental results on the BCI Competition IV-2a (BCIC-IV-2a) and BCI Competition IV-2b (BCIC-IV-2b) datasets show that our proposed method outperforms the state-of-the-art methods and achieves 4-class average accuracy of 85.03% on the BCIC-IV-2a dataset. The proposed method implies the effectiveness of multi-modal temporal information fusion in attention-based deep learning networks and provides a new perspective for MI-EEG decoding. The code is available at https://github.com/Ma-Xinzhi/EEG-TransNet.

Iterative Temporal-spatial Transformer-based Cardiac T1 Mapping MRI Reconstruction

The precise reconstruction of accelerated magnetic resonance imaging (MRI) brings about notable advantages, such as enhanced diagnostic precision and decreased examination costs. In contrast, traditional cardiac MRI necessitates repetitive acquisitions across multiple heartbeats, resulting in prolonged acquisition times. Significant strides have been made in accelerating MRI through deep learning-based reconstruction methods. However, these existing methods encounter certain limitations: (1) The intricate nature of heart reconstruction involving multiple complex time-series data poses a challenge in exploring nonlinear dependencies between temporal contexts. (2) Existing research often overlooks weight sharing in iterative frameworks, impeding the effective capturing of non-local information and, consequently, limiting improvements in model performance. In order to improve cardiac MRI reconstruction, we propose a novel temporal-spatial transformer with a strategy in this study. Based on the multi-level encoder and decoder transformer architecture, we conduct multi-level spatiotemporal information feature aggregation over several adjacent views, that create nonlinear dependencies among features and efficiently learn important information among adjacent cardiac temporal frames. Additionally, in order to improve contextual awareness between neighboring views, we add cross-view attention for temporal information fusion. Furthermore, we introduce an iterative strategy for training weights during the reconstruction process, which improves feature fusion in critical locations and reduces the number of computations required to calculate global feature dependencies. Extensive experiments have demonstrated the substantial superiority of this procedure over the most advanced techniques, suggesting that it has broad potential for clinical use.

The TSM-net: a new strategy for insulated bearings intelligent faults diagnosis

With the development of power semiconductor devices, pulse width modulation technology is widely used in high-power frequency conversion control motors, which significantly improves the dynamic performance of variable-speed drive system equipment. However, the high-frequency shaft voltage generated during the drive process acts on the bearing to generate high-frequency current. The damage caused by the shaft current sharply shortens the fatigue wear process of the bearing, which in turn leads to premature failure of the bearing. A high insulating ceramic coating is prepared on the outer surface and side face of the inner and outer rings of the bearing by plasma spraying. That is, an insulating protective film is formed on the outer surface of the bearing, which can effectively isolate or reduce the bearing current, prevent the occurrence of electric erosion, and prolong the service life of the variable speed drive system equipment. However, the vibration excitation generated by the variable-speed drive system equipment will cause cracks or fatigue damage to the insulating bearing, resulting in a very complex fault mechanism of the vibration signal. The fault signal characterization lacks a professional signal analysis method, especially the high-reliability, high-precision and long-life high-performance insulating bearing. There is no qualitative formula or characteristic index to explain its failure. To fill this research gap, a new strategy for optimizing the temporal information fusion model and introducing the self-attention mechanism is innovatively developed, and it is named TSM-Net model, and the first attempt is made to realize intelligent identification of insulated bearing faults. Specifically, a multi-channel insulated bearing time information fusion diagnostic model is designed, and the coarse-grained characteristics with timing law are extracted from the measured insulated bearing fault data. Then, the self-attention mechanism is introduced into the designed insulated bearing time information fusion diagnostic model to optimize, and the weight coefficient is continuously updated to calculate the correlation weight between the insulated bearing fault data and the data, so that the final decision of the TSM-Net model is more focused, so as to improve the diagnostic accuracy. Finally, comparing the proposed TSM-Net model with the current five advanced methods, it is found that the proposed TSM-Net model has good diagnostic accuracy for rail transit motor insulated bearing faults, which verifies the effectiveness and superiority of the strategy, and provides a new way for the fault diagnosis of insulated bearings of high-power inverter control motors.

Enhancing Crop Mapping Precision through Multi-Temporal Sentinel-2 Image and Spatial-Temporal Neural Networks in Northern Slopes of Tianshan Mountain

Northern Slopes of Tianshan Mountain (NSTM) in Xinjiang hold significance as a principal agricultural hub within the region’s arid zone. Accurate crop mapping across vast agricultural expanses is fundamental for intelligent crop monitoring and devising sustainable agricultural strategies. Previous studies on multi-temporal crop classification have predominantly focused on single-point pixel temporal features, often neglecting spatial data. In large-scale crop classification tasks, by using spatial information around the pixel, the contextual relationships of the crop can be obtained to reduce possible noise interference. This research introduces a multi-scale, multi-temporal classification framework centered on ConvGRU (convolutional gated recurrent unit). By leveraging the attention mechanism of the Strip Pooling Module (SPM), a multi-scale spatial feature extraction module has been designed. This module accentuates vital spatial and spectral features, enhancing the clarity of crop edges and reducing misclassifications. The temporal information fusion module integration features from various periods to bolster classification precision. Using Sentinel-2 imagery spanning May to October 2022, datasets for cotton, corn, and winter wheat of the NSTM were generated for the framework’s training and validation. The results demonstrate an impressive 93.03% accuracy for 10 m resolution crop mapping using 15-day interval, 12-band Sentinel-2 data for the three crops. This method outperforms other mainstream methods like Random Forest (RF), Long Short-Term Memory (LSTM), Transformer, and Temporal Convolutional Neural Network (TempCNN), showcasing a kappa coefficient of 0.9062, 7.52% and 2.42% improvement in Overall Accuracy compared to RF and LSTM, respectively, which demonstrate the potential of our model for large-scale crop classification tasks to enable high-resolution crop mapping on the NSTM.

A Novel 1D-Convolutional Spatial-Time Fusion Strategy for Data-Driven Fault Diagnosis of Aero-Hydraulic Pipeline Systems

Intelligent diagnosis of faults in an aero-hydraulic pipeline is important for condition monitoring of its systems. However, there are no more qualitative formulas or feature indicators to describe the faults of aero-hydraulic pipelines because of the complexity and diversity of aero-hydraulic pipeline systems, which leads to a very complex pipeline fault mechanism. In addition, although it is well known that the expression of interpretable and representable pipeline intelligent diagnosis models with pipeline fault characteristics are buried in high background noise and strong noise disturbance conditions in practical industrial scenarios, this has yet to be discussed. Inspired by the demand, this paper proposes a novel diagnosis strategy: the 1D-convolutional space-time fusion strategy for aero-engine hydraulic pipelines. Firstly, by optimizing the convolutional neural network and using it to design a one-dimensional convolutional neural network (1DCNN) with a wide input scale to expand the input field of perception, thereby obtaining more comprehensive spatial information of the pipeline data, which can effectively extract richer short sequence features. Secondly, a network of bidirectional gated recurrent Unit (Bi-GRU) is proposed, which integrates a short sequence of high-dimensional features for temporal information fusion, resulting in a certain degree of avoiding memory loss and gradient dispersion caused by the too-large step size. It is demonstrated that, for the noise signal and variable pressure signal, the fault identification accuracy approximated 95.9%, proving the proposed strategy’s robustness. By comparing with the other five methods, the proposed strategy has the ability to identify 10 different fault states in the aero-hydraulic pipeline with higher accuracy.

Multiple hypotheses based motion compensation for learned video compression

Recently, learned video compression has attracted copious research attention. However, among the existing methods, the motion used for alignment is limited to one hypothesis only, leading to inaccurate motion estimation, especially for the complicated scenes with complex movements. Motivated by multiple hypotheses philosophy in traditional video compression, we develop the multiple hypotheses based motion compensation for the learned video compression, in an effort to enhance the motion compensation efficiency by providing diverse hypotheses with efficient temporal information fusion. In particular, the multiple hypotheses module which produces multiple motions and warped features for mining sufficient temporal information, is proposed to provide various hypotheses inferences from the reference frame. To utilize these hypotheses more copiously, the hypotheses attention module is adopted by introducing the channel-wised squeeze-and-excitation layer and the multi-scale network. In addition, the context combination is employed to fuse the weighted hypotheses to generate effective contexts with powerful temporal priors. Finally, the valid contexts are used for promoting the compression efficiency by merging weighted warped features. Extensive experiments show that the proposed method can significantly improve the rate-distortion performance of learned video compression. Compared with the state-of-the-art method for end-to-end video compression, over 13% bit rate reductions on average in terms of PSNR and MS-SSIM can be achieved.

Stratification of Children with Autism Spectrum Disorder Through Fusion of Temporal Information in Eye-gaze Scan-Paths

Background: Looking pattern differences are shown to separate individuals with Autism Spectrum Disorder (ASD) and Typically Developing (TD) controls. Recent studies have shown that, in children with ASD, these patterns change with intellectual and social impairments, suggesting that patterns of social attention provide indices of clinically meaningful variation in ASD. Method: We conducted a naturalistic study of children with ASD (n = 55) and typical development (TD, n = 32). A battery of eye-tracking video stimuli was used in the study, including Activity Monitoring (AM), Social Referencing (SR), Theory of Mind (ToM), and Dyadic Bid (DB) tasks. This work reports on the feasibility of spatial and spatiotemporal scanpaths generated from eye-gaze patterns of these paradigms in stratifying ASD and TD groups. Algorithm: This article presents an approach for automatically identifying clinically meaningful information contained within the raw eye-tracking data of children with ASD and TD. The proposed mechanism utilizes combinations of eye-gaze scan-paths (spatial information), fused with temporal information and pupil velocity data and Convolutional Neural Network (CNN) for stratification of diagnosis (ASD or TD). Results: Spatial eye-gaze representations in the form of scanpaths in stratifying ASD and TD (ASD vs. TD: DNN: 74.4%) are feasible. These spatial eye-gaze features, e.g., scan-paths, are shown to be sensitive to factors mediating heterogeneity in ASD: age (ASD: 2–4 y/old vs. 10–17 y/old CNN: 80.5%), gender (Male vs. Female ASD: DNN: 78.0%) and the mixture of age and gender (5–9 y/old Male vs. 5–9 y/old Female ASD: DNN:98.8%). Limiting scan-path representations temporally increased variance in stratification performance, attesting to the importance of the temporal dimension of eye-gaze data. Spatio-Temporal scan-paths that incorporate velocity of eye movement in their images of eye-gaze are shown to outperform other feature representation methods achieving classification accuracy of 80.25%. Conclusion: The results indicate the feasibility of scan-path images to stratify ASD and TD diagnosis in children of varying ages and gender. Infusion of temporal information and velocity data improves the classification performance of our deep learning models. Such novel velocity fused spatio-temporal scan-path features are shown to be able to capture eye gaze patterns that reflect age, gender, and the mixed effect of age and gender, factors that are associated with heterogeneity in ASD and difficulty in identifying robust biomarkers for ASD.

Spatial and temporal information fusion for human action recognition via Center Boundary Balancing Multimodal Classifier

This paper proposes a novel multimodal data classifier named Center Boundary Balancing Multimodal Classifier (CBBMC) to fuse and classify the spatial and temporal descriptors for recognizing human actions from depth video sequences. CBBMC is a composite algorithm integrating feature fusion and feature classification, in which Center Boundary Balancing Projection (CBBP) is used to balance the center and boundary information of feature class spaces. In order to solve the problem of multimodal information redundancy and isolation, two feature selection and fusion schemes of CBBMC based on embedded feature selection are presented. Moreover, two new action descriptors called Gaussian Pyramid Depth Motion Images (GP-DMI) and Depth Temporal Maps (DTM) are introduced to capture the multi-scale spatial and fine-grained temporal information of human activities. Finally, we present an effective spatial and temporal information fusion framework based on CBBMC for human action recognition. In order to evaluate the performance of the proposed approach, extensive experiments are conducted. The proposed method achieved impressive results on four benchmark datasets, namely MSR Action3D (96.33%), UTD-MHAD (94.41%), DHA (95.65%), and NTU RGB+D (83.31% cross-subject and 87.66% cross-view), even though only the depth modality was used. The experimental results demonstrate the effectiveness of our method.

Deep Learning-Based Data Augmentation and Model Fusion for Automatic Arrhythmia Identification and Classification Algorithms.

Automated ECG-based arrhythmia detection is critical for early cardiac disease prevention and diagnosis. Recently, deep learning algorithms have been widely applied for arrhythmia detection with great success. However, the lack of labeled ECG data and low classification accuracy can have a significant impact on the overall effectiveness of a classification algorithm. In order to better apply deep learning methods to arrhythmia classification, in this study, feature extraction and classification strategy based on generative adversarial network data augmentation and model fusion are proposed to address these problems. First, the arrhythmia sparse data is augmented by generative adversarial networks. Then, aiming at the identification of different types of arrhythmias in long-term ECG, a spatial information fusion model based on ResNet and a temporal information fusion model based on BiLSTM are proposed. The model effectively fuses the location information of the nearest neighbors through the local feature extraction part of the generated ECG feature map and obtains the correlation of the global features by autonomous learning in multiple spaces through the BiLSTM network in the part of the global feature extraction. In addition, an attention mechanism is introduced to enhance the features of arrhythmia-type signal segments, and this mechanism can effectively focus on the extraction of key information to form a feature vector for final classification. Finally, it is validated by the enhanced MIT-BIH arrhythmia database. The experimental results demonstrate that the proposed classification technique enhances arrhythmia diagnostic accuracy by 99.4%, and the algorithm has high recognition performance and clinical value.

TERQA: Question answering over knowledge graph considering precise dependencies of temporal information on vectors

Time questions involve explicit and implicit constraints as well as complex time interval interactions, making them critical criteria for measuring the effect of knowledge base question answering. Although attention to temporal questions has spurred the development of temporal knowledge graphs, existing studies have focused on the simple splicing and fusion of temporal information with question or knowledge base embeddings, losing sight of the hidden interaction features between temporal information and embedded vectors. In this paper, we proposed TERQA, a temporal knowledge base question-answering approach to explore precise spatial dependencies between temporal information and embedded vectors. The exploration of the deep dependency between time and embedded vectors was divided into two stages. In the first stage, the Transformer model of depth extraction was employed to extract richer features from questions and the representation was enhanced with temporal information; in the second stage, high-level capsules were adopted to extract the low-level vector features for detailed pose determination, allowing a more precise deep dependency of temporal facts on embedded vectors. We conducted an experiment using two temporal question answering datasets, TempQuestions and CronQuestions, and the results showed that accuracy for TERQA improved 11.3% from baseline on the dataset TempQuestions with higher annotated information. Additionally, the adapted TERQA also showed varying degrees of improvements over the baseline in the larger but simply annotated dataset CronQuestions.

Temporal Group Fusion Network for Deep Video Inpainting

Video inpainting is a task of synthesizing spatio-temporal coherent content in missing regions of the given video sequence, which has recently drawn increasing attention. To utilize the temporal information across frames, most recent deep learning-based methods align reference frames to target frame firstly with explicit or implicit motion estimation and then integrate the information from the aligned frames. However, their performance relies heavily on the accuracy of frame-to-frame alignment. To alleviate the above problem, in this paper, a novel Temporal Group Fusion Network (TGF-Net) is proposed to effectively integrate temporal information through a two-stage fusion strategy. Specifically, the input frames are reorganized into different groups, where each group is followed by an intra-group fusion module to integrate information within the group. Different groups provide complementary information for the missing region. A temporal attention model is further designed to adaptively integrate the information across groups. Such a temporal information fusion way gets rid of the dependence on alignment operations, greatly improving the visual quality and temporal consistency of the inpainted results. In addition, a coarse alignment model is introduced at the beginning of the network to handle videos with large motion. Extensive experiments on DAVIS and Youtube-VOS datasets demonstrate the superiority of our proposed method in terms of PSNR/SSIM values, visual quality and temporal consistency, respectively.

Attention Mechanism Based on Improved Spatial-Temporal Convolutional Neural Networks for Traffic Police Gesture Recognition

Human action recognition has attracted extensive research efforts in recent years, in which traffic police gesture recognition is important for self-driving vehicles. One of the crucial challenges in this task is how to find a representation method based on spatial-temporal features. However, existing methods performed poorly in spatial and temporal information fusion, and how to extract features of traffic police gestures has not been well researched. This paper proposes an attention mechanism based on the improved spatial-temporal convolutional neural network (AMSTCNN) for traffic police gesture recognition. This method focuses on the action part of traffic police and uses the correlation between spatial and temporal features to recognize traffic police gestures, so as to ensure that traffic police gesture information is not lost. Specifically, AMSTCNN integrates spatial and temporal information, uses weight matching to pay more attention to the region where human action occurs, and extracts region proposals of the image. Finally, we use Softmax to classify actions after spatial-temporal feature fusion. AMSTCNN can strongly make use of the spatial-temporal information of videos and select effective features to reduce computation. Experiments on AVA and the Chinese traffic police gesture datasets show that our method is superior to several state-of-the-art methods.

Weakly Supervised Video Object Segmentation via Dual-attention Cross-branch Fusion

Recently, concerning the challenge of collecting large-scale explicitly annotated videos, weakly supervised video object segmentation (WSVOS) using video tags has attracted much attention. Existing WSVOS approaches follow a general pipeline including two phases, i.e., a pseudo masks generation phase and a refinement phase. To explore the intrinsic property and correlation buried in the video frames, most of them focus on the later phase by introducing optical flow as temporal information to provide more supervision. However, these optical flow-based studies are greatly affected by illumination and distortion and lack consideration of the discriminative capacity of multi-level deep features. In this article, with the goal of capturing more effective temporal information and investigating a temporal information fusion strategy accordingly, we propose a unified WSVOS model by adopting a two-branch architecture with a multi-level cross-branch fusion strategy, named as dual-attention cross-branch fusion network (DACF-Net). Concretely, the two branches of DACF-Net, i.e., a temporal prediction subnetwork (TPN) and a spatial segmentation subnetwork (SSN), are used for extracting temporal information and generating predicted segmentation masks, respectively. To perform the cross-branch fusion between TPN and SSN, we propose a dual-attention fusion module that can be plugged into the SSN flexibly. We also pose a cross-frame coherence loss (CFCL) to achieve smooth segmentation results by exploiting the coherence of masks produced by TPN and SSN. Extensive experiments demonstrate the effectiveness of proposed approach compared with the state-of-the-arts on two challenging datasets, i.e., Davis-2016 and YouTube-Objects.

Satellite Video Super-Resolution via Multiscale Deformable Convolution Alignment and Temporal Grouping Projection

As a new earth observation tool, satellite video has been widely used in remote-sensing field for dynamic analysis. Video super-resolution (VSR) technique has thus attracted increasing attention due to its improvement to spatial resolution of satellite video. However, the difficulty of remote-sensing image alignment and the low efficiency of spatial–temporal information fusion make poor generalization of the conventional VSR methods applied to satellite videos. In this article, a novel fusion strategy of temporal grouping projection and an accurate alignment module are proposed for satellite VSR. First, we propose a deformable convolution alignment module with a multiscale residual block to alleviate the alignment difficulties caused by scarce motion and various scales of moving objects in remote-sensing images. Second, a temporal grouping projection fusion strategy is proposed, which can reduce the complexity of projection and make the spatial features of reference frames play a continuous guiding role in spatial–temporal information fusion. Finally, a temporal attention module is designed to adaptively learn the different contributions of temporal information extracted from each group. Extensive experiments on Jilin-1 satellite video demonstrate that our method is superior to current state-of-the-art VSR methods.

Spatial–temporal analysis of urban ecological comfort index derived from remote sensing data: a case study of Hefei, China

In recent years, ecological and environmental degradations have occurred widely around the world, especially in urban areas with rapid economic development, intensive human activities, and massive urbanization. We aim at proposing an effective and comprehensive urban ecological environment evaluation method based on remote sensing. Harmonic analysis of time series method and spatial–temporal information fusion based on nonlocal means filter was used to improve the quality of long-term remote sensing data from 2003 to 2019. The greenness, dryness, heat, and atmospheric turbidity of the ecological environment were represented by normalized difference vegetation index, normalized difference built-up index, land surface temperature, and aerosol optical depth, respectively. An urban ecological comfort index (UECI) was established by the weighted combination of the above four indicators using entropy method. UECI was calculated at seasonal and annual scales from 2003 to 2019 in Hefei city, China, as a case study. Furthermore, Mann–Kendall trend test, spatial autocorrelation analysis, and spatial clustering analysis were performed on the UECI results. The experimental results show that UECI in Hefei had an increasing trend from 0.5446 to 0.5737 between 2003 and 2019, indicating that its ecological environment is actively improving. UECI is roughly equal in spring, summer, and autumn, and the worst is in winter. The responses of UECI to urban expansion, environmental protection policies, and extreme weather were also explored. Its sensitively analysis results and the fact that the spatial–temporal trends of UECI were consistent with the influencing factors reflected the stability and reliability of UECI, indicating that the proposed UECI performed well in Hefei and could be applied to measure urban ecological comfort level in other study areas.

Application of novel hybrid deep leaning model for cleaner production in a paper industrial wastewater treatment system

Developing monitoring system for paper industrial wastewater treatment system is an important route for wastewater reuse and recycling from wastewater, which are regarded as effective way for cleaner production. A novel hybrid deep leaning CLSTMA model, which based on sequential fusion convolutional neural network (CNN), long short term memory (LSTM) and attention mechanism (AM), was developed to monitor the water quality in a full-scale paper industrial wastewater treatment system for energy conservation and emissions reduction. The hybrid CLSTMA model for predicting water quality of paper industrial wastewater treatment system was divided into three steps: spatial information fusion by using CNN module, temporal information fusion by using LSTM module and variable weighted calculation by using AM module. Compare with other models (CNN, LSTM and CLSTM models), RMSE of CLSTMA model for the effluent chemical oxygen demand (CODeff) reduced by 23.3–31.55%, MAE of CLSTMA model reduced by 38.89–74.50%, R of CLSTMA model increased by 8.29–11.86%. For the effluent suspended solids (SSeff), compared with CNN and LSTM models, RMSE of CLSTMA model reduced by 10.26% and 9.92%, MAE of CLSTMA model reduced by 5.37% and 3.44%, R of CLSTMA model increased by 15.13% and 37.21%, respectively. While, R of CLSTMA was consistent with CLSTM model, but RMSE and MAE of CLSTMA model reduced by 16.07% and 7.49% than the CLSTM model. Simulation results demonstrate that the proposed CLSTMA model has a great potential in monitoring paper industrial wastewater treatment system for cleaner production.

Video anomaly detection with multi-scale feature and temporal information fusion

Video anomaly detection is a challenging task because of the uncertainty of abnormal events. The current method based on predictive frames has obtained better detection results compared with the previous reconstruction or hand-crafted methods. In current prediction methods, the characteristics considered previously are only of a single scale, and the time constraint information is not fully used. In our work, we proposed a new framework structure to achieve better abnormality detection rate. To address the objects of different scales in each video frame, we considered extracting the characteristics of different receptive fields to encode more spatial information. At the same time, we added temporal constraints to the network instead of using time-consuming optical flow information, and we completed the memory of temporal features through a ConvGRU module. Furthermore, while distinguishing abnormal events, we also considered temporal information and spatial information so that our framework could fully combine spatio-temporal information to correctly distinguish abnormal events from normal events. We obtained excellent results on three datasets, thus demonstrating the effectiveness of our method.

Self-adaptive combination method for temporal evidence based on negotiation strategy

In temporal information fusion, the information collected by sensors is obtained dynamically with the passage of time. Unlike the spatial information fusion, temporal fusion should be dynamic. Evidence theory has been applied to fuse temporal and spatial information; however, existing temporal fusion methods do not treat conflicting and non-conflicting evidence sources distinctively. Moreover, unlike spatial evidence sources, which are obtained simultaneously, temporal evidence sources cannot be evaluated simultaneously to obtain their degree of reliability. Thus, it is necessary to develop a method for temporal evidence combination. In this paper, a self-adaptive combination method for temporal evidence is proposed based on the negotiation strategy. In the proposed method, a set called an evidence set is constructed by the cumulative temporal fusion results of the previous moment, current moment, and future moment. The evidence set is evaluated as conflicting or non-conflicting according to the maximum power pignistic probability distance between each pair of evidence sources in the set. Then, temporal evidence sources are self-adaptively combined by different methods according to the degree of conflict. Numerical experiments were conducted to evaluate the performance of the proposed method. The results indicated that the proposed method is sufficiently effective and robust to support decision making.