Fusion Of Temporal Information Research Articles

Weakly Supervised Video Object Segmentation via Dual-attention Cross-branch Fusion

Recently, concerning the challenge of collecting large-scale explicitly annotated videos, weakly supervised video object segmentation (WSVOS) using video tags has attracted much attention. Existing WSVOS approaches follow a general pipeline including two phases, i.e., a pseudo masks generation phase and a refinement phase. To explore the intrinsic property and correlation buried in the video frames, most of them focus on the later phase by introducing optical flow as temporal information to provide more supervision. However, these optical flow-based studies are greatly affected by illumination and distortion and lack consideration of the discriminative capacity of multi-level deep features. In this article, with the goal of capturing more effective temporal information and investigating a temporal information fusion strategy accordingly, we propose a unified WSVOS model by adopting a two-branch architecture with a multi-level cross-branch fusion strategy, named as dual-attention cross-branch fusion network (DACF-Net). Concretely, the two branches of DACF-Net, i.e., a temporal prediction subnetwork (TPN) and a spatial segmentation subnetwork (SSN), are used for extracting temporal information and generating predicted segmentation masks, respectively. To perform the cross-branch fusion between TPN and SSN, we propose a dual-attention fusion module that can be plugged into the SSN flexibly. We also pose a cross-frame coherence loss (CFCL) to achieve smooth segmentation results by exploiting the coherence of masks produced by TPN and SSN. Extensive experiments demonstrate the effectiveness of proposed approach compared with the state-of-the-arts on two challenging datasets, i.e., Davis-2016 and YouTube-Objects.

Satellite Video Super-Resolution via Multiscale Deformable Convolution Alignment and Temporal Grouping Projection

As a new earth observation tool, satellite video has been widely used in remote-sensing field for dynamic analysis. Video super-resolution (VSR) technique has thus attracted increasing attention due to its improvement to spatial resolution of satellite video. However, the difficulty of remote-sensing image alignment and the low efficiency of spatial–temporal information fusion make poor generalization of the conventional VSR methods applied to satellite videos. In this article, a novel fusion strategy of temporal grouping projection and an accurate alignment module are proposed for satellite VSR. First, we propose a deformable convolution alignment module with a multiscale residual block to alleviate the alignment difficulties caused by scarce motion and various scales of moving objects in remote-sensing images. Second, a temporal grouping projection fusion strategy is proposed, which can reduce the complexity of projection and make the spatial features of reference frames play a continuous guiding role in spatial–temporal information fusion. Finally, a temporal attention module is designed to adaptively learn the different contributions of temporal information extracted from each group. Extensive experiments on Jilin-1 satellite video demonstrate that our method is superior to current state-of-the-art VSR methods.

Spatial–temporal analysis of urban ecological comfort index derived from remote sensing data: a case study of Hefei, China

In recent years, ecological and environmental degradations have occurred widely around the world, especially in urban areas with rapid economic development, intensive human activities, and massive urbanization. We aim at proposing an effective and comprehensive urban ecological environment evaluation method based on remote sensing. Harmonic analysis of time series method and spatial–temporal information fusion based on nonlocal means filter was used to improve the quality of long-term remote sensing data from 2003 to 2019. The greenness, dryness, heat, and atmospheric turbidity of the ecological environment were represented by normalized difference vegetation index, normalized difference built-up index, land surface temperature, and aerosol optical depth, respectively. An urban ecological comfort index (UECI) was established by the weighted combination of the above four indicators using entropy method. UECI was calculated at seasonal and annual scales from 2003 to 2019 in Hefei city, China, as a case study. Furthermore, Mann–Kendall trend test, spatial autocorrelation analysis, and spatial clustering analysis were performed on the UECI results. The experimental results show that UECI in Hefei had an increasing trend from 0.5446 to 0.5737 between 2003 and 2019, indicating that its ecological environment is actively improving. UECI is roughly equal in spring, summer, and autumn, and the worst is in winter. The responses of UECI to urban expansion, environmental protection policies, and extreme weather were also explored. Its sensitively analysis results and the fact that the spatial–temporal trends of UECI were consistent with the influencing factors reflected the stability and reliability of UECI, indicating that the proposed UECI performed well in Hefei and could be applied to measure urban ecological comfort level in other study areas.

Application of novel hybrid deep leaning model for cleaner production in a paper industrial wastewater treatment system

Developing monitoring system for paper industrial wastewater treatment system is an important route for wastewater reuse and recycling from wastewater, which are regarded as effective way for cleaner production. A novel hybrid deep leaning CLSTMA model, which based on sequential fusion convolutional neural network (CNN), long short term memory (LSTM) and attention mechanism (AM), was developed to monitor the water quality in a full-scale paper industrial wastewater treatment system for energy conservation and emissions reduction. The hybrid CLSTMA model for predicting water quality of paper industrial wastewater treatment system was divided into three steps: spatial information fusion by using CNN module, temporal information fusion by using LSTM module and variable weighted calculation by using AM module. Compare with other models (CNN, LSTM and CLSTM models), RMSE of CLSTMA model for the effluent chemical oxygen demand (CODeff) reduced by 23.3–31.55%, MAE of CLSTMA model reduced by 38.89–74.50%, R of CLSTMA model increased by 8.29–11.86%. For the effluent suspended solids (SSeff), compared with CNN and LSTM models, RMSE of CLSTMA model reduced by 10.26% and 9.92%, MAE of CLSTMA model reduced by 5.37% and 3.44%, R of CLSTMA model increased by 15.13% and 37.21%, respectively. While, R of CLSTMA was consistent with CLSTM model, but RMSE and MAE of CLSTMA model reduced by 16.07% and 7.49% than the CLSTM model. Simulation results demonstrate that the proposed CLSTMA model has a great potential in monitoring paper industrial wastewater treatment system for cleaner production.

Video anomaly detection with multi-scale feature and temporal information fusion

Video anomaly detection is a challenging task because of the uncertainty of abnormal events. The current method based on predictive frames has obtained better detection results compared with the previous reconstruction or hand-crafted methods. In current prediction methods, the characteristics considered previously are only of a single scale, and the time constraint information is not fully used. In our work, we proposed a new framework structure to achieve better abnormality detection rate. To address the objects of different scales in each video frame, we considered extracting the characteristics of different receptive fields to encode more spatial information. At the same time, we added temporal constraints to the network instead of using time-consuming optical flow information, and we completed the memory of temporal features through a ConvGRU module. Furthermore, while distinguishing abnormal events, we also considered temporal information and spatial information so that our framework could fully combine spatio-temporal information to correctly distinguish abnormal events from normal events. We obtained excellent results on three datasets, thus demonstrating the effectiveness of our method.

Self-adaptive combination method for temporal evidence based on negotiation strategy

In temporal information fusion, the information collected by sensors is obtained dynamically with the passage of time. Unlike the spatial information fusion, temporal fusion should be dynamic. Evidence theory has been applied to fuse temporal and spatial information; however, existing temporal fusion methods do not treat conflicting and non-conflicting evidence sources distinctively. Moreover, unlike spatial evidence sources, which are obtained simultaneously, temporal evidence sources cannot be evaluated simultaneously to obtain their degree of reliability. Thus, it is necessary to develop a method for temporal evidence combination. In this paper, a self-adaptive combination method for temporal evidence is proposed based on the negotiation strategy. In the proposed method, a set called an evidence set is constructed by the cumulative temporal fusion results of the previous moment, current moment, and future moment. The evidence set is evaluated as conflicting or non-conflicting according to the maximum power pignistic probability distance between each pair of evidence sources in the set. Then, temporal evidence sources are self-adaptively combined by different methods according to the degree of conflict. Numerical experiments were conducted to evaluate the performance of the proposed method. The results indicated that the proposed method is sufficiently effective and robust to support decision making.

Deep Reinforcement Learning-Based Progressive Sequence Saliency Discovery Network for Mitosis Detection In Time-Lapse Phase-Contrast Microscopy Images.

Mitosis detection plays an important role in the analysis of cell status and behavior and is therefore widely utilized in many biological research and medical applications. In this article, we propose a deep reinforcement learning-based progressive sequence saliency discovery network (PSSD)for mitosis detection in time-lapse phase contrast microscopy images. By discovering the salient frames when cell state changes in the sequence, PSSD can more effectively model the mitosis process for mitosis detection. We formulate the discovery of salient frames as a Markov Decision Process (MDP)that progressively adjusts the selection positions of salient frames in the sequence, and further leverage deep reinforcement learning to learn the policy in the salient frame discovery process. The proposed method consists of two parts: 1)the saliency discovery module that selects the salient frames from the input cell image sequence by progressively adjusting the selection positions of salient frames; 2)the mitosis identification module that takes a sequence of salient frames and performs temporal information fusion for mitotic sequence classification. Since the policy network of the saliency discovery module is trained under the guidance of the mitosis identification module, PSSD can comprehensively explore the salient frames that are beneficial for mitosis detection. To our knowledge, this is the first work to implement deep reinforcement learning to the mitosis detection problem. In the experiment, we evaluate the proposed method on the largest mitosis detection dataset, C2C12-16. Experiment results show that compared with the state-of-the-arts, the proposed method can achieve significant improvement for both mitosis identification and temporal localization on C2C12-16.

Evidence reasoning for temporal uncertain information based on relative reliability evaluation

As an important tool for information fusion, evidence theory has been widely applied in many areas. Unlike spatial evidence combination, temporal evidence combination is sequential and dynamic, which calls for new temporal evidence combination rule. In this paper, temporal evidence is combined based on relative reliability evaluation and evidence discounting. We first recall the method of evidence reliability evaluation based on intuitionistic fuzzy multiple criteria decision making (ERE-IFMCDM). Based on ERE-IFMCDM method, relative reliability factors of evidence sources in neighboring time nodes can be obtained. Then, according to evidence discounting and Dempster's combination rule, a method for temporal evidence combination based on the relative reliability factor is developed. Numerical examples and simulation demonstrate that the proposed method is time sensitive, which can reflect the dynamic feature of temporal information fusion. Moreover, the proposed method can deal with conflict in temporal fusion much better, which can enhance the anti-interference performance of the fusion system. The fusion framework proposed in this paper provides a dynamic perspective of the fusion of temporal uncertain information. It is helpful for the structural design of intelligent fusion system based on multiple temporal-spatial information.

Underwater video dehazing based on spatial–temporal information fusion

In this paper, a novel multidimensional underwater video dehazing method is presented to restore and enhance the underwater degraded videos. Videos in the underwater suffer from medium scattering and light absorption. The absorption of light traveling in the water makes the underwater hazing videos different from the atmosphere hazing videos. In order to dehaze the underwater videos, a spatial---temporal information fusion method is proposed which includes two main parts. One is transmission estimation, which is based on the correlation between the adjacent frames of videos to keep the color consistency, where fast tracking and the least square method are used to reduce the influence of camera and object motions and water flowing. Another part is background light estimation to keep consistent atmospheric light values in a video. Extensive experimental results demonstrate that the proposed algorithm can have superior haze removing and color balancing capabilities.