Temporal Context Information Research Articles

A Novel Crop Classification Method Based on the Tensor-GCN for Time-Series PolSAR Data

Time-series polarimetric synthetic aperture radar (PolSAR) has been proven to be an effective technique for crop classification and agricultural activity monitoring. However, the characterization and utilization of time-series PolSAR data by existing methods are still inadequate. They are unable to extract and utilize time-varying features, which can describe the dynamic changes of crop polarimetric information. In this paper, we propose a tensor form to comprehensively describe the information of time-series PolSAR data, including spatial context information, polarimetric scattering information, and temporal context information. And we define a novel similarity value for the tensors (TSV), which can simultaneously consider distance and shape similarity of tensors. Then, we construct a tensor-based graph representation to capture the global similarity information of time-series PolSAR data. Finally, we propose a tensor-based graph convolutional network (Tensor-GCN) to extract deep features of graph node tensors for crop classification. Experimental results and analysis on two time-series PolSAR data firmly demonstrate the superiority of the proposed Tensor-GCN to other state-of-the-art methods.

End-to-End Temporal Action Detection With Transformer.

Temporal action detection (TAD) aims to determine the semantic label and the temporal interval of every action instance in an untrimmed video. It is a fundamental and challenging task in video understanding. Previous methods tackle this task with complicated pipelines. They often need to train multiple networks and involve hand-designed operations, such as non-maximal suppression and anchor generation, which limit the flexibility and prevent end-to-end learning. In this paper, we propose an end-to-end Transformer-based method for TAD, termed TadTR. Given a small set of learnable embeddings called action queries, TadTR adaptively extracts temporal context information from the video for each query and directly predicts action instances with the context. To adapt Transformer to TAD, we propose three improvements to enhance its locality awareness. The core is a temporal deformable attention module that selectively attends to a sparse set of key snippets in a video. A segment refinement mechanism and an actionness regression head are designed to refine the boundaries and confidence of the predicted instances, respectively. With such a simple pipeline, TadTR requires lower computation cost than previous detectors, while preserving remarkable performance. As a self-contained detector, it achieves state-of-the-art performance on THUMOS14 (56.7% mAP) and HACS Segments (32.09% mAP). Combined with an extra action classifier, it obtains 36.75% mAP on ActivityNet-1.3. Code is available at https://github.com/xlliu7/TadTR.

MSFF-Net: Multi-scale feature fusing networks with dilated mixed convolution and cascaded parallel framework for sound event detection

Sound Event Detection (SED) needs to identify the sound events in a recording and detect the onset and offset times of them. The former desires features with long short-term dependencies to detect sound events with different durations and the latter needs fine-grained dependency. Although our previous proposed Multi-Scale Fully Convolutional Networks (MS-FCN) uses cascaded dilated convolution to model temporal context information and multi-scale information is considered, there are two shortages to deal with: the ignorance of neighboring information and fine-grained dependencies, and neglecting intermediate-length temporal dependencies. The first shortage is caused by the skipping elements sampling mechanism of dilated convolution, by which the neighboring information and fine-grained dependencies are ignored. To overcome this shortage, the paper proposes the dilated mixed convolution module, which mixes dilated convolution and standard convolutions to capture both the fine-grained and long-term dependencies and give weight to neighboring information. The second shortage is caused by the too fast increase of temporal dependent length in cascaded dilated convolution module, which causes too much intermediate temporal information to be ignored. For this shortage, this paper proposes Dilated Temporal Pyramid Pooling module (DTPP), in which parallel dilated convolutions with multiple dilation factors are used to capture the intermediate temporal information with a proper temporal dependent length. As cascaded module has been demonstrated to be valid and efficient to model the temporal context in MS-FCN and DTPP module can capture the ignored temporal information of cascaded module, taking the advantages of both, this paper proposes the cascaded parallel module to capture richer temporal dependencies. Based on that, Multi-Scale Feature Fusion Networks (MSFF-Net) is proposed, which obtains competitive performance on three open datasets.

Transformer-based CNNs: Mining Temporal Context Information for Multi-sound COVID-19 Diagnosis.

Due to the COronaVIrus Disease 2019 (COVID-19) pandemic, early screening of COVID-19 is essential to prevent its transmission. Detecting COVID-19 with computer audition techniques has in recent studies shown the potential to achieve a fast, cheap, and ecologically friendly diagnosis. Respiratory sounds and speech may contain rich and complementary information about COVID-19 clinical conditions. Therefore, we propose training three deep neural networks on three types of sounds (breathing/counting/vowel) and assembling these models to improve the performance. More specifically, we employ Convolutional Neural Networks (CNNs) to extract spatial representations from log Mel spectrograms and a multi-head attention mechanism in the transformer to mine temporal context information from the CNNs' outputs. The experimental results demonstrate that the transformer-based CNNs can effectively detect COVID-19 on the DiCOVA Track-2 database (AUC: 70.0%) and outperform simple CNNs and hybrid CNN-RNNs.

Video Object Detection Using Event-Aware Convolutional Lstm and Object Relation Networks

Common video-based object detectors exploit temporal contextual information to improve the performance of object detection. However, detecting objects under challenging conditions has not been thoroughly studied yet. In this paper, we focus on improving the detection performance for challenging events such as aspect ratio change, occlusion, or large motion. To this end, we propose a video object detection network using event-aware ConvLSTM and object relation networks. Our proposed event-aware ConvLSTM is able to highlight the area where those challenging events take place. Compared with traditional ConvLSTM, with the proposed method it is easier to exploit temporal contextual information to support video-based object detectors under challenging events. To further improve the detection performance, an object relation module using supporting frame selection is applied to enhance the pooled features for target ROI. It effectively selects the features of the same object from one of the reference frames rather than all of them. Experimental results on ImageNet VID dataset show that the proposed method achieves mAP of 81.0% without any post processing and can handle challenging events efficiently in video object detection.

Multiscale and Multitemporal Road Detection from High Resolution SAR Images Using Attention Mechanism

Road detection from images has emerged as an important way to obtain road information, thereby gaining much attention in recent years. However, most existing methods only focus on extracting road information from single temporal intensity images, which may cause a decrease in image resolution due to the use of spatial filter methods to avoid coherent speckle noises. Some newly developed methods take into account the multi-temporal information in the preprocessing stage to filter the coherent speckle noise in the SAR imagery. They ignore the temporal characteristic of road objects such as the temporal consistency for the road objects in the multitemporal SAR images that cover the same area and are taken at adjacent times, causing the limitation in detection performance. In this paper, we propose a multiscale and multitemporal network (MSMTHRNet) for road detection from SAR imagery, which contains the temporal consistency enhancement module (TCEM) and multiscale fusion module (MSFM) that are based on attention mechanism. In particular, we propose the TCEM to make full use of multitemporal information, which contains temporal attention submodule that applies attention mechanism to capture temporal contextual information. We enforce temporal consistency constraint by the TCEM to obtain the enhanced feature representations of SAR imagery that help to distinguish the real roads. Since the width of roads are various, incorporating multiscale features is a promising way to improve the results of road detection. We propose the MSFM that applies learned weights to combine predictions of different scale features. Since there is no public dataset, we build a multitemporal road detection dataset to evaluate our methods. State-of-the-art semantic segmentation network HRNetV2 is used as a baseline method to compare with MSHRNet that only has MSFM and the MSMTHRNet. The MSHRNet(TAF) whose input is the SAR image after the temporal filter is adopted to compare with our proposed MSMTHRNet. On our test dataset, MSHRNet and MSMTHRNet improve over the HRNetV2 by 2.1% and 14.19%, respectively, in the IoU metric and by 3.25% and 17.08%, respectively, in the APLS metric. MSMTHRNet improves over the MSMTHRNet(TAF) by 8.23% and 8.81% in the IoU metric and APLS metric, respectively.

Phase sensitive masking-based single channel speech enhancement using conditional generative adversarial network

We propose PSMGAN, an efficient phase sensitive masking-based single-channel speech enhancement technique using a conditional generative adversarial network (cGAN). The time–frequency (T-F) masking-based speech enhancement approaches through deep neural networks (DNNs) have shown large speech intelligibility improvements. However, these approaches fail to achieve better enhancement results at low signal-to-noise ratio (SNR) conditions since they ignore the phase information during reconstruction. Alternatively, GANs have been introduced effectively for speech enhancement and achieved improved performance due to the adversarial training. Motivated by the recent success of GAN, we introduce the phase sensitive masking (PSM) in a cGAN framework for speech enhancement task. The reason for choosing a conditional generative model is that the data generation process can be controlled with the use of additional temporal context information. In addition, we use gradient penalty regularization in the discriminator of the cGAN network to avoid vanishing gradients problem which in turn stabilizes the training of the cGAN network and increases the quality of the generated samples. The use of PSM is due to the fact that it involves both amplitude and phase information and produces an improved estimate of clean speech signal with higher SNR as compared to other T-F masks. Experimental results show the proposed PSM based cGAN architecture has shown significant improvements in performance measures compared to other baselines such as SEGAN, Deep Feature Loss, MetricGAN, AECNN, DNN-cIRM, and end-to-end approach with reference to quality and intelligibility.

Multi-Modal Adaptive Fusion Transformer Network for the Estimation of Depression Level.

Depression is a severe psychological condition that affects millions of people worldwide. As depression has received more attention in recent years, it has become imperative to develop automatic methods for detecting depression. Although numerous machine learning methods have been proposed for estimating the levels of depression via audio, visual, and audiovisual emotion sensing, several challenges still exist. For example, it is difficult to extract long-term temporal context information from long sequences of audio and visual data, and it is also difficult to select and fuse useful multi-modal information or features effectively. In addition, how to include other information or tasks to enhance the estimation accuracy is also one of the challenges. In this study, we propose a multi-modal adaptive fusion transformer network for estimating the levels of depression. Transformer-based models have achieved state-of-the-art performance in language understanding and sequence modeling. Thus, the proposed transformer-based network is utilized to extract long-term temporal context information from uni-modal audio and visual data in our work. This is the first transformer-based approach for depression detection. We also propose an adaptive fusion method for adaptively fusing useful multi-modal features. Furthermore, inspired by current multi-task learning work, we also incorporate an auxiliary task (depression classification) to enhance the main task of depression level regression (estimation). The effectiveness of the proposed method has been validated on a public dataset (AVEC 2019 Detecting Depression with AI Sub-challenge) in terms of the PHQ-8 scores. Experimental results indicate that the proposed method achieves better performance compared with currently state-of-the-art methods. Our proposed method achieves a concordance correlation coefficient (CCC) of 0.733 on AVEC 2019 which is 6.2% higher than the accuracy (CCC = 0.696) of the state-of-the-art method.

Spatiotemporal Continuous Impervious Surface Mapping by Fusion of Landsat Time Series Data and Google Earth Imagery

The monitoring of impervious surfaces in urban areas using remote sensing with fine spatial and temporal resolutions is crucial for monitoring urban development and environmental changes in urban areas. Spatiotemporal super-resolution mapping (STSRM) fuses fine-spatial-coarse-temporal remote sensing data with coarse-spatial-fine-temporal data, allowing for urban impervious surface mapping at both fine-spatial and fine-temporal resolutions. The STSRM involves two main steps: unmixing the coarse-spatial-fine-temporal remote sensing data to class fraction images, and downscaling the fraction images to sub-pixel land cover maps. Yet, challenges exist in each step when applying STSRM in mapping impervious surfaces. First, the impervious surfaces have high spectral variability (i.e., high intra-class and low inter-class variability), which impacts the accurate extraction of sub-pixel scale impervious surface fractions. Second, downscaling the fraction images to sub-pixel land cover maps is an ill-posed problem and would bring great uncertainty and error in the predictions. This paper proposed a new Spatiotemporal Continuous Impervious Surface Mapping (STCISM) method to deal with these challenges in fusing Landsat and Google Earth imagery. The STCISM used the Multiple Endmember Spectral Mixture Analysis and the Fisher Discriminant Analysis to minimize the within-class variability and maximize the between-class variability to reduce the spectral unmixing uncertainty. In addition, the STCISM adopted a new temporal consistency check model to incorporate temporal contextual information to reduce the uncertainty in the time-series impervious surface prediction maps. Unlike the traditional temporal consistency check model that assumed the impervious-to-pervious conversion is unlikely to happen, the new model allowed the bidirectional conversions between pervious and impervious surfaces. The temporal consistency check was used as a post-procession method to correct the errors in the prediction maps. The proposed STCISM method was used to predict time-series impervious surface maps at 5 m resolution of Google Earth image at the Landsat frequency. The results showed that the proposed STCISM outperformed the STSRM model without using the temporal consistency check and the STSRM model using the temporal consistency check based on the unidirectional pervious-to-impervious surface conversion rule.

CompFeat: Comprehensive Feature Aggregation for Video Instance Segmentation

Video instance segmentation is a complex task in which we need to detect, segment, and track each object for any given video. Previous approaches only utilize single-frame features for the detection, segmentation, and tracking of objects and they suffer in the video scenario due to several distinct challenges such as motion blur and drastic appearance change. To eliminate ambiguities introduced by only using single-frame features, we propose a novel comprehensive feature aggregation approach (CompFeat) to refine features atboth frame-level and object-level with temporal and spatial context information. The aggregation process is carefully designed with a new attention mechanism which significantly increases the discriminative power of the learned features. We further improve the tracking capability of our model through a siamese design by incorporating both feature similarities and spatial similarities. Experiments conducted on the YouTube-VIS dataset validate the effectiveness of proposed CompFeat.

Temporal context guides visual exploration during scene recognition.

Memories for episodes are temporally structured. Cognitive models derived from list-learning experiments attribute this structure to the retrieval of temporal context information that indicates when a memory occurred. These models predict key features of memory recall, such as the strong tendency to retrieve studied items in the order in which they were first encountered. Can such models explain ecological memory behaviors, such as eye movements during encoding and retrieval of complex visual stimuli? We tested predictions from retrieved-context models using three data sets involving recognition memory and free viewing of complex scenes. Subjects reinstated sequences of eye movements from one scene-viewing episode to the next. Moreover, sequence reinstatement decayed over time and was associated with successful memory. We observed memory-driven reinstatement even after accounting for intrinsic scene properties that produced consistent eye movements. These findings confirm predictions of retrieved-context models, suggesting retrieval of temporal context influences complex behaviors generated during naturalistic memory experiences. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

A Robust Detector for Automated Welding Seam Tracking System

AbstractAccurate locating of the weld seam under strong noise is the biggest challenge for automated welding. In this paper, we construct a robust seam detector on the framework of deep learning object detection algorithm. The representative object algorithm, a single shot multibox detector (SSD), is studied to establish the seam detector framework. The improved SSD is applied to seam detection. Under the SSD object detection framework, combined with the characteristics of the seam detection task, the multifeature combination network (MFCN) is proposed. The network comprehensively utilizes the local information and global information carried by the multilayer features to detect a weld seam and realizes the rapid and accurate detection of the weld seam. To solve the problem of single-frame seam image detection algorithm failure under continuous super-strong noise, the sequence image multifeature combination network (SMFCN) is proposed based on the MFCN detector. The recurrent neural network (RNN) is used to learn the temporal context information of convolutional features to accurately detect the seam under continuous super-noise. Experimental results show that the proposed seam detectors are extremely robust. The SMFCN can maintain extremely high detection accuracy under continuous super-strong noise. The welding results show that the laser vision seam tracking system using the SMFCN can ensure that the welding precision meets industrial requirements under a welding current of 150 A.

Learning long-term temporal contexts using skip RNN for continuous emotion recognition

BackgroundContinuous emotion recognition as a function of time assigns emotional values to every frame in a sequence. Incorporating long-term temporal context information is essential for continuous emotion recognition tasks. MethodsFor this purpose, we employ a window of feature frames in place of a single frame as inputs to strengthen the temporal modeling at the feature level. The ideas of frame skipping and temporal pooling are utilized to alleviate the resulting redundancy. At the model level, we leverage the skip recurrent neural network to model the long-term temporal variability by skipping trivial information for continuous emotion recognition. ResultsThe experimental results using the AVEC 2017 database demonstrate that our proposed methods are beneficial to a performance improvement. Further, the skip long short-term memory (LSTM) model can focus on the critical emotional state when training the models, thereby achieving a better performance than the LSTM model and other methods.

DuroNet

Urban crime is an ongoing problem in metropolitan development and attracts general concern from the international community. As an effective means of defending urban safety, crime prediction plays a crucial role in patrol force allocation and public safety. However, urban crime data is a macro result of crime patterns overlapped by various irrelevant factors that cause inhomogeneous noises—local outliers and irregular waves. These noises might obstruct the learning process of crime prediction models and result in a deviation of performance. To tackle the problem, we propose a novel paradigm of <underline>Du</underline>al-<underline>ro</underline>bust Enhanced Spatial-temporal Learning <underline>Net</underline>work (DuroNet), an encoder-decoder architecture that possesses an adaptive robustness for reducing the effect of outliers and waves. The robustness is mainly reflected on two aspects. One is a locality enhanced module that employs local temporal context information to smooth the deviation of outliers and dynamic spatial information to assist in understanding normal points. The other is a self-attention-based pattern representation module to weaken the effect of irregular waves by learning attentive weights. Finally, extensive experiments are conducted on two real-world crime datasets before and after adding Gaussian noises. The results demonstrate the superior performance of our DuroNet over the state-of-the-art methods.

Relational Reasoning for Group Activity Recognition via Self-Attention Augmented Conditional Random Field.

This paper presents a new relational network for group activity recognition. The essence of the network is to integrate conditional random fields (CRFs) with self-attention to infer the temporal dependencies and spatial relationships of the actors. This combination can take advantage of the capability of CRFs in modelling the actors' features that depend on each other and the capability of self-attention in learning the temporal evolution and spatial relational contexts of every actor in videos. Additionally, there are two distinct facets of our CRF and self-attention. First, the pairwise energy of the new CRF relies on both of the temporal self-attention and spatial self-attention, which apply the self-attention mechanism to the features in time and space, respectively. Second, to address both local and non-local relationships in group activities, the spatial self-attention takes account of a collection of cliques with different scales of spatial locality. The associated mean-field inference thereafter can thus be reformulated as a self-attention network to generate the relational contexts of the actors and their individual action labels. Lastly, a bidirectional universal transformer encoder (UTE) is utilized to aggregate the forward and backward temporal context information, scene information and relational contexts for group activity recognition. A new loss function is also employed, consisting of not only the cost for the classification of individual actions and group activities, but also a contrastive loss to address the miscellaneous relational contexts between actors. Simulations show that the new approach can surpass previous works on four commonly used datasets.

Deep Predictive Video Compression Using Mode-Selective Uni- and Bi-Directional Predictions Based on Multi-Frame Hypothesis

Recently, deep learning-based image compression has shown significant performance improvement in terms of coding efficiency and subjective quality. However, there has been relatively less effort on video compression based on deep neural networks. In this paper, we propose an end-to-end deep predictive video compression network, called DeepPVCnet, using mode-selective uni- and bi-directional predictions based on multi-frame hypothesis with a multi-scale structure and a temporal-context-adaptive entropy model. Our DeepPVCnet jointly compresses motion information and residual data that are generated from the multi-scale structure via the feature transformation layers. Recent deep learning-based video compression methods were proposed in a limited compression environment using only P-frame or B-frame. Learned from the lesson of the conventional video codecs, we firstly incorporate a mode-selective framework into our DeepPVCnet with uni- and bi-directional predictive modes in a rate-distortion minimization sense. Also, we propose a temporal-context-adaptive entropy model that utilizes the temporal context information of the reference frames for the current frame coding. The autoregressive entropy models for CNN-based image and video compression is difficult to compute with parallel processing. On the other hand, our temporal-context-adaptive entropy model utilizes temporally coherent context from the reference frames, so that the context information can be computed in parallel, which is computationally and architecturally advantageous. Extensive experiments show that our DeepPVCnet outperforms AVC/H.264, HEVC/H.265 and state-of-the-art methods in an MS-SSIM perspective.

Privacy-Aware Data Fusion and Prediction With Spatial-Temporal Context for Smart City Industrial Environment

As one of the cyber–physical–social systems that plays a key role in people's daily activities, a smart city is producing a considerable amount of industrial data associated with transportation, healthcare, business, social activities, and so on. Effectively and efficiently fusing and mining such data from multiple sources can contribute much to the development and improvements of various smart city applications. However, the industrial data collected from the smart city are often sensitive and contain partial user privacy such as spatial–temporal context information. Therefore, it is becoming a necessity to secure user privacy hidden in the smart city data before these data are integrated together for further mining, analyses, and prediction. However, due to the inherent tradeoff between data privacy and data availability, it is often a challenging task to protect users’ context privacy while guaranteeing accurate data analysis and prediction results after data fusion. Considering this challenge, a novel privacy-aware data fusion and prediction approach for the smart city industrial environment is put forward in this article, which is based on the classic locality-sensitive hashing technique. At last, our proposal is evaluated by a set of experiments based on a real-world dataset. Experimental results show better prediction performances of our approach compared to other competitive ones.

Human action recognition using convolutional LSTM and fully-connected LSTM with different attentions

This paper aims to address the human action recognition issue by using convolutional long short-term memory networks (Conv-LSTM) and fully-connected LSTM (FC-LSTM) with different attentions. To this end, the spatial-temporal dual-attention network (STDAN), which is mainly composed of feature extraction, attention and fusion modules, is designed. Different from the features of high-level fully-connected layer mostly used in previous work, the features of convolution and fully-connected layers of convolutional neural network (CNN) are both extracted in STDAN, which can enrich the initial level of video representation. Besides, the Conv-LSTM and FC-LSTM are employed to handle the long-duration sequential features with different temporal context information. To reinforce the spatial-temporal attention ability, a temporal attention module (TAM) and a joint spatial-temporal attention module (JSTAM) are implemented. Through the principle components analysis (PCA) and features fusion, the potential of STDAN is effectively explored and weighted. Finally, the experimental results show that the proposed STDAN has better recognition performance than existing state-of-the-art methods.

TEINet: Towards an Efficient Architecture for Video Recognition

Efficiency is an important issue in designing video architectures for action recognition. 3D CNNs have witnessed remarkable progress in action recognition from videos. However, compared with their 2D counterparts, 3D convolutions often introduce a large amount of parameters and cause high computational cost. To relieve this problem, we propose an efficient temporal module, termed as Temporal Enhancement-and-Interaction (TEI Module), which could be plugged into the existing 2D CNNs (denoted by TEINet). The TEI module presents a different paradigm to learn temporal features by decoupling the modeling of channel correlation and temporal interaction. First, it contains a Motion Enhanced Module (MEM) which is to enhance the motion-related features while suppress irrelevant information (e.g., background). Then, it introduces a Temporal Interaction Module (TIM) which supplements the temporal contextual information in a channel-wise manner. This two-stage modeling scheme is not only able to capture temporal structure flexibly and effectively, but also efficient for model inference. We conduct extensive experiments to verify the effectiveness of TEINet on several benchmarks (e.g., Something-Something V1&V2, Kinetics, UCF101 and HMDB51). Our proposed TEINet can achieve a good recognition accuracy on these datasets but still preserve a high efficiency.

Temporal Context Enhanced Feature Aggregation for Video Object Detection

Video object detection is a challenging task because of the presence of appearance deterioration in certain video frames. One typical solution is to aggregate neighboring features to enhance per-frame appearance features. However, such a method ignores the temporal relations between the aggregated frames, which is critical for improving video recognition accuracy. To handle the appearance deterioration problem, this paper proposes a temporal context enhanced network (TCENet) to exploit temporal context information by temporal aggregation for video object detection. To handle the displacement of the objects in videos, a novel DeformAlign module is proposed to align the spatial features from frame to frame. Instead of adopting a fixed-length window fusion strategy, a temporal stride predictor is proposed to adaptively select video frames for aggregation, which facilitates exploiting variable temporal information and requiring fewer video frames for aggregation to achieve better results. Our TCENet achieves state-of-the-art performance on the ImageNet VID dataset and has a faster runtime. Without bells-and-whistles, our TCENet achieves 80.3% mAP by only aggregating 3 frames.