Inter-frame Correlation Research Articles

A multi-stage spatio-temporal adaptive network for video super-resolution

Video super-resolution aims at restoring the spatial resolution of the reference frame based on consecutive input low-resolution (LR) frames. Existing implicit alignment-based video super-resolution methods commonly utilize convolutional LSTM (ConvLSTM) to handle sequential input frames. However, vanilla ConvLSTM processes input features and hidden states independently in operations and has limited ability to handle the inter-frame temporal redundancy in low-resolution fields. In this paper, we propose a multi-stage spatio-temporal adaptive network (MS-STAN). A spatio-temporal adaptive ConvLSTM (STAC) module is proposed to handle input features in low-resolution fields. The proposed STAC module utilizes the correlation between input features and hidden states in the ConvLSTM unit and modulates the hidden states adaptively conditioned on fused spatio-temporal features. A residual stacked bidirectional (RSB) architecture is further proposed to fully exploit the processing ability of the STAC unit. The proposed STAC and RSB architecture promote the vanilla ConvLSTM’s ability to exploit the inter-frame correlations, thus improving the reconstruction quality. Furthermore, different from existing methods that only aggregate features from the temporal branch once at a specified stage of the network, the proposed network is organized in a multi-stage manner. The corresponding temporal correlation in features at different stages can be fully exploited. Experimental results on Vimeo-90K-T and UMD10 datasets show that the proposed method has comparable performance with current video super-resolution methods. The code is available at https://github.com/yhjoker/MS-STAN.

Robust Dynamic Hand Gesture Recognition Based on Millimeter Wave Radar Using Atten-TsNN

The millimeter-wave radar can sense the subtle movement of the hand. However, the traditional hand gesture recognition methods do not work well in dynamic interference scenarios. To address this issue, a robust hand gesture recognition method is proposed based on the self-attention time-series neural networks. Firstly, the original radar echo is constructed in terms of frame, sequence, and channel at the input terminal of the network. A one-dimensional time-series neural network is built, and the time-distributed layer is used as the wrapper to extract the feature from each frame sequence independently. Then the self-attention mechanism is employed to assign the adequate weights to the sequence of frames entered in parallel to obtain the inter-frame correlation and suppress the random interference. Finally, the Global AvgPooling layer is used to reduce the number of channels, and the fully connected layer outputs the label of the gesture. The experimental results show that the proposed method can achieve a high recognition rate in the presence of random dynamic interference.

Multiple forgery detection in video using inter-frame correlation distance with dual-threshold

Multiple forgery detection in video using inter-frame correlation distance with dual-threshold

Multi-object tracking based on attention networks for Smart City system

The Smart City is a hot topic at present. Pedestrian tracking and behavior analysis play an important role in Smart City system. Therefore, it is urgent to design an universal and robust multi-object tracking(MOT) algorithm. MOT is a challenging problem with numerous significant applications, including robot navigation, autonomous driving and pedestrian behavior analysis. Among the existing MOT approaches, MOT models which joint learning detection and Re-ID are attracting increasing attention. Subsequently, there is competition between these two tasks jointly in a network inevitably leading to performance degradation. To address these difficulties, we propose an adaptive joint learning approach, called UnionTrack, for MOT in this paper. It handles with the problem of uniform learning between tasks in online MOT system. Firstly, UnionTrack introduces a scale-aware attention network that aggregates features from different scales for object localization and learning of the corresponding discriminative embeddings. Secondly, a cross-correlation module is designed to construct task-relevant feature maps and interact adaptively during training detection and Re-ID. Then, we perform inter-frame correlation at different stages by using high and low confidence detection to avoid occlusions and backgrounds. UnionTrack is trained in an end-to-end manner to better accommodate the training between detection and Re-ID tasks. Finally, our proposed UnionTrack is compared with state-of-the-art methods in extensively experiments and detailed performance analysis is performed in MOT15, MOT16, MOT17 and MOT20 which are all urban scene to demonstrate the effectiveness of our proposed method. Looking forward to playing a role in the Smart City systems.

Motion cues guided feature aggregation and enhancement for video object segmentation

Video object segmentation (VOS) aims to separate unknown target objects from various given video sequences. Although many recent successful methods boosted the performance of VOS, especially those using deep convolution neural networks (CNNs), it is still difficult to aggregate deep features as well as motion cues effectively, which can be important to associate valid information of adjacent frames in video sequences. To tackle this problem, we propose a simple yet effective feature optimization method for VOS based on motion information. To achieve this, we construct a two-branch deep network and use computed motion cues (i.e., optical flow) to jointly optimize global and local interframe correlation information. Additionally, a clustering-based feature enhancement module is proposed to further fuse motion information and enhance the feature saliency of the target area. Optimized feature maps show a significant performance improvement in the final VOS tasks, especially those with rapid target movement. Experiments on the DAVIS16, DAVIS17, YouTube-Objects and YouTube-VOS datasets demonstrate that our simple feature aggregation and enhancement method for VOS improves segmentation accuracy effectively and gains an impressive result compared to many state-of-the-art methods.

A separation and interaction framework for causal multi-channel speech enhancement

Multi-channel speech enhancement aims at extracting the desired speech using a microphone array, which has many potential applications, such as video conferencing, automatic speech recognition, and hearing aids. Recently, deep learning-based spatial filters have achieved remarkable improvements over traditional beamformers, and the desired speech is often inferred directly using the noisy features without modeling the interference. In this work, a novel two-stage framework is proposed to extract the desired speech under the guidance of both the estimated interference and the desired signal. The resulting framework, called a Separation and Interaction Network (SI-Net), includes two components: the first module separates speech and interference coarsely, and the second sub-network serves as the post-processing module to suppress the residual noise and regenerate some missing speech components simultaneously under the guidance of previously estimated speech and interference characters. Because these two modules are both differentiable, the proposed framework can be trained in an end-to-end manner. In addition, a causal spatial-temporal attention module is designed to effectively model the inter-channel and inter-frame correlations simultaneously. Moreover, under this framework, we adopt the channel shuffle and gated fusion strategies for the interaction between speech and interference components to deliver the knowledge about both “where to suppress and where to enhance”. Experiments conducted on the simulated multi-channel speech dataset illustrate the superiority of the proposed framework over state-of-the-art baselines, while can still support real-time processing.

Spatial–Temporal Graph Transformer With Sign Mesh Regression for Skinned-Based Sign Language Production

Sign language production aims to automatically generate coordinated sign language videos from spoken language. As a typical sequence to sequence task, the existing methods are mostly to regard the skeletons as a whole sequence, however, those do not take the rich graph information among both joints and edges into consideration. In this paper, we propose a novel method named Spatial-Temporal Graph Transformer (STGT) to deal with this problem. Specifically, according to kinesiology, we first design a novel graph representation to achieve graph features from skeletons. Then the spatial-temporal graph self-attention utilizes graph topology to capture the intra-frame and inter-frame correlations, respectively. Our key innovation is that the attention maps are calculated on both spatial and temporal dimensions in turn, meanwhile, graph convolution is used to strengthen the short-term features of skeletal structure. Finally, due to the generated skeletons are based on the form of skeleton points and lines so far. In order to visualize the generated sign language videos, we design a sign mesh regression module to render the skeletons into skinned animations including body and hands posture. Comparing with states of art baseline on RWTH-PHONEIX Weather-2014T in Experiment Section, STGT can obtain the highest values on BLEU and ROUGE, which indicates our method produces most accurate and intuitive sign language videos.

Multiframe Detection of Sea-Surface Small Target Using Deep Convolutional Neural Network

Sea-surface small target detection is challenging for maritime radar. Unfortunately, conventional detection methods are often limited to complex marine environment and low signal-to-clutter ratio (SCR). This article presents a multiframe detection approach for sea-surface small target by using deep convolutional neural network. The moving targets can be reconstructed and detected from the sequential range–Doppler (RD) spectra. A two-step detection framework is proposed, where the intraframe and interframe detections are achieved using the differences in features and interframe correlations between the moving target and sea clutter, respectively. The proposed approach has been verified on both the simulated and real sea-surface small targets, which shows better detection performance than the conventional multiframe detection algorithms. Additionally, this approach exhibits acceptable generalization ability.

Multi-Frame Constrained Block Sparse Bayesian Learning for Flexible Tactile Sensing Using Electrical Impedance Tomography

In this paper, we presented a multi-frame constrained block sparse Bayesian learning (MFC-BSBL) reconstruction algorithm to tackle the challenge of poor-quality reconstruction images in electrical impedance tomography (EIT) for tactile sensing. The fundamental idea of MFC-BSBL is to explore the sparsity, intra-frame correlation, and inter-frame correlation of impedance distributions by extending the Bayesian inference framework. To verify the proposed algorithm, we conducted numerical simulations for different cases to identify one, multiple, round, and square targets. The simulation results demonstrated that this method can effectively detect the target positions and shapes by reducing artifacts and noise in the reconstructed images. To demonstrate the application of this approach to real EIT-based tactile sensing, we conducted real-contact detection experiments using the EIT tactile sensor system. Compared with traditional methods, the tactile sensor system using the MFC-BSBL algorithm can achieve accurate contact detection and significantly reduce artifacts and noise.

Subblock-Based Motion Derivation and Inter Prediction Refinement in the Versatile Video Coding Standard

Efficient representation and coding of fine-granular motion information is one of the key research areas for exploiting inter-frame correlation in video coding. Representative techniques towards this direction are affine motion compensation (AMC), decoder-side motion vector refinement (DMVR), and subblock-based temporal motion vector prediction (SbTMVP). Fine-granular motion information is derived at subblock level for all the three coding tools. In addition, the obtained inter prediction can be further refined by two optical flow-based coding tools, the bi-directional optical flow (BDOF) for bi-directional inter prediction and the prediction refinement with optical flow (PROF) exclusively used in combination with AMC. The aforementioned five coding tools have been extensively studied and finally adopted in the Versatile Video Coding (VVC) standard. This paper presents technical details of each tool and highlights the design elements with the consideration of typical hardware implementations. Following the common test conditions defined by Joint Video Experts Team (JVET) for the development of VVC, 5.7% bitrate reduction on average is achieved by the five tools. For test sequences characterized by large and complex motion, up to 13.4% bitrate reduction is observed. Additionally, visual quality improvement is demonstrated and analyzed.

Dynamic Gesture Recognition Method Based on Millimeter-wave Radar by One-Dimensional Series Neural Network

For the most of the existing gesture recognition methods based on the radar sensor, the parameters such as the distance, Doppler, and angle are estimated using the radar echo at first. And then the obtained data spectra are inputted into the convolutional neural networks to classify the gestures. The implementation process is complicated. A dynamic gesture recognition method is proposed based on the millimeter-wave radar using the One-Dimensional Series connection Neural Networks (1D-ScNN) in this paper. Firstly, the original echo of dynamic gesture is obtained by the millimeter-wave radar. The gesture features are extracted by the one-dimensional convolution and pooling operations, and then are inputted into the one-dimensional inception v3 structure. In order to aggregate the one-dimensional features, the Long Short-Term Memory (LSTM) modular is connected to the end of the network. The inter-frame correlation of dynamic gestures echo is fully utilized to improve the recognition accuracy and the convergence speed of training. The experimental results show that the proposed method is simple in implementation and has a fast convergence speed. The classification accuracy can reach more than 96.0%, which is higher than the traditional gesture classification methods.

Adaptive Block-Based Compressed Video Sensing Based on Saliency Detection and Side Information.

The setting of the measurement number for each block is very important for a block-based compressed sensing system. However, in practical applications, we only have the initial measurement results of the original signal on the sampling side instead of the original signal itself, therefore, we cannot directly allocate the appropriate measurement number for each block without the sparsity of the original signal. To solve this problem, we propose an adaptive block-based compressed video sensing scheme based on saliency detection and side information. According to the Johnson–Lindenstrauss lemma, we can use the initial measurement results to perform saliency detection and then obtain the saliency value for each block. Meanwhile, a side information frame which is an estimate of the current frame is generated on the reconstruction side by the proposed probability fusion model, and the significant coefficient proportion of each block is estimated through the side information frame. Both the saliency value and significant coefficient proportion can reflect the sparsity of the block. Finally, these two estimates of block sparsity are fused, so that we can simultaneously use intra-frame and inter-frame correlation for block sparsity estimation. Then the measurement number of each block can be allocated according to the fusion sparsity. Besides, we propose a global recovery model based on weighting, which can reduce the block effect of reconstructed frames. The experimental results show that, compared with existing schemes, the proposed scheme can achieve a significant improvement in peak signal-to-noise ratio (PSNR) at the same sampling rate.

Single-channel speech enhancement based on improved frame-iterative spectral subtraction in the modulation domain

Aiming at the problem of music noise introduced by classical spectral subtraction, a short- time modulation domain (STM) spectral subtraction method has been successfully applied for singlechannel speech enhancement. However, due to the inaccurate voice activity detection (VAD), the residual music noise and enhanced performance still need to be further improved, especially in the low signal to noise ratio (SNR) scenarios. To address this issue, an improved frame iterative spectral subtraction in the STM domain (IMModSSub) is proposed. More specifically, with the inter-frame correlation, the noise subtraction is directly applied to handle the noisy signal for each frame in the STM domain. Then, the noisy signal is classified into speech or silence frames based on a predefined threshold of segmented SNR. With these classification results, a corresponding mask function is developed for noisy speech after noise subtraction. Finally, exploiting the increased sparsity of speech signal in the modulation domain, the orthogonal matching pursuit (OMP) technique is employed to the speech frames for improving the speech quality and intelligibility. The effectiveness of the proposed method is evaluated with three types of noise, including white noise, pink noise, and hfchannel noise. The obtained results show that the proposed method outperforms some established baselines at lower SNRs (−5 to +5 dB).

Side information hybrid generation based on improved motion vector field

The quality of side information has an important impact on the performance of a distributed video coding system. At present, the generation of side information is mainly based on the translational motion model using the inter-frame correlation for motion estimation. Considering that the generated side information is prone to block effect and ghosting, as well as the situation that the nonlinear motion is not fully considered and the intra-frame correlation is not fully utilized, a side information hybrid generation algorithm based on an improvement model for the motion vector field is proposed. The side information frame to be generated for the current Wyner-Ziv frame is divided into easy-to-estimate and difficult-to-estimate macroblocks. For difficult-to-estimate macroblocks, the Horn and Schunck dense optical flow method is used to generate reliable motion vectors, for easy-to-estimate macroblocks, the block matching method is used to generate unreliable motion vectors which are modified by the proposed scheme, and then, the improved motion vectors are used for motion compensation to produce the final side information frame. Experiment results show that the quality of side information obtained by using the improved motion vector field for motion compensation has been significantly improved, thus the overall performance of the distributed video coding system has been effectively improved.

Skeleton-based action recognition via spatial and temporal transformer networks

Skeleton-based Human Activity Recognition has achieved great interest in recent years as skeleton data has demonstrated being robust to illumination changes, body scales, dynamic camera views, and complex background. In particular, Spatial–Temporal Graph Convolutional Networks (ST-GCN) demonstrated to be effective in learning both spatial and temporal dependencies on non-Euclidean data such as skeleton graphs. Nevertheless, an effective encoding of the latent information underlying the 3D skeleton is still an open problem, especially when it comes to extracting effective information from joint motion patterns and their correlations. In this work, we propose a novel Spatial–Temporal Transformer network (ST-TR) which models dependencies between joints using the Transformer self-attention operator. In our ST-TR model, a Spatial Self-Attention module (SSA) is used to understand intra-frame interactions between different body parts, and a Temporal Self-Attention module (TSA) to model inter-frame correlations. The two are combined in a two-stream network, whose performance is evaluated on three large-scale datasets, NTU-RGB+D 60, NTU-RGB+D 120, and Kinetics Skeleton 400, consistently improving backbone results. Compared with methods that use the same input data, the proposed ST-TR achieves state-of-the-art performance on all datasets when using joints’ coordinates as input, and results on-par with state-of-the-art when adding bones information.

An Improved Background Subtraction Method for Adaptive Rate Compressive Sensing

In Compressive Video Sensing application, the using of Adaptive Rate Compressive Sensing (ARCS) method can predict and adjust the sampling rate for each video frame, reduce the total sampling rate and improve the quality of reconstructed image. In order to use the inter-frame correlation of the video signal efficiently and reduce the sampling rate of the whole signal, an Improved Background Subtraction (IBS) method is proposed in this paper. By alternately using the frame without any foreground objects and the previous frame as the background of the signal, the sparsity of the foreground signal is improved and the total sampling rate is reduced. Experimental results show that, compare with the traditional background subtraction method, the IBS method can significantly reduce the total video sampling rate without apparent degradation of reconstructed image quality under the same ARCS method.

Exploiting inter-frame regional correlation for efficient action recognition

Temporal feature extraction is an important issue in video-based action recognition. Optical flow is a popular method to extract temporal feature, which produces excellent performance thanks to its capacity of capturing pixel-level correlation information between consecutive frames. However, such a pixel-level correlation is extracted at the cost of high computational complexity and large storage resource. In this paper, we propose a novel temporal feature extraction method, Attentive Correlated Temporal Feature (ACTF), by exploring inter-frame correlation within a certain region. The proposed ACTF exploits both bilinear and linear correlations between successive frames on the regional level. Our method has the advantage of achieving performance comparable to or better than optical flow-based methods while avoiding the introduction of optical flow. Experimental results demonstrate our proposed method achieves the competitive performances of 96.3% on UCF101 and 76.3% on HMDB51 benchmark datasets.

A total variation and group sparsity-based algorithm for nuclear radiation-contaminated video restoration

ABSTRACT Nuclear radiation-contaminated video deblurring is an important issue of the robot vision system and has been widely studied. In this paper, a hybrid radiation-contaminated video frame enhancement algorithm is proposed that utilizes both intra-frame and inter-frame correlation by a two-stage strategy. In the first stage, total variation (TV) transformation are used to locate the spot areas, and then local TV is employed to restore spot areas. The preliminary deblurring result not only enhances the video frame and similar patch matching accuracy but also provides reliable estimates of filtering parameters. In the second stage, visual group technology and improved k-nearest neighbours (k-NN) method is used to select similar frames and reference patches respectively. The final enhanced video frame is obtained by a novel patch-based group sparse method. Experimental results clearly show that the proposed method outperforms other state-of-the-art methods in both quantitative evaluation indices and visual quality measurements.

LapFormer: surgical tool detection in laparoscopic surgical video using transformer architecture

ABSTRACT One of the most essential steps in the surgical workflow analysis is recognition of surgical tool presence. We propose a method to detect the presence of surgical tools in laparoscopic surgery videos, called LapFormer. The novelty of LapFormer is to use a Transformer architecture, which is a feed-forward neural network architecture with attention mechanism, growing in popularity for natural language processing, for analysing inter-frame correlation in videos instead of using recurrent neural network families. To the best of our knowledge, no methods using a Transformer architecture for analysing laparoscopic surgery videos have been proposed. We evaluate our method on a dataset called Cholec80, which contains 80 videos of cholecystectomy surgeries. We confirm that our proposed method outperforms the conventional methods such as single-frame analysis with convolutional neural networks or multiple frame analysis with recurrent neural networks by 20.3 and 17.3 points in macro-F1 score, respectively. We also conduct an ablation study on how hyper-parameters for Transformer block in our proposed method affect the performance of the detection.

Visual Scene-Aware Hybrid and Multi-Modal Feature Aggregation for Facial Expression Recognition.

Facial expression recognition (FER) technology has made considerable progress with the rapid development of deep learning. However, conventional FER techniques are mainly designed and trained for videos that are artificially acquired in a limited environment, so they may not operate robustly on videos acquired in a wild environment suffering from varying illuminations and head poses. In order to solve this problem and improve the ultimate performance of FER, this paper proposes a new architecture that extends a state-of-the-art FER scheme and a multi-modal neural network that can effectively fuse image and landmark information. To this end, we propose three methods. To maximize the performance of the recurrent neural network (RNN) in the previous scheme, we first propose a frame substitution module that replaces the latent features of less important frames with those of important frames based on inter-frame correlation. Second, we propose a method for extracting facial landmark features based on the correlation between frames. Third, we propose a new multi-modal fusion method that effectively fuses video and facial landmark information at the feature level. By applying attention based on the characteristics of each modality to the features of the modality, novel fusion is achieved. Experimental results show that the proposed method provides remarkable performance, with 51.4% accuracy for the wild AFEW dataset, 98.5% accuracy for the CK+ dataset and 81.9% accuracy for the MMI dataset, outperforming the state-of-the-art networks.