Inter-frame Correlation Research Articles

Fast inter-frame motion correction in contrast-free ultrasound quantitative microvasculature imaging using deep learning

Contrast-free ultrasound quantitative microvasculature imaging shows promise in several applications, including the assessment of benign and malignant lesions. However, motion represents one of the major challenges in imaging tumor microvessels in organs that are prone to physiological motions. This study aims at addressing potential microvessel image degradation in in vivo human thyroid due to its proximity to carotid artery. The pulsation of the carotid artery induces inter-frame motion that significantly degrades microvasculature images, resulting in diagnostic errors. The main objective of this study is to reduce inter-frame motion artifacts in high-frame-rate ultrasound imaging to achieve a more accurate visualization of tumor microvessel features. We propose a low-complex deep learning network comprising depth-wise separable convolutional layers and hybrid adaptive and squeeze-and-excite attention mechanisms to correct inter-frame motion in high-frame-rate images. Rigorous validation using phantom and in-vivo data with simulated inter-frame motion indicates average improvements of 35% in Pearson correlation coefficients (PCCs) between motion corrected and reference data with respect to that of motion corrupted data. Further, reconstruction of microvasculature images using motion-corrected frames demonstrates PCC improvement from 31 to 35%. Another thorough validation using in-vivo thyroid data with physiological inter-frame motion demonstrates average improvement of 20% in PCC and 40% in mean inter-frame correlation. Finally, comparison with the conventional image registration method indicates the suitability of proposed network for real-time inter-frame motion correction with 5000 times reduction in motion corrected frame prediction latency.

Fully Unsupervised Deepfake Video Detection Via Enhanced Contrastive Learning.

Nowadays, Deepfake videos are widely spread over the Internet, which severely impairs the public trustworthiness and social security. Although more and more reliable detectors have recently sprung up for resisting against that new-emerging tampering technique, some challengeable issues still need to be addressed, such that most of Deepfake video detectors under the framework of the supervised mechanism require a large scale of samples with accurate labels for training. When the amount of the training samples with the true labels are not enough or the training data are maliciously poisoned by adversaries, the supervised classifier is probably not reliable for detection. To tackle that tough issue, it is proposed to design a fully unsupervised Deepfake detector. In particular, in the whole procedure of training or testing, we have no idea of any information about the true labels of samples. First, we novelly design a pseudo-label generator for labeling the training samples, where the traditional hand-crafted features are used to characterize both types of samples. Second, the training samples with the pseudo-labels are fed into the proposed enhanced contrastive learner, in which the discriminative features are further extracted and continually refined by iteration on the guidance of the contrastive loss. Last, relying on the inter-frame correlation, we complete the final binary classification between real and fake videos. A large scale of experimental results empirically verify the effectiveness of our proposed unsupervised Deepfake detector on the benchmark datasets including FF++, Celeb-DF, DFD, DFDC, and UADFV. Furthermore, our proposed well-performed detector is superior to the current unsupervised method, and comparable to the baseline supervised methods. More importantly, when facing the problem of the labeled data poisoned by malicious adversaries or insufficient data for training, our proposed unsupervised Deepfake detector performs its powerful superiority.

PASS: Patch Automatic Skip Scheme for Efficient On-Device Video Perception.

Real-time video perception tasks are often challenging on resource-constrained edge devices due to the issues of accuracy drop and hardware overhead, where saving computations is the key to performance improvement. Existing methods either rely on domain-specific neural chips or priorly searched models, which require specialized optimization according to different task properties. These limitations motivate us to design a general and task-independent methodology, called Patch Automatic Skip Scheme (PASS), which supports diverse video perception settings by decoupling acceleration and tasks. The gist is to capture inter-frame correlations and skip redundant computations at patch level, where the patch is a non-overlapping square block in visual. PASS equips each convolution layer with a learnable gate to selectively determine which patches could be safely skipped without degrading model accuracy. Specifically, we are the first to construct a self-supervisory procedure for gate optimization, which learns to extract contrastive representations from frame sequences. The pre-trained gates can serve as plug-and-play modules to implement patch-skippable neural backbones, and automatically generate proper skip strategy to accelerate different video-based downstream tasks, e.g., outperforming state-of-the-art MobileHumanPose in 3D pose estimation and FairMOT in multiple object tracking, by up to 9.43 × and 12.19 × speedups, respectively, on NVIDIA Jetson Nano devices.

Enhanced Video Super-Resolution Network towards Compressed Data

Video super-resolution (VSR) algorithms aim at recovering a temporally consistent high-resolution (HR) video from its corresponding low-resolution (LR) video sequence. Due to the limited bandwidth during video transmission, most available videos on the internet are compressed. Nevertheless, few existing algorithms consider the compression factor in practical applications. In this paper, we propose an enhanced VSR model towards compressed videos, termed as ECVSR, to simultaneously achieve compression artifacts reduction and SR reconstruction end-to-end. ECVSR contains a motion-excited temporal adaption network (METAN) and a multi-frame SR network (SRNet). The METAN takes decoded LR video frames as input and models inter-frame correlations via bidirectional deformable alignment and motion-excited temporal adaption, where temporal differences are calculated as motion prior to excite the motion-sensitive regions of temporal features. In SRNet, cascaded recurrent multi-scale blocks (RMSB) are employed to learn deep spatio-temporal representations from adapted multi-frame features. Then, we build a reconstruction module for spatio-temporal information integration and HR frame reconstruction, which is followed by a detail refinement module for texture and visual quality enhancement. Extensive experimental results on compressed videos demonstrate the superiority of our method for compressed VSR. Code will be available at https://github.com/lifengcs/ECVSR .

Burst-Enhanced Super-Resolution Network (BESR).

Multi-frame super-resolution (MFSR) leverages complementary information between image sequences of the same scene to increase the resolution of the reconstructed image. As a branch of MFSR, burst super-resolution aims to restore image details by leveraging the complementary information between noisy sequences. In this paper, we propose an efficient burst-enhanced super-resolution network (BESR). Specifically, we introduce Geformer, a gate-enhanced transformer, and construct an enhanced CNN-Transformer block (ECTB) by combining convolutions to enhance local perception. ECTB efficiently aggregates intra-frame context and inter-frame correlation information, yielding an enhanced feature representation. Additionally, we leverage reference features to facilitate inter-frame communication, enhancing spatiotemporal coherence among multiple frames. To address the critical processes of inter-frame alignment and feature fusion, we propose optimized pyramid alignment (OPA) and hybrid feature fusion (HFF) modules to capture and utilize complementary information between multiple frames to recover more high-frequency details. Extensive experiments demonstrate that, compared to state-of-the-art methods, BESR achieves higher efficiency and competitively superior reconstruction results. On the synthetic dataset and real-world dataset of BurstSR, our BESR achieves PSNR values of 42.79 dB and 48.86 dB, respectively, outperforming other MFSR models significantly.

Spatio-temporal interactive fusion based visual object tracking method

Visual object tracking tasks often struggle with utilizing inter-frame correlation information and handling challenges like local occlusion, deformations, and background interference. To address these issues, this paper proposes a spatio-temporal interactive fusion (STIF) based visual object tracking method. The goal is to fully utilize spatio-temporal background information, enhance feature representation for object recognition, improve tracking accuracy, adapt to object changes, and reduce model drift. The proposed method incorporates feature-enhanced networks in both temporal and spatial dimensions. It leverages spatio-temporal background information to extract salient features that contribute to improved object recognition and tracking accuracy. Additionally, the model’s adaptability to object changes is enhanced, and model drift is minimized. A spatio-temporal interactive fusion network is employed to learn a similarity metric between the memory frame and the query frame by utilizing feature enhancement. This fusion network effectively filters out stronger feature representations through the interactive fusion of information. The proposed tracking method is evaluated on four challenging public datasets. The results demonstrate that the method achieves state-of-the-art (SOTA) performance and significantly improves tracking accuracy in complex scenarios affected by local occlusion, deformations, and background interference. Finally, the method achieves a remarkable success rate of 78.8% on TrackingNet, a large-scale tracking dataset.

Annihilation-Net: Learned annihilation relation for dynamic MR imaging.

Deep learning methods driven by the low-rank regularization have achieved attractive performance in dynamic magnetic resonance (MR) imaging. The effectiveness of existing methods lies mainly in their ability to capture interframe relationships using network modules, which are lackinterpretability. This study aims to design an interpretable methodology for modeling interframe relationships using convolutiona networks, namely Annihilation-Net and use it for accelerating dynamicMRI. Based on the equivalence between Hankel matrix product and convolution, we utilize convolutional networks to learn the null space transform for characterizing low-rankness. We employ low-rankness to represent interframe correlations in dynamic MR imaging, while combining with sparse constraints in the compressed sensing framework. The corresponding optimization problem is solved in an iterative form with the semi-quadratic splitting method (HQS). The iterative steps are unrolled into a network, dubbed Annihilation-Net. All the regularization parameters and null space transforms are set as learnable in theAnnihilation-Net. Experiments on the cardiac cine dataset show that the proposed model outperforms other competing methods both quantitatively and qualitatively. The training set and test set have 800 and 118 images,respectively. The proposed Annihilation-Net improves the reconstruction quality of accelerated dynamic MRI with betterinterpretability.

Glimpse and focus: Global and local-scale graph convolution network for skeleton-based action recognition

In the 3D skeleton-based action recognition task, learning rich spatial and temporal motion patterns from body joints are two foundational yet under-explored problems. In this paper, we propose two methods for improving these problems: (I) a novel glimpse-focus action recognition strategy that captures multi-range pose features from the whole body and key body parts jointly; (II) a powerful temporal feature extractor JD-TC that enriches trajectory features by inferring different inter-frame correlations for different joints. By coupling these two proposals, we develop a powerful skeleton-based action recognition system that extracts rich pose and trajectory features from a skeleton sequence and outperforms previous state-of-the-art methods on three large-scale datasets.

A Method for Generating Geometric Image Sequences for Non-Isomorphic 3D-Mesh Sequence Compression

As virtual reality and 3D-modeling technology continue to advance, the amount of digital geometric media data is growing at an explosive rate. For example, 3D meshes, an important type of digital geometric media, can precisely record geometric information on a model’s surface. However, as the complexity and precision of 3D meshes increase, it becomes more challenging to store and transmit them. The traditional method of compressing non-isomorphic 3D-mesh sequences through frame-by-frame compression is inefficient and destroys the inter-frame correlations of the sequences. To tackle these issues, this study investigates the generation of time-dependent geometric image sequences for compressing non-isomorphic 3D-mesh sequences. Two methods are proposed for generating such sequences: one through image registration and the other through parametrization-geometry cooperative registration. Based on the experimental compression results of the video-coding algorithms, it was observed that the proposed geometric image-sequence-generation method offers superior objective and subjective qualities, as compared to the traditional method.

TASTNet: An end-to-end deep fingerprinting net with two-dimensional attention mechanism and spatio-temporal weighted fusion for video content authentication

In this paper, a robust fingerprinting scheme for video content authentication with two-dimensional attention mechanism and spatio-temporal weighted fusion called TASTNet is proposed, which can automatically extracts key spatio-temporal features from the input video and maps them to the corresponding fingerprint. Detailedly, the two-dimensional attention mechanism is applied to resist different kinds of digital manipulations for robustness enhancement. To incorporate perceptual characteristics, a spatio-temporal weighted fusion method based on LTSM is presented to integrate frame-level features into video-level features while retaining the temporal order. In the process of fusion, key frames are allocated with larger weights according to inter-frame correlation. With these two steps, we can obtain representative video features that contain principal perception information. In addition, the proposed scheme utilizes deep metric learning for training, and we design multiple constraints to make the generated fingerprint more compact and discriminable. Extensive experiments demonstrate that our scheme can achieve superior performances with respect to robustness and discrimination compared with some state-of-the-art schemes.

Semi-supervised video object segmentation via an edge attention gated graph convolutional network

Video object segmentation (VOS) exhibits heavy occlusions, large deformation, and severe motion blur. While many remarkable convolutional neural networks are devoted to the VOS task, they often mis-identify background noise as the target or output coarse object boundaries, due to the failure of mining detail information and high-order correlations of pixels within the whole video. In this work, we propose an edge attention gated graph convolutional network (GCN) for VOS. The seed point initialization and graph construction stages construct a spatio-temporal graph of the video by exploring the spatial intra-frame correlation and the temporal inter-frame correlation of superpixels. The node classification stage identifies foreground superpixels by using an edge attention gated GCN which mines higher-order correlations between superpixels and propagates features among different nodes. The segmentation optimization stage optimizes the classification of foreground superpixels and reduces segmentation errors by using a global appearance model which captures the long-term stable feature of objects. In summary, the key contribution of our framework is twofold: (a) the spatio-temporal graph representation can propagate the seed points of the first frame to subsequent frames and facilitate our framework for the semi-supervised VOS task; (b) the edge attention gated GCN can learn the importance of each node with respect to both the neighboring nodes and the whole task with small number of layers. Experiments on Davis 2016 and Davis 2017 datasets show that our framework achieves the excellent performance with only small training samples (45 video sequences).

Infrared Small Target Detection via $L_{0}$ Sparse Gradient Regularized Tensor Spectral Support Low-Rank Decomposition

Infrared small target detection has been extensively investigated by incorporating the low-rank and sparse prior into tensor decomposition frameworks. Despite its success, the said paradigm remains several limitations in complex scenes, such as: (1) the inadequate spatial-temporal information exploitation among sequential patches; (2) the incomplete suppression of the complex background interference. To mitigate the defects, this paper provides a tensor decomposition method integrating spatial-temporal l0 sparse gradient regularization and spectral support constraint. First, we present a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">skillfully Connected Multiframe Patch Group</i> (CMPG) model to explore local spatial information and adjacent interframe correlation among multiframes patches. Then, for CMPG model, a scalable tensor spectral support constraint is employed to distinctively regularized its redundant and rare compo-nents. Considering the nonlocal uniqueness of small targets and the local continuity of rare distractors, an extended spatial-temporal <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$l_{0}$</tex-math></inline-formula> gradient constraint is embedded into target-background separation for better suppression of structural clutter, and a reweighted scheme is also used to eliminate isolated non-target points. The final model is efficiently solved by the alternating direction method of multipliers (ADMM). Experiments are conducted on extensive simulating datasets and real scenes, suggesting that the proposed method achieves a considerable boost against other competitors in terms of visual effect and subjective evaluation.

Deep Wyner-Ziv Decoding Using Auxiliary Hierarchical Features

Wyner-Ziv (WZ) video coding shifts the burden of complex calculations from the encoder to the decoder, making it suitable for video coding scenarios with limited resources. Motivated by deep learning has shown superior performance over traditional methods, this paper proposes deep WZ video coding with the help of auxiliary hierarchical features in the decoder. It uses an autoencoder to encode WZ frames. In the decoder side, an inter-frame correlation model called SI-Net is employed to enhance WZ frame quality with Key frames. The auxiliary hierarchical features are extracted from Key frames through multi-level downsampling and employed in autoencoder to optimize feature extraction, avoiding gradient disappearance caused by deep networks. Since the auxiliary hierarchical features of Key frames describe spatial information and expand the network’s perception of video frame features, a high-quality WZ frame reconstructed can be obtained. Compared with previous work, our method shows obvious superiority on four video datasets with different degrees of motion.

Dual feature enhanced video super-resolution network based on low-light scenarios

Video super-resolution (VSR) reconstruction technique aims to improve the spatiotemporal resolution of consecutive low resolution (LR) frames. Most of previous methods use the correlation of inter-frame to restore pixels. It is still a challenge to exploit intra-frame and inter-frame correlations to recover high-resolution (HR) frames, especially reconstruction of video in low-light conditions for sharp imaging results has been a bottleneck in current industrial environments. Therefore, this paper proposes a novel VSR network. Specifically, we first generate the hidden information of the current frame by using the appropriate combination of the front and rear frames. Then, a re-fusion block (RFB) is designed, which utilizes the hidden information to re-fuse with corresponding LR frame. After that, we integrate an improved dual attention mechanism (DAM) into network to extract more accurate feature of intra-frame without increasing the number of parameters. The technology has important practical significance in low-light and dim scenes, so we collect some industrial video sequences and make datasets to complete VSR task. Experimental results show that our model significantly outperforms state-of-the-art methods in performance.

Real-Time Quality Monitoring of Ultrathin Sheets Edge Welding Based on Microvision Sensing and SOCIFS-SVM

Ultrathin sheets edge welding is a critical technology for aircraft, aerospace, or much high-end equipment manufacturing industrial applications. However, its online quality monitoring is still challenging due to the complicated and multicoupled transport phenomena in the mesoscale molten pool. This article presents an intelligent methodology for real-time monitoring of edge welds forming quality using microvision sensing and support vector machine based on swarm optimized and computationally inexpensive floating selection (SOCIFS-SVM). The clear images of the mesoscale molten pool are continuously acquired by using the passive microvision sensor. Based on the analysis for the dynamic behavior of the mesoscale molten pool and the morphology of the formed edge welds, a robust image processing procedure is developed to extract the single-frame spatial features and interframe correlation features from the microvision image sequence of the molten pool. A Kalman filter is applied to suppress the noise caused by the pulse current, etc. Then, the SOCIFS-SVM-based approach is established to predict three typical types of welding states. Experimental results with multiple groups of different welding parameters show that the proposed method can effectively and robustly identify the lack of fusion, humping, and sound weld with the highest accuracy of 96.09%. In addition, the established model can still achieve mean accuracy of 94.98% for additional untrained data, indicating that the model has strong generalization ability. The proposed real-time quality monitoring method makes online defect diagnosis of ultrathin sheets edge welding possible and provides a basis for online quality closed-loop control.

SDDNet: Infrared small and dim target detection network

AbstractThis study focuses on developing deep learning methods for small and dim target detection. We model infrared images as the union of the target region and background region. Based on this model, the target detection problem is considered a two‐class segmentation problem that divides an image into the target and background. Therefore, a neural network called SDDNet for single‐frame images is constructed. The network yields target extraction results according to the original images. For multiframe images, a network called IC‐SDDNet, a combination of SDDNet and an interframe correlation network module is constructed. SDDNet and IC‐SDDNet achieve target detection rates close to 1 on typical datasets with very low false positives, thereby performing significantly better than current methods. Both models can be executed end to end, so both are very convenient to use, and their implementation efficiency is very high. Average speeds of 540+/230+ FPS and 170+/60+ FPS are achieved with SDDNet and IC‐SDDNet on a single Tesla V100 graphics processing unit and a single Jetson TX2 embedded module respectively. Additionally, neither network needs to use future information, so both networks can be directly used in real‐time systems. The well‐trained models and codes used in this study are available at https://github.com/LittlePieces/ObjectDetection.

MAP-Inspired Deep Unfolding Network for Distributed Compressive Video Sensing

Due to the lack of probabilistic inference guidance, the network structure of current deep learning-based Distributed Compressive Video Sensing (DCVS) algorithms cannot efficiently exploit frame measurements, inter-frame correlations, and intra-frame correlations. In this paper, we infer the optimization equation for DCVS reconstruction according to the maximum a posteriori estimation and propose a Deep Unfolding Multi-Hypothesis Aggregation Network (DUMHAN) to obtain high-quality reconstruction frames. In each stage of the DUMHAN, three modules are designed to implement the three update steps in the inferred optimization equation, which correspond to the three information flows, namely measurement, intra-frame, and inter-frame information flows. In particular, a Multi-Hypothesis Aggregation (MHA) module is developed to enhance the utilization of inter-frame and intra-frame correlation by generating and fusing multiple hypothesis sets. Moreover, we propose the stage-by-stage optical flow update approach in DUMHAN to improve the accuracy of both intra-frame and inter-frame correlation modeling. The experiments demonstrate that the proposed DUMHAN achieves state-of-the-art performance.

Video Summarization With Spatiotemporal Vision Transformer.

Video summarization aims to generate a compact summary of the original video for efficient video browsing. To provide video summaries which are consistent with the human perception and contain important content, supervised learning-based video summarization methods are proposed. These methods aim to learn important content based on continuous frame information of human-created summaries. However, simultaneously considering both of inter-frame correlations among non-adjacent frames and intra-frame attention which attracts the humans for frame importance representations are rarely discussed in recent methods. To address these issues, we propose a novel transformer-based method named spatiotemporal vision transformer (STVT) for video summarization. The STVT is composed of three dominant components including the embedded sequence module, temporal inter-frame attention (TIA) encoder, and spatial intra-frame attention (SIA) encoder. The embedded sequence module generates the embedded sequence by fusing the frame embedding, index embedding and segment class embedding to represent the frames. The temporal inter-frame correlations among non-adjacent frames are learned by the TIA encoder with the multi-head self-attention scheme. Then, the spatial intra-frame attention of each frame is learned by the SIA encoder. Finally, a multi-frame loss is computed to drive the learning of the network in an end-to-end trainable manner. By simultaneously using both inter-frame and intra-frame information, our method outperforms state-of-the-art methods in both of the SumMe and TVSum datasets. The source code of the spatiotemporal vision transformer will be available at https://github.com/nchucvml/STVT.

Video Instance Segmentation by Instance Flow Assembly

Instance segmentation is a challenging task aiming at classifying and segmenting all object instances of specific classes. While two-stage box-based methods achieve top performances in the image domain, they cannot easily extend their superiority into the video domain. This is because they usually deal with features or images cropped from the detected bounding boxes without alignment, failing to capture pixel-level temporal consistency. We embrace the observation that bottom-up methods dealing with box-free features could offer accurate spacial correlations across frames, which can be fully utilized for object and pixel level tracking. We first propose our bottom-up framework equipped with a temporal context fusion module to better encode inter-frame correlations. Intra-frame cues for semantic segmentation and object localization are simultaneously extracted and reconstructed by corresponding decoders after a shared backbone. For efficient and robust tracking among instances, we introduce an instance-level correspondence across adjacent frames, which is represented by a center-to-center flow, termed as instance flow, to assemble messy dense temporal correspondences. Experiments demonstrate that the proposed method outperforms the state-of-the-art online methods (taking image-level input) on the challenging Youtube-VIS dataset [46].

A multi-stage spatio-temporal adaptive network for video super-resolution

Video super-resolution aims at restoring the spatial resolution of the reference frame based on consecutive input low-resolution (LR) frames. Existing implicit alignment-based video super-resolution methods commonly utilize convolutional LSTM (ConvLSTM) to handle sequential input frames. However, vanilla ConvLSTM processes input features and hidden states independently in operations and has limited ability to handle the inter-frame temporal redundancy in low-resolution fields. In this paper, we propose a multi-stage spatio-temporal adaptive network (MS-STAN). A spatio-temporal adaptive ConvLSTM (STAC) module is proposed to handle input features in low-resolution fields. The proposed STAC module utilizes the correlation between input features and hidden states in the ConvLSTM unit and modulates the hidden states adaptively conditioned on fused spatio-temporal features. A residual stacked bidirectional (RSB) architecture is further proposed to fully exploit the processing ability of the STAC unit. The proposed STAC and RSB architecture promote the vanilla ConvLSTM’s ability to exploit the inter-frame correlations, thus improving the reconstruction quality. Furthermore, different from existing methods that only aggregate features from the temporal branch once at a specified stage of the network, the proposed network is organized in a multi-stage manner. The corresponding temporal correlation in features at different stages can be fully exploited. Experimental results on Vimeo-90K-T and UMD10 datasets show that the proposed method has comparable performance with current video super-resolution methods. The code is available at https://github.com/yhjoker/MS-STAN.