Motion Estimation Module Research Articles

UVC: An Unified Deep Video Compression Framework

Recently, many works have applied deep learning techniques to video compression tasks, achieving promising results and advancing the field of deep learning video compression (DLVC). However, the architecture design of existing DLVC is rigid and limited in terms of flexibility. Specifically, different networks must be designed for different scenarios, such as delay-constrained or non-delay-constrained scenarios. Frequent switching between networks would reduce the speed of modern deep learning platforms and increase maintenance costs. To address this problem, we propose a unified deep video compression (UVC) framework that can be freely switched to different application scenarios without changing the network architecture. Our proposed UVC framework is based on the explicit-compression and implicit-generation perspective, which contains two sub-networks: the explicit reference frame compression network (ERFCN) and the implicit reference frame generation network (IRFGN). The aim of ERFCN is to compress the current frame with the help of the reference frame. To improve the performance of ERFCN, we first introduce the Transformer in this network, which can fully remove the spatial redundancy of the input image and is beneficial for the following inter-prediction process. We also develop a novel long-range motion estimation module for inter-prediction to generate motion vectors based on global motion information between two frames, which can handle long-range complex motion relations. The aim of IRFGN is to capture the temporal relationship between forward and backward reconstructed frames and synthesize a high-quality implicit reference frame for the current frame. To achieve this, we design the split spatial-temporal attention and multi-scale prediction module. We conduct extensive experiments on three widely used video compression databases (HEVC, UVG, and MCL-JVC), and the results demonstrate the superiority of our approach over other related DLVC methods.

A fast and efficient data reuse scheme for HEVC Integer Motion Estimation hardware architecture

ABSTRACT High-Efficiency Video Coding (HEVC) is a video compression standard designed to improve video compression efficiency compared to its predecessor, Advanced Video Coding (AVC). HEVC provides approximately twice the compression efficiency compared to the AVC. In the HEVC system, the most computational complexity module is the Motion Estimation (ME). ME module helps reduce redundancy by compensating for motion during video compression. However, the computational complexity makes it a bottleneck in the design of high-resolution video encoders. This paper proposes an efficient architecture for the Integer ME (IME) module of the HEVC encoder. The proposed architecture introduces an efficient memory usage scheme to support the Full Search motion search algorithm. The full pipeline architecture includes 4096 Processing Elements (PE) and an adder tree to compute the Sum of Absolute Difference (SAD) for every Prediction Unit (PU) partition. The proposed architecture was designed and implemented on Xilinx Virtex-7 XC7VX550T FPGA. Our design achieved up to 275 FPS and approximately 10 FPS at 4 K video for the Search Regions of 32 × 16 and 128 × 128 pixels, respectively.

Self-supervised rigid object 3-D motion estimation from monocular video

Independent object 3D motion estimation is a fundamental problem in 3D computer vision. Directly segmenting and estimating rigid object 3D motion from a consistent video frame is an ill-posed problem. We present a self-supervised framework for segmenting moving independent rigid objects and estimating their motion information including location, driving direction, and speed from a monocular video. Specifically, we first estimate depth, optical flow, and camera pose between a pair of video frames, and then synthesize a new 3D viewpoint from this pair. Subsequently, the Motion Recurrent All-Pairs Field Transforms (MRAFT) module is introduced to extract 3D scene flow and a motion area binary mask from a pair of images and depth. After that, a Rigid Object Motion Estimation Module (ROMEM) with a slot attention mechanism is proposed to extract rigid object motion masks from a multi-layer motion field, including optical flow, depth changes, refined scene flow, and motion masks. Finally, 2D images and 3D scene reconstruction errors are used to facilitate self-supervised training for rigid object motion. Experiments on the FlyingThings3D and KITTI datasets demonstrate that our method outperforms other advanced algorithms in estimating depth, optical flow, scene flow, and rigid moving object masks, demonstrating the benefits of our approach.

Improving Audio-Visual Segmentation with Bidirectional Generation

The aim of audio-visual segmentation (AVS) is to precisely differentiate audible objects within videos down to the pixel level. Traditional approaches often tackle this challenge by combining information from various modalities, where the contribution of each modality is implicitly or explicitly modeled. Nevertheless, the interconnections between different modalities tend to be overlooked in audio-visual modeling. In this paper, inspired by the human ability to mentally simulate the sound of an object and its visual appearance, we introduce a bidirectional generation framework. This framework establishes robust correlations between an object's visual characteristics and its associated sound, thereby enhancing the performance of AVS. To achieve this, we employ a visual-to-audio projection component that reconstructs audio features from object segmentation masks and minimizes reconstruction errors. Moreover, recognizing that many sounds are linked to object movements, we introduce an implicit volumetric motion estimation module to handle temporal dynamics that may be challenging to capture using conventional optical flow methods. To showcase the effectiveness of our approach, we conduct comprehensive experiments and analyses on the widely recognized AVSBench benchmark. As a result, we establish a new state-of-the-art performance level in the AVS benchmark, particularly excelling in the challenging MS3 subset which involves segmenting multiple sound sources. Code is released in: https://github.com/OpenNLPLab/AVS-bidirectional.

GLOCAL: A self-supervised learning framework for global and local motion estimation

Motions in videos are typically a mixture of local dynamic object motions and global camera motion, which are inconsistent in some cases, and even interfere with each other, causing difficulties in various downstream applications, such as video stabilization that requires the global motion, and action recognition that consumes local motions. Therefore, it is crucial to estimate them separately. Existing methods separate two motions from the mixed motion fields, such as optical flow. However, the quality of mixed motion determines the higher bounds of the performance. In this work, we propose a framework, GLOCAL, to directly estimate global and local motions simultaneously from adjacent frames in a self-supervised manner. Our GLOCAL consists of a Global Motion Estimation (GME) module and a Local Motion Estimation (LME) module. The GME module involves a mixed motion estimation backbone, an implicit bottleneck structure for feature dimension reduction, and an explicit bottleneck for global motion recovery based on the global motion bases with foreground mask under the training guidance of proposed global reconstruction loss. An attention U-Net is adopted for LME which produces local motions while excluding motion of irrelevant regions under the guidance of proposed local reconstruction loss. Our method can achieve better performance than the existing methods on the homography estimation dataset DHE and the action recognition dataset NCAA and UCF-101.

Fast Fluid Simulation via Dynamic Multi-Scale Gridding

Recent works on learning-based frameworks for Lagrangian (i.e., particle-based) fluid simulation, though bypassing iterative pressure projection via efficient convolution operators, are still time-consuming due to excessive amount of particles. To address this challenge, we propose a dynamic multi-scale gridding method to reduce the magnitude of elements that have to be processed, by observing repeated particle motion patterns within certain consistent regions. Specifically, we hierarchically generate multi-scale micelles in Euclidean space by grouping particles that share similar motion patterns/characteristics based on super-light motion and scale estimation modules. With little internal motion variation, each micelle is modeled as a single rigid body with convolution only applied to a single representative particle. In addition, a distance-based interpolation is conducted to propagate relative motion message among micelles. With our efficient design, the network produces high visual fidelity fluid simulations with the inference time to be only 4.24 ms/frame (with 6K fluid particles), hence enables real-time human-computer interaction and animation. Experimental results on multiple datasets show that our work achieves great simulation acceleration with negligible prediction error increase.

Toward Fine-Grained Talking Face Generation.

Talking face generation is the process of synthesizing a lip-synchronized video when given a reference portrait and an audio clip. However, generating a fine-grained talking video is nontrivial due to several challenges: 1) capturing vivid facial expressions, such as muscle movements; 2) ensuring smooth transitions between consecutive frames; and 3) preserving the details of the reference portrait. Existing efforts have only focused on modeling rigid lip movements, resulting in low-fidelity videos with jerky facial muscle deformations. To address these challenges, we propose a novel Fine-gRained mOtioN moDel (FROND), consisting of three components. In the first component, we adopt a two-stream encoder to capture local facial movement keypoints and embed their overall motion context as the global code. In the second component, we design a motion estimation module to predict audio-driven movements. This enables the learning of local key point motion in the continuous trajectory space to achieve smooth temporal facial movements. Additionally, the local and global motions are fused to estimate a continuous dense motion field, resulting in spatially smooth movements. In the third component, we devise a novel implicit image decoder based on an implicit neural network. This decoder recovers high-frequency information from the input image, resulting in a high-fidelity talking face. In summary, the FROND refines the motion trajectories of facial keypoints into a continuous dense motion field, which is followed by a decoder that fully exploits the inherent smoothness of the motion. We conduct quantitative and qualitative model evaluations on benchmark datasets. The experimental results show that our proposed FROND significantly outperforms several state-of-the-art baselines.

Spike-based Motion Estimation for Object Tracking through Bio-inspired Unsupervised Learning.

Neuromorphic vision sensors, whose pixels output events/spikes asynchronously with a high temporal resolution according to the scene radiance change, are naturally appropriate for capturing high-speed motion in the scenes. However, how to utilize the events/spikes to smoothly track high-speed moving objects is still a challenging problem. Existing approaches either employ time-consuming iterative optimization, or require large amounts of labeled data to train the object detector. To this end, we propose a bio-inspired unsupervised learning framework, which takes advantage of the spatiotemporal information of events/spikes generated by neuromorphic vision sensors to capture the intrinsic motion patterns. Without off-line training, our models can filter the redundant signals with dynamic adaption module based on short-term plasticity, and extract the motion patterns with motion estimation module based on the spike-timing-dependent plasticity. Combined with the spatiotemporal and motion information of the filtered spike stream, the traditional DBSCAN clustering algorithm and Kalman filter can effectively track multiple targets in extreme scenes. We evaluate the proposed unsupervised framework for object detection and tracking tasks on synthetic data, publicly available event-based datasets, and spiking camera datasets. The experiment results show that the proposed model can robustly detect and smoothly track the moving targets on various challenging scenarios and outperforms state-of-the-art approaches.

Deep HDR Deghosting by Motion-Attention Fusion Network.

Multi-exposure image fusion (MEF) methods for high dynamic range (HDR) imaging suffer from ghosting artifacts when dealing with moving objects in dynamic scenes. The state-of-the-art methods use optical flow to align low dynamic range (LDR) images before merging, introducing distortion into the aligned LDR images from inaccurate motion estimation due to large motion and occlusion. In place of pre-alignment, attention-based methods calculate the correlation between the reference LDR image and non-reference LDR images, thus excluding misaligned regions in LDR images. Nevertheless, they also exclude the saturated details at the same time. Taking advantage of both the alignment and attention-based methods, we propose an efficient Deep HDR Deghosting Fusion Network (DDFNet) guided by optical flow and image correlation attentions. Specifically, the DDFNet estimates the optical flow of the LDR images by a motion estimation module and encodes that optical flow as a flow feature. Additionally, it extracts correlation features between the reference LDR and other non-reference LDR images. The optical flow and correlation features are employed to adaptably combine information from LDR inputs in an attention-based fusion module. Following the merging of features, a decoder composed of Dense Networks reconstructs the HDR image without ghosting. Experimental results indicate that the proposed DDFNet achieves state-of-the-art image fusion performance on different public datasets.

Robust consensus-aware network for 3D point registration

Outlier correspondence removal is an important task for feature-based point cloud registration. Given putative correspondences contaminated by outliers between two overlapped scans, we propose a Robust Consensus-Aware Network, which labels the correspondences as inliers or outliers and predicts the rigid transformation to align the point clouds. The proposed method dedicates to mining the global consensus of correct correspondences (inliers). So it can learn distinctive features for each correspondence. Specifically, the proposed network comprises three novel operations. First, by capturing the global consensus information in an attentive manner, the network projects the input correspondences into a discriminative feature space. Next, we exploit the feature similarity among correspondences to establish interactions within inlier or outlier correspondences, and aggregate the features of correct correspondences for outlier removal. Finally, we recover the rigid transformation by mining multi-level context with a motion estimation module. Extensive experiments on real-world datasets demonstrate that our approach achieves high registration accuracy and efficiency.

ProgressiveMotionSeg: Mutually Reinforced Framework for Event-Based Motion Segmentation

Dynamic Vision Sensor (DVS) can asynchronously output the events reflecting apparent motion of objects with microsecond resolution, and shows great application potential in monitoring and other fields. However, the output event stream of existing DVS inevitably contains background activity noise (BA noise) due to dark current and junction leakage current, which will affect the temporal correlation of objects, resulting in deteriorated motion estimation performance. Particularly, the existing filter-based denoising methods cannot be directly applied to suppress the noise in event stream, since there is no spatial correlation. To address this issue, this paper presents a novel progressive framework, in which a Motion Estimation (ME) module and an Event Denoising (ED) module are jointly optimized in a mutually reinforced manner. Specifically, based on the maximum sharpness criterion, ME module divides the input event into several segments by adaptive clustering in a motion compensating warp field, and captures the temporal correlation of event stream according to the clustered motion parameters. Taking temporal correlation as guidance, ED module calculates the confidence that each event belongs to real activity events, and transmits it to ME module to update energy function of motion segmentation for noise suppression. The two steps are iteratively updated until stable motion segmentation results are obtained. Extensive experimental results on both synthetic and real datasets demonstrate the superiority of our proposed approaches against the State-Of-The-Art (SOTA) methods.

MCP-Net: Inter-frame Motion Correction with Patlak Regularization for Whole-body Dynamic PET.

Inter-frame patient motion introduces spatial misalignment and degrades parametric imaging in whole-body dynamic positron emission tomography (PET). Most current deep learning inter-frame motion correction works consider only the image registration problem, ignoring tracer kinetics. We propose an inter-frame Motion Correction framework with Patlak regularization (MCP-Net) to directly optimize the Patlak fitting error and further improve model performance. The MCP-Net contains three modules: a motion estimation module consisting of a multiple-frame 3-D U-Net with a convolutional long short-term memory layer combined at the bottleneck; an image warping module that performs spatial transformation; and an analytical Patlak module that estimates Patlak fitting with the motion-corrected frames and the individual input function. A Patlak loss penalization term using mean squared percentage fitting error is introduced to the loss function in addition to image similarity measurement and displacement gradient loss. Following motion correction, the parametric images were generated by standard Patlak analysis. Compared with both traditional and deep learning benchmarks, our network further corrected the residual spatial mismatch in the dynamic frames, improved the spatial alignment of Patlak Ki/Vb images, and reduced normalized fitting error. With the utilization of tracer dynamics and enhanced network performance, MCP-Net has the potential for further improving the quantitative accuracy of dynamic PET. Our code is released at https://github.com/gxq1998/MCP-Net.

DSTnet: Deformable Spatio-Temporal Convolutional Residual Network for Video Super-Resolution

Video super-resolution (VSR) aims at generating high-resolution (HR) video frames with plausible and temporally consistent details using their low-resolution (LR) counterparts, and neighboring frames. The key challenge for VSR lies in the effective exploitation of intra-frame spatial relation and temporal dependency between consecutive frames. Many existing techniques utilize spatial and temporal information separately and compensate motion via alignment. These methods cannot fully exploit the spatio-temporal information that significantly affects the quality of resultant HR videos. In this work, a novel deformable spatio-temporal convolutional residual network (DSTnet) is proposed to overcome the issues of separate motion estimation and compensation methods for VSR. The proposed framework consists of 3D convolutional residual blocks decomposed into spatial and temporal (2+1) D streams. This decomposition can simultaneously utilize input video’s spatial and temporal features without a separate motion estimation and compensation module. Furthermore, the deformable convolution layers have been used in the proposed model that enhances its motion-awareness capability. Our contribution is twofold; firstly, the proposed approach can overcome the challenges in modeling complex motions by efficiently using spatio-temporal information. Secondly, the proposed model has fewer parameters to learn than state-of-the-art methods, making it a computationally lean and efficient framework for VSR. Experiments are conducted on a benchmark Vid4 dataset to evaluate the efficacy of the proposed approach. The results demonstrate that the proposed approach achieves superior quantitative and qualitative performance compared to the state-of-the-art methods.

Design and Implementation of an Efficient Multi-Pattern Motion Estimation Search Algorithm for HEVC/H.265

Motion Estimation (ME) is the most computationally intensive block in high efficiency video coding (HEVC) due to its complex partition schemes. It consumes a large amount of power in the encoder. So designing a ME module on hardware platform imposes significant challenges. Many hardware oriented integer ME (IME) algorithms have been proposed in the literature to meet the challenges but suffer from coding efficiency degradation. In this paper we propose a low complexity IME algorithm and its hardware implementation. The proposed algorithm has been shown for two different pattern structures (PS) with 38 search points. It achieves bjontegaard bitrate (BD-BR) of 0.5% decrease and 0.077% increase for two PSs compared to test zone search (TZS) in HM 16.8. It results 8.725% and 9.072% saving in encoding time respectively. The proposed architecture supports all the HEVC partitions. It provides real time encoding of 4096 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times $ </tex-math></inline-formula> 2160 @60 fps operated at the maximum frequency of 353 MHz with 1591 K NAND equivalent gate count and 8.32 kB on chip memory using 90 nm technology library. Therefore, it can be used in consumer electronics applications such as security surveillance, video recording systems etc. that require high efficient HEVC encoder.

Saliency Guided Visual Tracking via Correlation Filter With Log-Gabor Filter

Correlation filter (CF) based tracking algorithms have tremendously contributed to the field of visual tracking due to the high computational efficiency and competitive performance. Nonetheless, most CF-based trackers are vulnerable to the influence of occlusion and boundary effect, which results in suboptimal performance. In this article, we propose saliency guided visual tracking via correlation filter with log-Gabor filter to robustify its performance under occlusion and boundary effect challenges. Firstly, we propose the CF with log-Gabor filter to get a robust appearance model. The log-Gabor filter is adopted to preprocess the sequence to gain the log-Gabor feature, which provides important cues for tracking since it encodes the texture information. Secondly, considering the prior information, we embed the novel saliency guided adaptive spatial feature selection to filter learning to preserve the spatial structure in the lower manifold and mitigate boundary distortion. Thirdly, the occlusion estimating strategy, performing on-line evaluation of tracking, triggers the motion estimation module to optimize the optimal location. Experiments on benchmark databases demonstrate the enhanced discrimination and interpretability of the proposed tracker and its superiority over other trackers.

Dense 3D-Convolutional Neural Network for Person Re-Identification in Videos

Person re-identification aims at identifying a certain pedestrian across non-overlapping multi-camera networks in different time and places. Existing person re-identification approaches mainly focus on matching pedestrians on images; however, little attention has been paid to re-identify pedestrians in videos. Compared to images, video clips contain motion patterns of pedestrians, which is crucial to person re-identification. Moreover, consecutive video frames present pedestrian appearance with different body poses and from different viewpoints, providing valuable information toward addressing the challenge of pose variation, occlusion, and viewpoint change, and so on. In this article, we propose a Dense 3D-Convolutional Network (D3DNet) to jointly learn spatio-temporal and appearance representation for person re-identification in videos. The D3DNet consists of multiple three-dimensional (3D) dense blocks and transition layers. The 3D dense blocks enlarge the receptive fields of visual neurons in both spatial and temporal dimensions, leading to discriminative appearance representation as well as short-term and long-term motion patterns of pedestrians without the requirement of an additional motion estimation module. Moreover, we formulate a loss function consisting of an identification loss and a center loss to minimize intra-class variance and maximize inter-class variance simultaneously, toward addressing the challenge of large intra-class variance and small inter-class variance. Extensive experiments on two real-world video datasets of person identification, i.e., MARS and iLIDS-VID, have shown the effectiveness of the proposed approach.

Parallel implementations of frame rate up-conversion algorithm using OpenCL on heterogeneous computing devices

As a video post-processing technology, frame rate up-conversion (FRUC) converts a low frame rate video into a higher one by inserting intermediate frames between adjacent original frames. Because computing consumption grows rapidly with the increase of video resolution and frame rate, accelerating FRUC by parallel computing may serve as an appropriate method. In this paper, an effective parallel FRUC algorithm is proposed, which consists mainly of two parts: parallel motion estimation algorithm (Three-dimensional Recursive Search algorithm, 3DRS algorithm) and parallel motion compensation algorithm. We design macro-block-level parallelism and candidate motion vector level parallelism strategies based on different granularity in the motion estimation module, and pixel-level parallelism in the motion compensation module. The proposed parallel FRUC algorithm has been tested on different hardware platforms. The results show that the method achieves significant speedups of up to 96× for 1920 × 1080 video and 254× for 3840 × 2160 video when compared with sequential implementation on CPU. Moreover, the OpenCL program of the parallel FRUC algorithm shows good portability on various GPU platforms.

Acceleration of the integer motion estimation in JEM through pre-analysis techniques

ITU-T Video Coding Expert Group and ISO/IEC Moving Picture Expert Group are studying the potential need for standardization of the future video coding technology with a compression capability that significantly exceeds that of the current High Efficiency Video Coding (HEVC) standard, including its current extensions. Both groups are working together on this exploration activity in a collaboration effort known as Joint Video Exploration Team to evaluate compression technology designs proposed by their experts in this area. Preliminary results show that the new model achieves 25% bitrate reduction, but at a cost of extremely high computational complexity (11 $$\times {}$$ ) with respect to HEVC. This paper proposes a pre-analysis algorithm designed to extract motion information of a frame, which is later used in the Motion Estimation (ME) module to speed up the encoder, showing that around 27% of the reference frames can be skipped and that more than 62% of the time is saved in the integer ME operation with a negligible impact of 0.11% in BD-rate.

A Control Architecture for Time‐Optimal Landing of a Quadrotor Onto a Moving Platform

AbstractWe address the problem of autonomous landing of a quadrotor onto a heaving (moving vertically) platform in this paper. A control architecture that consists of a motion estimation module, a trajectory generation module and a tracking control module is proposed. The motion estimation module estimates the absolute motion of the platform and the quadrotor with the measurements from an on‐board accelerometer and vision measurements. Based on these estimates, the trajectory generation module generates a time‐optimal reference trajectory. With the reference trajectory and motion estimation, the tracking control module synthesizes a control command that enables robust tracking of the reference trajectory. Experimental results and comparison with a state‐of‐the‐art landing controller demonstrate the effectiveness of the proposed control architecture.

Fast coding unit selection and motion estimation algorithm based on early detection of zero block quantified transform coefficients for high‐efficiency video coding standard

High-efficiency video coding (HEVC) is the newest video coding standard developed by the joint video team, consisting of ITU-T video coding experts group and ISO/IEC Moving Picture Experts Group. The HEVC standard has aggregated an exhaustive algorithm for mode decision based on a recursive quad-tree structured coding tree block. Moreover, several specific features have been incorporated into the motion estimation (ME) process to improve its coding efficiency. However, they resulted in very high computational complexity. To accelerate the encoding process, fast mode decision algorithms for the partitioning module and also for the ME module were proposed in this study. These algorithms are based on early zero block detection technique. To improve the efficiency of these algorithms, an overall algorithm which combines the two techniques has been implemented. The performance of the proposed algorithm was checked through a comparative analysis in terms of encoding time and compression rate. Compared to HEVC test model 10.0, the authors’ proposed algorithms bring a great reduction of the HEVC complexity encoder with a saving time, which can reach 25% in average for different tested videos and a slight coding loss in terms of image quality and compression rate.