Global Motion Estimation Research Articles

GLOCAL: A self-supervised learning framework for global and local motion estimation

Motions in videos are typically a mixture of local dynamic object motions and global camera motion, which are inconsistent in some cases, and even interfere with each other, causing difficulties in various downstream applications, such as video stabilization that requires the global motion, and action recognition that consumes local motions. Therefore, it is crucial to estimate them separately. Existing methods separate two motions from the mixed motion fields, such as optical flow. However, the quality of mixed motion determines the higher bounds of the performance. In this work, we propose a framework, GLOCAL, to directly estimate global and local motions simultaneously from adjacent frames in a self-supervised manner. Our GLOCAL consists of a Global Motion Estimation (GME) module and a Local Motion Estimation (LME) module. The GME module involves a mixed motion estimation backbone, an implicit bottleneck structure for feature dimension reduction, and an explicit bottleneck for global motion recovery based on the global motion bases with foreground mask under the training guidance of proposed global reconstruction loss. An attention U-Net is adopted for LME which produces local motions while excluding motion of irrelevant regions under the guidance of proposed local reconstruction loss. Our method can achieve better performance than the existing methods on the homography estimation dataset DHE and the action recognition dataset NCAA and UCF-101.

Accurate Detection and Localization of Individual Free Street Parking Spaces Using AI and Innovative Global Motion Estimation

Accurate Detection and Localization of Individual Free Street Parking Spaces Using AI and Innovative Global Motion Estimation

An intelligent optimization algorithm with a deep learning‐enabled block‐based motion estimation model

AbstractThe motion estimation process has gained significant attention in several video applications such as video compression, global motion estimation, and so on. The deep learning (DL) models that perform effectively in computer vision tasks can be employed for motion estimation. Yet, they have high computational complexity and model parameters since motion estimation in the pixel domain is not effective in practical applications. Instead, a block‐based motion estimation process has been developed where the blocks of sequential images are compared. To that end, this paper presents the Anas platyrhynchos optimizer with deep learning‐enabled block‐based motion estimation (APODL‐BBME) model. The proposed model estimates motion using block‐based concepts and DL approaches. To accomplish this, the training and testing frames are separated into non‐overlapping block categories, and the blocks are filtered via the bilateral filtering (BF) approach. In addition, a histogram of gradients (HOG) and densely connected network (DenseNet) model are employed to extract features, which are then fed into the bidirectional long short‐term memory (BiLSTM) model to classify the input features. Finally, the APO algorithm is applied to optimally tune the hyperparameters of the BiLSTM model, which helps to improve the overall motion estimation efficacy, showing the novelty of the work. To demonstrate the enhanced performance of the APODL‐BBME model, a comprehensive analysis is carried out, and when we compare the results, the APODL‐BBME model outperforms recent motion estimation approaches.

Behaviour Detection and Recognition of College Basketball Players Based on Multimodal Sequence Matching and Deep Neural Networks.

This study fuses multimodal sequence matching with a deep neural network algorithm for college basketball player behavior detection and recognition to conduct in-depth research and analysis, analyzing the basic components of basketball technical action videos by studying the practical application of technical actions in professional games and teaching videos from self-published authors of short video platforms. The characteristics of the dataset are also analyzed through literature research related to the basketball action dataset. On the established basketball technical action dataset, combined with the SSD target detection algorithm, the video images with cropping of human motion regions to reduce the size of image frames, generating a basketball technical action dataset based on cropped frames, reduces the amount of network training and improves the efficiency of subsequent action recognition training. In this study, by analyzing the characteristics of basic camera motion, a univariate global motion model is proposed to introduce a quadratic term to accurately express the shaking transformation, while the horizontal and vertical motion are independently represented to reduce the model complexity. Comparative experimental results show that the proposed model achieves a good balance between complexity and accuracy of global motion representation. It is suitable for global motion modeling in behavior recognition applications, laying the foundation for global and local motion estimation. On this basis, the visual feature change pattern of the key area of the scene (basketball area) is combined with the group behavior recognition based on motion patterns and the success-failure classification based on key visual information to achieve basketball semantic event recognition. The experimental results at NCAA show that the fusion of global and local motion patterns can effectively improve group behavior recognition performance. The semantic event recognition algorithm combining motion patterns and video key visual information achieves the best performance.

LiDAR Point Cloud Compression by Vertically Placed Objects Based on Global Motion Prediction

A point cloud acquired through a Light Detection And Ranging (LiDAR) sensor can be illustrated as a continuous frame with a time axis. Since the frame-by-frame point cloud has a high correlation between frames, a higher compression efficiency can be obtained by using an inter-prediction scheme, and for this purpose, Geometry-based Point Cloud Compression (G-PCC) in the Moving Picture Expert Group (MPEG) opened Inter-Exploratory Model (Inter-EM) which experiments on continuous LiDAR based point cloud frames compression through inter-prediction. The points of the LiDAR based point cloud have two different types of motion: global motion brought about by a vehicle with a LiDAR sensor and local motion generated by an object e.g., a walking person. Thus, Inter-EM consists of a compression structure in terms of both global and local motion, and the Inter-EM’s global motion compensation technology increases the compression efficiency via a single matrix describing the global motion of points. However, this is difficult to predict with a single matrix, which causes imprecise global motion estimation since the objects in a LiDAR-based point cloud show different global motion estimates according to object characteristics such as shape and position. Therefore, this paper proposes a global motion prediction and compensation scheme that considers the characteristics of objects for efficient compression of LiDAR-based point cloud frames. The proposed global motion prediction and compensation scheme achieved maximum gain of −22.0% and average of −9.4% in terms of the Bjontegaard-Delta-rate (BD-rate), and effectively compressed the LiDAR-based sparse point cloud.

Lung tumor segmentation in 4D CT images using motion convolutional neural networks.

Manual delineation on all breathing phases of lung cancer 4D CT image datasets can be challenging, exhaustive, and prone to subjective errors because of both the large number of images in the datasets and variations in the spatial location of tumors secondary to respiratory motion. The purpose of this work is to present a new deep learning-based framework for fast and accurate segmentation of lung tumors on 4D CT image sets. The proposed DL framework leverages motion region convolutional neural network (R-CNN). Through integration of global and local motion estimation network architectures, the network can learn both major and minor changes caused by tumor motion. Our network design first extracts tumor motion information by feeding 4D CT images with consecutive phases into an integrated backbone network architecture, locating volume-of-interest (VOIs) via a regional proposal network and removing irrelevant information via a regional convolutional neural network. Extracted motion information is then advanced into the subsequent global and local motion head network architecture to predict corresponding deformation vector fields (DVFs) and further adjust tumor VOIs. Binary masks of tumors are then segmented within adjusted VOIs via a mask head. A self-attention strategy is incorporated in the mask head network to remove any noisy features that might impact segmentation performance. We performed two sets of experiments. In the first experiment, a five-fold cross-validation on 20 4D CT datasets, each consisting of 10 breathing phases (i.e., 200 3D image volumes in total). The network performance was also evaluated on an additional unseen 200 3D images volumes from 20 hold-out 4D CT datasets. In the second experiment, we trained another model with 40 patients' 4D CT datasets from experiment 1 and evaluated on additional unseen nine patients' 4D CT datasets. The Dice similarity coefficient (DSC), center of mass distance (CMD), 95th percentile Hausdorff distance (HD95 ), mean surface distance (MSD), and volume difference (VD) between the manual and segmented tumor contour were computed to evaluate tumor detection and segmentation accuracy. The performance of our method was quantitatively evaluated against four different methods (VoxelMorph, U-Net, network without global and local networks, and network without attention gate strategy) across all evaluation metrics through a paired t-test. The proposed fully automated DL method yielded good overall agreement with the ground truth for contoured tumor volume and segmentation accuracy. Our model yielded significantly better values of evaluation metrics (p<0.05) than all four competing methods in both experiments. On hold-out datasets of experiment 1 and 2, our method yielded DSC of 0.86 and 0.90 compared to 0.82 and 0.87, 0.75 and 0.83, 081 and 0.89, and 0.81 and 0.89 yielded by VoxelMorph, U-Net, network without global and local networks, and networks without attention gate strategy. Tumor VD between ground truth and our method was the smallest with the value of 0.50 compared to 0.99, 1.01, 0.92, and 0.93 for between ground truth and VoxelMorph, U-Net, network without global and local networks, and networks without attention gate strategy, respectively. Our proposed DL framework of tumor segmentation on lung cancer 4D CT datasets demonstrates a significant promise for fully automated delineation. The promising results of this work provide impetus for its integration into the 4D CT treatment planning workflow to improve the accuracy and efficiency of lung radiotherapy.

Robust Global Motion Estimation for Video Stabilization Based on Improved K-Means Clustering and Superpixel.

Obtaining accurate global motion is a crucial step for video stabilization. This paper proposes a robust and simple method to implement global motion estimation. We don’t extend the framework of 2D video stabilization but add a “plug and play” module to motion estimation based on feature points. Firstly, simple linear iterative clustering (SLIC) pre-segmentation is used to obtain superpixels of the video frame, clustering is performed according to the superpixel centroid motion vector and cluster center with large value is eliminated. Secondly, in order to obtain accurate global motion estimation, an improved K-means clustering is proposed. We match the feature points of the remaining superpixels between two adjacent frames, establish a feature points’ motion vector space, and use improved K-means clustering for clustering. Finally, the richest cluster is being retained, and the global motion is obtained by homography transformation. Our proposed method has been verified on different types of videos and has efficient performance than traditional approaches. The stabilization video has an average improvement of 0.24 in the structural similarity index than the original video and 0.1 higher than the traditional method.

Global motion estimation with iterative optimization-based independent univariate model for action recognition

Motion information used in the existed video action recognition schemes is mixing of global motion(GM) and local motion(LM). In fact, GM & LM have their respective semantic concepts. Thus, it is promising to decouple GM and LM from the mixed motions. Numerous efforts have been made on the design of global motion models for video encoding, video dejittering, video denoising, and so on. Nevertheless, some of the models are too basic to cover the camera motions in action recognition while others are over-complicated. In this paper, we focus on the characteristic of the action recognition and propose a novel independent univariate GM model. It ignores camera rotation, which appears rarely in action recognition videos, and represents the GM in x and y direction respectively. Furthermore, GM is position invariant because it is from the universal camera motion. Pixels with global motions are subjected to the same parametric model and pixels with mixed motion can be seen as outliers. Motivated by this, we develop an iterative optimization scheme for GM estimation which removes the outlier points step by step and estimates global motions in a coarse-to-fine manner. Finally, the LM is estimated through a Spatio-temporal threshold-based method. Experimental results demonstrate that the proposed GM model makes a better trade-off between the model complexity and the robustness. And the iterative optimization scheme is more effective than the existed algorithms. The compared experiments using four popular action recognition models on UCF-101 (for action recognition) and NCAA (for group activity recognition) demonstrate that local motions are more effective than the mixed motions.

BlockNet: A Deep Neural Network for Block-Based Motion Estimation Using Representative Matching

Owing to the limitations of practical realizations, block-based motion is widely used as an alternative for pixel-based motion in video applications such as global motion estimation and frame rate up-conversion. We hereby present BlockNet, a compact but effective deep neural architecture for block-based motion estimation. First, BlockNet extracts rich features for a pair of input images. Then, it estimates coarse-to-fine block motion using a pyramidal structure. In each level, block-based motion is estimated using the proposed representative matching with a simple average operator. The experimental results show that BlockNet achieved a similar average end-point error with and without representative matching, whereas the proposed matching incurred 18% lower computational cost than full matching.

Research on feature point registration method for wireless multi-exposure images in mobile photography

In the mobile shooting environment, the multi-exposure is easy to occur due to the impact of the jitter and the sudden change of ambient illumination, so it is necessary to deal with the feature point registration of the multi-exposure image under mobile photography to improve the image quality. A feature point registration technique is proposed based on white balance offset compensation. The global motion estimation of the image is carried out, and the spatial neighborhood information is integrated into the amplitude detection of the multi-exposure image under mobile photography, and the amplitude characteristics of the multi-exposure image under the mobile shooting are extracted. The texture information of the multi-exposure image is compared to that of a global moving RGB 3D bit plane random field, and the white balance deviation of the multi-exposure image is compensated. At different scales, suitable white balance offset compensation function is used to describe the feature points of the multi-exposure image, the parallax analysis and corner detection of the target pixel of the multi-exposure image are carried out, and the image stabilization is realized by combining the feature registration method. The simulation results show that the proposed method has high accuracy and good registration performance for multi-exposure image feature points under mobile photography, and the image quality is improved.

ADEQUATE IMPROVEMENT FOR SPATIAL VIDEO SCALING FOR VIDEO SURVEILLANCE APPLICATIONS

Spatial Video Scaling (SVS) is a global standard method for video Surveillance Applications. The proposed Spatial Video Scaling (SVS) is designed to estimate the objects in the occupied area and its interconnected vectors. By introducing this SVS framework based interconnected motion vector used to convert the Three-dimensional (3-D) space to the two-dimensional image plane (2-D) at sufficient distance. It consists of several stages of video enhancement and shows an overview of the proposed spatial video scaling (SVS) method. The proposed SVS method is suitable for applications where the accuracy of motion estimation is vital. The proposed SVS video encoding is offered in four phases. The primary phase is the pre-processing stage; in this, unnecessary background areas were removed. Then the Local motion Estimation vectors and global motion Estimation vectors are calculated for better elements detection. Then the extraction of features with Spatial Video Scaling based video enhancement and detection technique. Finally, the performance factors is calculated based on the best features that are found. Finally, the sum of the objects in the movement of the image is smoothed, and the video is compensated.

Reconstruction of undersampled 3D non-Cartesian image-based navigators for coronary MRA using an unrolled deep learning model.

To rapidly reconstruct undersampled 3D non-Cartesian image-based navigators (iNAVs) using an unrolled deep learning (DL) model, enabling nonrigid motion correction in coronary magnetic resonance angiography (CMRA). An end-to-end unrolled network is trained to reconstruct beat-to-beat 3D iNAVs acquired during a CMRA sequence. The unrolled model incorporates a nonuniform FFT operator in TensorFlow to perform the data-consistency operation, and the regularization term is learned by a convolutional neural network (CNN) based on the proximal gradient descent algorithm. The training set includes 6,000 3D iNAVs acquired from 7 different subjects and 11 scans using a variable-density (VD) cones trajectory. For testing, 3D iNAVs from 4 additional subjects are reconstructed using the unrolled model. To validate reconstruction accuracy, global and localized motion estimates from DL model-based 3D iNAVs are compared with those extracted from 3D iNAVs reconstructed with -ESPIRiT. Then, the high-resolution coronary MRA images motion corrected with autofocusing using the -ESPIRiT and DL model-based 3D iNAVs are assessed for differences. 3D iNAVs reconstructed using the DL model-based approach and conventional -ESPIRiT generate similar global and localized motion estimates and provide equivalent coronary image quality. Reconstruction with the unrolled network completes in a fraction of the time compared to CPU and GPU implementations of -ESPIRiT (20× and 3× speed increases, respectively). We have developed a deep neural network architecture to reconstruct undersampled 3D non-Cartesian VD cones iNAVs. Our approach decreases reconstruction time for 3D iNAVs, while preserving the accuracy of nonrigid motion information offered by them for correction.

How to Estimate Global Motion Non-Iteratively From a Coarsely Sampled Motion Vector Field

Nowadays, as the amount of video content increases, the importance of global motion estimation is increasing more and more. This paper considers motion estimation within a compressed domain and presents a robust non-iterative global motion estimation algorithm that efficiently eliminates outliers. The said algorithm is comprised of two parts. The first part uses the distinct-outlier mask based on statistical analysis to eliminate the distinct outliers and uses the current-object mask based on the reference-object mask to eliminate the object outliers. In the second part, the proposed median error designed to be robust to the outlier ratio has been used to estimate weights with higher values as the motion vector gets closer to global motion. Using these weights, the proposed algorithm can accurately estimate global motion non-iteratively, although the outlier ratio may change. The simulation results demonstrate that the proposed algorithm demonstrates the highest estimation accuracy and processing speed in synthetic motion fields as well as real video sequences.

Fast global motion estimation on single instruction multiple data processors for real-time devices

Global motion estimation is a fundamental tool for image processing and computer vision. Since low-cost real-time devices have limited computational resources, the computational burden of global motion estimation remains considerable for low-cost devices. Hence, a fast global motion estimation algorithm for single instruction multiple data (SIMD) processors with which a CPU in modern mobile computing devices is commonly equipped is proposed. Conventional projection-based global motion estimation is modified to improve the prediction accuracy for videos containing large motions. To significantly increase the parallelism, the exact sizes of the variables in each processing step of the proposed algorithm are determined by considering the size of an SIMD register. Simulation results demonstrate that the proposed scheme is considerably fast and accurate, even for videos containing large motions.

Multi-frame super-resolution reconstruction based on global motion estimation using a novel CNN descriptor

In this paper, we introduce a novel feature descriptor based on deep learning that trains a model to match the patches of images on scenes captured under different viewpoints and lighting conditions for Multi-frame super-resolution. The patch matching of images capturing the same scene in varied circumstances and diverse manners is challenging. We develop a model which maps the raw image patch to a low dimensional feature vector. As our experiments show, the proposed approach is much better than state-of-the-art descriptors and can be considered as a direct replacement of SURF. The results confirm that these techniques further improve the performance of the proposed descriptor. Then we propose an improved Random Sample Consensus algorithm for removing false matching points. Finally, we show that our neural network based image descriptor for image patch matching outperforms state-of-the-art methods on a number of benchmark datasets and can be used for image registration with high quality in multi-frame super-resolution reconstruction.

Subpixel real-time jitter detection algorithm and implementation for polarimetric and helioseismic imager

The polarimetric and helioseismic imager instrument for the Solar Orbiter mission from the European Space Agency requires a high stability while capturing images, specially for the polarimetric ones. For this reason, an image stabilization system has been included in the instrument. It uses global motion estimation techniques to estimate the jitter in real time with subpixel resolution. Due to instrument requirements, the algorithm has to be implemented in a Xilinx Virtex-4QV field programmable gate array. The algorithm includes a 2-D paraboloid interpolation algorithm based on 2-D bisection. We describe the algorithm implementation and the tests that have been made to verify its performance. The jitter estimation has a mean error of 125 pixel of the correlation tracking camera. The paraboloid interpolation algorithm provides also better results in terms of resources and time required for the calculation (at least a 20% improvement in both cases) than those based on direct calculation.

A Hardware–Software Co-Design Framework for Real-Time Video Stabilization

Live digital video is a valuable source of information in security, broadcast and industrial quality control applications. Motion jitter due to camera and platform instability is a common artefact found in captured video which renders it less effective for subsequent computer vision tasks such as detection and tracking of objects, background modeling, mosaicking, etc. The process of algorithmically compensating for the motion jitter is hence a mandatory pre-processing step in many applications. This process, called video stabilization, requires estimation of global motion from consecutive video frames and is constrainted by additional challenges such as preservation of intentional motion and native frame resolution. The problem is exacerbated in the presence of local motion of foreground objects and requires robust compensation of the same. As such achieving real-time performance for this computationally intensive operation is a difficult task for embedded processors with limited computational and memory resources. In this work, development of an optimized hardware–software co-design framework for video stabilization has been investigated. Efficient video stabilization depends on the identification of key points in the frame which in turn requires dense feature calculation at the pixel level. This task has been identified to be most suitable for offloading the pipelined hardware implemented in the FPGA fabric due to the involvement of complex memory and computation operations. Subsequent tasks to be performed for the overall stabilization algorithm utilize these sparse key points and have been found to be efficiently handled in the software. The proposed Hardware–Software (HW–SW) co-design framework has been implemented on Zedboard FPGA platform which houses Xilinx Zynq SOC equipped with ARM A9 processor. The proposed implementation scheme can process real-time video stream input at 28 frames per second and is at least twice faster than the corresponding software-only approach. Two different hardware accelerator designs have been implemented using different high-level synthesis tools using rapid prototyping principle and consume less than 50% of logic resources available on the host FPGA while being at least 30% faster than contemporary designs.

Dense motion estimation of particle images via a convolutional neural network

In this paper, we propose a supervised learning strategy for the fluid motion estimation problem (i.e., extracting the velocity fields from particle images). The purpose of this work is to design a convolutional neural network (CNN) for estimating dense motion field for particle image velocimetry (PIV), which allows to improve the computational efficiency without reducing the accuracy. First, the network model is developed based on FlowNetS, which is recently proposed for end-to-end optical flow estimation in the computer vision community. The input of the network is a particle image pair and the output is a velocity field with displacement vectors at every pixel. Second, a synthetic dataset of fluid flow images is generated to train the CNN model. To our knowledge, this is the first time a CNN has been used as a global motion estimator for particle image velocimetry. Experimental evaluations indicate that the trained CNN model can provide satisfactory results in both artificial and laboratory PIV images. The proposed estimator is also applied to the experiment of turbulent boundary layer. In addition, the computational efficiency of the CNN estimator is much superior to those of the traditional cross-correction and optical flow methods.

Video stabilization performance enhancement for low-texture videos

Digital video stabilization (DVS) aims to remove irregular global motion effects from an image sequence. This work aims at developing a real-time video stabilization algorithm for rectifying high-frequency jitter in marine surveillance applications. A DVS system consists of a global motion estimation system and motion correction system. The development of global motion estimation system resistant to failures in low texture videos is the primary goal. Due to the computational advantage and inherent properties, the phase correlation method is adopted as the basic global motion estimation algorithm. The basic algorithm is then modified to adapt to the varying texture content of the video sequences under consideration. An adaptive phase correlation-based global motion estimation is suggested and verified on the videos of varying textures.

A Robust and Efficient Video Anti-Shaking Algorithm for Low-End Smartphone Platforms

Feature phones or low-end smart phones are an important part of our daily life. While lacking the powerful embedded camera and processing capability of the high-end smart phone, the feature phones with the low-end camera and processing capability are still widely used owing to its lower cost. Consequently, image and video processing algorithms are needed for many intelligent applications on the low-end smart phones. In this paper, we propose a novel robust and efficient video anti-shaking algorithm for the low-end smartphone platform. It provides an integrated application for the low-end smartphone platform, which includes a novel global motion estimation method, and a rapid and efficient local motion detection method. The overall framework is proposed considering the limited camera perception quality and limited mobile processing capability of the feature phone. The proposed algorithm can not only obtain a good visual effect but also be implemented in real-time for the low-end mobile platforms with very limited computational resource. Experimental results demonstrate that the proposed method can achieve a similar video stabilization effect to the Optical Image Stabilization hardware in high-end smartphones.