Motion Estimation Process Research Articles

Real-time performance-driven finger motion synthesis

This paper presents a method to estimate and synthesize the motion of a character’s fingers in real-time during the performance capture process. For the motion estimation and synthesis process, different motion datasets are used that contain the full-body motion of a character, including the motion of its fingers. The motion datasets have been pre-processed for being efficiently handled in real-time estimation process. During the runtime of the application, the system recognizes the actions of the user’s hands and assembles the necessary motion of the character’s fingers. By using a hierarchical Hidden Markov Model (HMM), the system learns the phase of the gestures as well as the progress of the motion. To eliminate the searching process of the most probable motion, prior constraints between segment states were assigned manually. During the runtime of our application, by using a forward HMM algorithm, the system synthesizes the necessary motion of a character’s fingers in real-time. The presented methodology is evaluated both numerically (system performance and estimation rate) and perceptually. The results show that, even when a reduced number of finger gestures are used, the synthesized motion can be described as perceptually consistent with the ground truth motion data.

Efficient motion estimation algorithm on the layers

The two-layer motion estimation (TME) algorithm searches the motion vectors on the layers with partial distortion measures to boost the computational speed of motion estimation. An efficient motion estimation algorithm is presented in which a fast partial distortion calculation method (FPDCM) has been developed to speed up the motion estimation process on the layers without affecting motion prediction quality. This FPDCM efficiently calculates the partial distortions on the layers faster by removing the redundant computations involved in the partial distortion calculation. As a result, the speed with motion estimation on the layers by using FPDCM is increased when compared with the TME algorithm. Experimental results are given to show the superiority of the proposed algorithm over TME as well as the state-of-the-art fast block matching algorithms.

Sensor fusion of camera, GPS and IMU using fuzzy adaptive multiple motion models

A tracking system that will be used for Augmented Reality (AR) applications has two main requirements: accuracy and frame rate. The first requirement is related to the performance of the pose estimation algorithm and how accurately the tracking system can find the position and orientation of the user in the environment. Accuracy problems of current tracking devices, considering that they are low-cost devices, cause static errors during this motion estimation process. The second requirement is related to dynamic errors (the end-to-end system delay; occurring because of the delay in estimating the motion of the user and displaying images based on this estimate. This paper investigates combining the vision-based estimates with measurements from other sensors, GPS and IMU, in order to improve the tracking accuracy in outdoor environments. The idea of using Fuzzy Adaptive Multiple Models (FAMM) was investigated using a novel fuzzy rule-based approach to decide on the model that results in improved accuracy and faster convergence for the fusion filter. Results show that the developed tracking system is more accurate than a conventional GPS-IMU fusion approach due to additional estimates from a camera and fuzzy motion models. The paper also presents an application in cultural heritage context.

Context-adaptive based CU processing for 3D-HEVC.

The 3D High Efficiency Video Coding (3D-HEVC) standard aims to code 3D videos that usually contain multi-view texture videos and its corresponding depth information. It inherits the same quadtree prediction structure of HEVC to code both texture videos and depth maps. Each coding unit (CU) allows recursively splitting into four equal sub-CUs. At each CU depth level, it enables 10 types of inter modes and 35 types of intra modes in inter frames. Furthermore, the inter-view prediction tools are applied to each view in the test model of 3D-HEVC (HTM), which uses variable size disparity-compensated prediction to exploit inter-view correlation within neighbor views. It also exploits redundancies between a texture video and its associated depth using inter-component coding tools. These achieve the highest coding efficiency to code 3D videos but require a very high computational complexity. In this paper, we propose a context-adaptive based fast CU processing algorithm to jointly optimize the most complex components of HTM including CU depth level decision, mode decision, motion estimation (ME) and disparity estimation (DE) processes. It is based on the hypothesis that the optimal CU depth level, prediction mode and motion vector of a CU are correlated with those from spatiotemporal, inter-view and inter-component neighboring CUs. We analyze the video content based on coding information from neighboring CUs and early predict each CU into one of five categories i.e., DE-omitted CU, ME-DE-omitted CU, SPLIT CU, Non-SPLIT CU and normal CU, and then each type of CU adaptively adopts different processing strategies. Experimental results show that the proposed algorithm saves 70% encoder runtime on average with only a 0.1% BD-rate increase on coded views and 0.8% BD-rate increase on synthesized views. Our algorithm outperforms the state-of-the-art algorithms in terms of coding time saving or with better RD performance.

Two-bit transform using local binary pattern method for low-complexity block motion estimation

Low bit-depth representation-based motion estimation approaches have been drawing considerable attention recently, mainly because of their small hardware footprint. In this paper, a new two-bit transform using a local binary pattern (LBP) for low-complexity motion estimation is proposed. The proposed approach utilizes the LBP method to obtain two-bit representations of video frames in the binarization process. Video frames are transformed into their low bit-depth representations by the LBP and then a motion estimation process is carried out using these binary frames. A Boolean exclusive-OR operation is used to calculate the number of nonmatching points metric instead of the conventional sum of absolute differences metric in the motion estimation stage. The proposed method reduces the computational complexity, especially in the binarization stage, while improving the motion estimation accuracy compared to existing one-bit and two-bit transform-based low-complexity motion estimation approaches in the literature.

GPU-Based Fluid Motion Estimation Using Energy Constraint

Although motion estimation (ME) approaches for fluid flows have been widely studied in computer vision domain, most existing ME algorithms cannot accurately deal with regions with both slight and drastic brightness changes. To address this issue, this paper introduces a novel data structure called brightness distribution matrix (BDM) which can be used to accurately model regional brightness. Based on our proposed consistency constraints and energy function, we can obtain motion vectors from image sequences with high accuracy. Since the BDM-based ME approach requires a large number of computations when dealing with complex fluid scenarios, to reduce the overall ME time, a parallelized version of our approach is developed based on graphics processing unit (GPU). Experimental results show that our GPU-based approach not only can be used to improve the ME quality for complex fluid images, but also can reduce the overall ME processing time (up to 7.06 times improvement).

Fast Motion Estimation Based on Content Property for Low-Complexity H.265/HEVC Encoder

The high definition (HD) and ultra HD videos can be widely applied in broadcasting applications. However, with the increased resolution of video, the volume of the raw HD visual information data increases significantly, which becomes a challenge for storage, processing, and transmitting the HD visual data. The state-of-the-art video compression standard-H.265/High Efficiency Video Coding (HEVC) compresses the raw HD visual data efficiently, while the high compression rate comes at the cost of heavy computation load. Hence, reducing the encoding complexity becomes vital for the H.265/HEVC encoder to be used in broadcasting applications. In this paper, based on the best motion vector selection correlation among the different size prediction modes, we propose a fast motion estimation (ME) method to reduce the encoding complexity of the H.265/HEVC encoder. First, according to the prediction unit (PU) partition type, all PUs are classified into two classes, parent PU and children PUs, respectively. Then, based on the best motion vector selection correlation between the parent PU and children PUs, the block matching search process of the children PUs is adaptively skipped if their parent PU chooses the initial search point as its final optimal motion vector in the ME process. Experimental results show that the proposed method achieves an average of 20% ME time saving as compared with the original HM-TZSearch. Meanwhile, the rate distortion performance degradation is negligible.

Multi-reference combinatorial strategy towards longer long-term dense motion estimation

This paper addresses the estimation of accurate long-term dense motion fields from videos of complex scenes. With computer vision applications such as video editing in mind, we exploit optical flows estimated with various inter-frame distances and combine them through multi-step integration and statistical selection (MISS). In this context, managing numerous combinations of multi-step optical flows requires a complexity reduction scheme to overcome computational and memory issues. Our contribution is two-fold. First, we provide an exhaustive analysis of available single-reference complexity reduction strategies. Second, we propose a simple and efficient alternative related to multi-reference frames multi-step integration and statistical selection (MR-MISS). Our method automatically inserts intermediate reference frames once matching failures are detected to re-generate the motion estimation process and re-correlates the resulting dense trajectories. By this way, it reaches longer accurate displacement fields while efficiently reducing the complexity. Experiments on challenging sequences reveal improved results compared to state-of-the-art methods including existing MISS schemes both in terms of complexity reduction and accuracy improvement.

Fast coding unit selection and motion estimation algorithm based on early detection of zero block quantified transform coefficients for high‐efficiency video coding standard

High-efficiency video coding (HEVC) is the newest video coding standard developed by the joint video team, consisting of ITU-T video coding experts group and ISO/IEC Moving Picture Experts Group. The HEVC standard has aggregated an exhaustive algorithm for mode decision based on a recursive quad-tree structured coding tree block. Moreover, several specific features have been incorporated into the motion estimation (ME) process to improve its coding efficiency. However, they resulted in very high computational complexity. To accelerate the encoding process, fast mode decision algorithms for the partitioning module and also for the ME module were proposed in this study. These algorithms are based on early zero block detection technique. To improve the efficiency of these algorithms, an overall algorithm which combines the two techniques has been implemented. The performance of the proposed algorithm was checked through a comparative analysis in terms of encoding time and compression rate. Compared to HEVC test model 10.0, the authors’ proposed algorithms bring a great reduction of the HEVC complexity encoder with a saving time, which can reach 25% in average for different tested videos and a slight coding loss in terms of image quality and compression rate.

Adaptive Prediction based Diamond Search Algorithm (APDS) for Video Encoder of Solar Powered UAV

Aerial Vehicle (UAV) provide bird's eye view over an intersection or a large area, and provide real-time surveillance of area under observation. UAVs have been playing a vital role in disaster management due to the increased sensing and processing capabilities. This paper proposes a fast adaptive prediction based diamond search Motion Estimation (ME) algorithm for Sun Falcon 2, a solar powered UAV's video encoder to cope the computational complexity, low power and increased quality of ME process requirement. Results show that the proposed Adaptive Predict Diamond Search (APDS) ME algorithm performs best in the term of PSNR, MSE and number of Search Points (SP), for approximately all the video sequences. Moreover, performance of APDS is decreased a little bit in term of number of SP when compared to Hexagon search algorithm but its PSNR is still considerably high for those video sequences. The average PSNR improvement rate of APDS is 0.62, 2.67, 0.82, 0.83 and 2.31 for Diamond Search (DS), HexBS, FHS, FSS and MDS respectively, while the average SIR is 25.4404, 6.3374, 48.274 and 205.55 for DS, FHS, FSS and MDS respectively.

Novel Approach of Small and Large Displacement in Optical Flow for Object Detection

Optical flow estimation is oldest but still most active research domain in image processing. This method is used in motion estimation process. Here, we have discussed the Optical flow method to detect the moving object in video. We have also discussed object detection technique which is used in object detection for moving object. Thereafter, we have chosen optical flow method of detection for further improvement in object detection task. We have discussed different types of most popular techniques of optical flow and also recent survey paper and identified problems (research gaps) in the existing techniques. We found the most recent benchmark of optical flow MPI Sintel. According to the benchmark small displacement, large displacements, occlusions problem are there in object detection using optical flow method. In this report, we have discussed a novel approach for small and large displacement in optical flow for object detection. Finally, we compared the existing work with our approach using simulation results obtained in MATLAB & Visual Studio platform.

Bilateral frame rate up‐conversion algorithm based on the comparison of texture complexity

A new bilateral frame rate up-conversion (FRUC) algorithm based on the bi-directional comparison of the texture complexity in the reference frame is proposed. In the proposed algorithm, the accuracy of the motion estimation (ME) process is determined by the comparison of the prediction size in the reference region of the bi-directional ME process. Base on this comparison, the adaptive refinement process is adjusted. In the experimental results, the proposed algorithm outperforms the conventional bilateral FRUC algorithm by 0.27 dB.

FPGA-Based Real-Time Moving Target Detection System for Unmanned Aerial Vehicle Application

Moving target detection is the most common task for Unmanned Aerial Vehicle (UAV) to find and track object of interest from a bird’s eye view in mobile aerial surveillance for civilian applications such as search and rescue operation. The complex detection algorithm can be implemented in a real-time embedded system using Field Programmable Gate Array (FPGA). This paper presents the development of real-time moving target detection System-on-Chip (SoC) using FPGA for deployment on a UAV. The detection algorithm utilizes area-based image registration technique which includes motion estimation and object segmentation processes. The moving target detection system has been prototyped on a low-cost Terasic DE2-115 board mounted with TRDB-D5M camera. The system consists of Nios II processor and stream-oriented dedicated hardware accelerators running at 100 MHz clock rate, achieving 30-frame per second processing speed for 640 × 480 pixels’ resolution greyscale videos.

Automated selection of the optimal cardiac phase for single-beat coronary CT angiography reconstruction.

Reconstructing a low-motion cardiac phase is expected to improve coronary artery visualization in coronary computed tomography angiography (CCTA) exams. This study developed an automated algorithm for selecting the optimal cardiac phase for CCTA reconstruction. The algorithm uses prospectively gated, single-beat, multiphase data made possible by wide cone-beam imaging. The proposed algorithm differs from previous approaches because the optimal phase is identified based on vessel image quality (IQ) directly, compared to previous approaches that included motion estimation and interphase processing. Because there is no processing of interphase information, the algorithm can be applied to any sampling of image phases, making it suited for prospectively gated studies where only a subset of phases are available. An automated algorithm was developed to select the optimal phase based on quantitative IQ metrics. For each reconstructed slice at each reconstructed phase, an image quality metric was calculated based on measures of circularity and edge strength of through-plane vessels. The image quality metric was aggregated across slices, while a metric of vessel-location consistency was used to ignore slices that did not contain through-plane vessels. The algorithm performance was evaluated using two observer studies. Fourteen single-beat cardiac CT exams (Revolution CT, GE Healthcare, Chalfont St. Giles, UK) reconstructed at 2% intervals were evaluated for best systolic (1), diastolic (6), or systolic and diastolic phases (7) by three readers and the algorithm. Pairwise inter-reader and reader-algorithm agreement was evaluated using the mean absolute difference (MAD) and concordance correlation coefficient (CCC) between the reader and algorithm-selected phases. A reader-consensus best phase was determined and compared to the algorithm selected phase. In cases where the algorithm and consensus best phases differed by more than 2%, IQ was scored by three readers using a five point Likert scale. There was no statistically significant difference between inter-reader and reader-algorithm agreement for either MAD or CCC metrics (p > 0.1). The algorithm phase was within 2% of the consensus phase in 15/21 of cases. The average absolute difference between consensus and algorithm best phases was 2.29% ± 2.47%, with a maximum difference of 8%. Average image quality scores for the algorithm chosen best phase were 4.01 ± 0.65 overall, 3.33 ± 1.27 for right coronary artery (RCA), 4.50 ± 0.35 for left anterior descending (LAD) artery, and 4.50 ± 0.35 for left circumflex artery (LCX). Average image quality scores for the consensus best phase were 4.11 ± 0.54 overall, 3.44 ± 1.03 for RCA, 4.39 ± 0.39 for LAD, and 4.50 ± 0.18 for LCX. There was no statistically significant difference (p > 0.1) between the image quality scores of the algorithm phase and the consensus phase. The proposed algorithm was statistically equivalent to a reader in selecting an optimal cardiac phase for CCTA exams. When reader and algorithm phases differed by >2%, image quality as rated by blinded readers was statistically equivalent. By detecting the optimal phase for CCTA reconstruction, the proposed algorithm is expected to improve coronary artery visualization in CCTA exams.

Video Encoding Acceleration in Cloud Gaming

Cloud computing provides reliable, affordable, and flexible resources for many applications and users with constrained computing resources and capabilities. The cloud computing concept is becoming an appealing paradigm for many industries including the gaming industry, leading to the introduction of cloud gaming architectures. Despite its advantages, cloud gaming suffers from unguaranteed end-to-end delay as well as server side’s computational complexity. In this paper, a novel algorithm for reducing the computational complexity and hence speeding up the video encoding speed is proposed. Specifically, by performing minimum modifications in the game engine and the video codec, some information from the game engine is fed into the video encoder to bypass the motion estimation (ME) process. Our results show that the proposed method achieves up to 39% speedup in the ME process, leading to a 24% acceleration in the total encoding process.

Optimized Predictive Zonal Search (OPZS) for block-based motion estimation

Predictive Motion Estimation (ME) techniques are fast ME algorithms that have been widely used in video CODECs due to their performance efficiency and low computational complexity. In this paper, a new approach named Optimized Predictive Zonal Search (OPZS) is proposed, which improves the state-of-the-art Enhanced Predictive Zonal Search (EPZS) algorithm. The new initial predicted motion vectors employ fewer, but more optimized predictors, as well as improve the termination procedure. The proposed OPZS not only achieves better output quality and significantly higher compression ratio, but also reduces the number of search points and ME processing time even further. Simulation results after encoding 6 different video test sequences with OPZS along with EPZS and three other algorithms implemented under the same conditions has shown up to 33% compression ratio enhancement. This is achieved by searching up to 14% less points, which means less computational complexity and up to 0.4dB higher PSNR values. The simplicity and high performance of OPZS make it more practical for implementation on both hardware and software platforms. Since OPZS provides a combination of low computational complexity and high quality encoded images, it is energy efficient and suitable for portable video processing in computation- or power-constrained applications. Furthermore, because of the higher compression ratio this newly proposed algorithm is more compatible with limited capacity storage media and transmission networks with limited band-width.

A novel Cooperative Motion Estimation Algorithm based on Particle Swarm Optimization and its multicore implementation

A cooperative motion estimation (ME) scheme using a modified Particle Swarm Optimization (PSO) algorithm is presented. The proposed algorithm is based on a multi-swarm PSO model where a swarm of PSO particles is defined for each macroblock (MB) in the frame. Motion estimation is then performed in a cooperative manner concurrently for all the MBs in the frame. Cooperation between neighboring MBs during the motion estimation process is allowed through a communication step to exchange information about the motion vectors found so far in the estimation process. This synergic relationship between the swarms of adjacent MBs allows refining the motion search and leads to both a faster convergence of the PSO process and an improvement in the resulting motion vectors. Several techniques are also proposed to improve the search capacity and computational complexity of the PSO iterations. A novel PSO initialization scheme that exploits the existing temporal correlation is proposed to remove dependency between adjacent MBs. A fitness function history preservation mechanism is also presented to prevent redundant repeated calculations of the fitness function of a given search point by the PSO particles which dramatically decreases the computational complexity. Moreover, the maximum allowed velocity of the particles is adaptively varied during the PSO iterative process which provides a balance between search exploration and exploitation. The proposed scheme exhibits a high level of data parallelism since it is capable of performing motion estimation for all the MBs of the frame in parallel rather than serially. As a result, the presented algorithm is amenable to parallel processing techniques. In this paper, a multicore implementation of our proposed algorithm is performed using the MATLAB® Parallel Computing Toolbox™ (PCT). Extensive simulations are performed to analyze the performance of the presented algorithm. It is found that the presented scheme provides improvements in terms of accuracy and computational complexity as compared to conventional fast motion estimation techniques and two state-of-the-art PSO-based ME schemes. An analysis of the parallel performance shows that the presented scheme is highly scalable and that the parallel efficiency increases with the increase in video resolution. The multicore implementation of the proposed algorithm using MATLAB could achieve a speedup of 6.21 on eight CPU cores for high-definition (HD) video sequences. The multicore performance of the proposed scheme is also compared with existing parallel algorithms in the literature and is shown to give superior results.

High speed on-chip multiple cosine transform generator

Speeding up video encoding process is an important issue nowadays due to the high demand of increasing the video transmission speed for real-time video applications. Many algorithms and hardware implementations have been proposed for achieving such demand using pixel domain motion estimation. Moreover, much savings in computations and hardware complexity can be achieved using frequency domain motion estimation (FDME). The key issues in the FDME encoder is to generate four different transforms that are used for deciding the best match motion vector in the motion estimation (ME) process. The related transformed-discrete cosine transform (RT-DCT) generator is responsible for generating such transforms in the FDME encoder. In this paper, a fast RT-DCT generator is proposed to speed up the FDME encoding time by reducing its complexity. That makes the proposed RT-DCT appropriated for real-time video applications with limited resources, such as mobile internet device, cellular phones, personal digital assistant (PDA), and ultra-mobile personal computer. Moreover, the proposed architecture does not scarify the resolution. By adding the proposed RT-DCT generator to the whole FDME system, implementation and simulation results show that the system performs ME for SDTV video sequence at 371 MHz.

Bilateral frame rate upconversion algorithm based on image texture complexity compensation

A new bilateral frame rate upconversion algorithm using the image texture complexity characteristic is proposed. In the proposed algorithm, as the cost function is adjusted according to the image texture complexity, the accuracy of motion estimation (ME) is improved. In the experimental results, the proposed algorithm outperforms the conventional bilateral ME process by 0.32 dB.

Performance Analysis of Fast Block Matching Motion Estimation Algorithms

Motion Estimation (ME) is an integral part of any video encoder and a large number of Block Matching Motion Estimation (BMME) Algorithms are proposed to cope the computational complexity and increase quality of ME process requirement. Therefore, it is necessary to evaluate the performance of these ME algorithms for different motion activities. In this paper five fast famous BMME algorithms are considered to evaluate their performance on the basis of ME time, search points, PSNR and Means Square Error (MSE). The algorithms evaluated in this paper are considered for state of the art video compression standards like MPEG 1, to MPEG4 and H.261 to H.264. Results show that the PSNR of Diamond Search (DS) is best for all test video sequences, whereas, Hardware Modified DS takes maximum number of search points to calculate motion vector. Moreover, hexagon search algorithm takes minimum number of search points but its PSNR is considerably lower than the other algorithms.