Video Motion Estimation Research Articles

Research on a Fast Image-Matching Algorithm Based on Nonlinear Filtering

Computer vision technology is being applied at an unprecedented speed in various fields such as 3D scene reconstruction, object detection and recognition, video content tracking, pose estimation, and motion estimation. To address the issues of low accuracy and high time complexity in traditional image feature point matching, a fast image-matching algorithm based on nonlinear filtering is proposed. By applying nonlinear diffusion filtering to scene images, details and edge information can be effectively extracted. The feature descriptors of the feature points are transformed into binary form, occupying less storage space and thus reducing matching time. The adaptive RANSAC algorithm is utilized to eliminate mismatched feature points, thereby improving matching accuracy. Our experimental results on the Mikolajcyzk image dataset comparing the SIFT algorithm with SURF-, BRISK-, and ORB-improved algorithms based on the SIFT algorithm conclude that the fast image-matching algorithm based on nonlinear filtering reduces matching time by three-quarters, with an overall average accuracy of over 7% higher than other algorithms. These experiments demonstrate that the fast image-matching algorithm based on nonlinear filtering has better robustness and real-time performance.

Gaussian mixture model based phase prior learning for video motion estimation

Optic measurement nowadays shows more prominent features over the traditional sensor-based measurement techniques as it does not need sensor installment and can provide simultaneous measurements with very high spatial resolution. Phase-based video processing has been widely used in optic measurement for capturing small motions. It relies on the assumptions that the variance of noise has to be much lower than the pure signal and small motions are encoded in the phase shift of an individual pixel’s color which fundamentally restricts its usefulness. To tackle these issues, this paper proposes a novel perspective of phase estimation by utilizing the image priors on phase patches. The patch group based Gaussian Mixture Model (PG-GMM) learning algorithm is used to learn the nonlocal self-similarity (NSS) prior from training images. Then the phase information is modeled by learned patch group priors, and further optimized using the steps of gaussian component selection and weighted sparse coding. The proposed method is validated in both magnification and modal analysis applications in the example analysis. Compared to the phase-based method, the proposed method achieves high-quality magnifications on the real video signal, with fewer artifacts and better anti-noise performance. Clearer time-domain motion estimates of video components can also be extracted by the proposed method.

Euler equations and trace properties of minimizers of a functional for motion compensated inpainting

<p style='text-indent:20px;'>We compute the Euler equations of a functional useful for simultaneous video inpainting and motion estimation, which was obtained in [<xref ref-type="bibr" rid="b17">17</xref>] as the relaxation of a modified version of the functional proposed in [<xref ref-type="bibr" rid="b16">16</xref>]. The functional is defined on vectorial functions of bounded variations, therefore we also get the Euler equations holding on the singular sets of minimizers, highlighting in particular the conditions on the jump sets. Such conditions are expressed by means of traces of geometrically meaningful vector fields and characterized as pointwise limits of averages on cylinders with axes parallel to the unit normals to the jump sets.</p>

Motion estimation in hazy videos

• A novel end-to-end deep network is proposed for haze removal from hazy video frames. • Failure of conventional approaches in estimating motion information in hazy videos has been analyzed. • A novel haze removal → motion estimation approach is proposed for motion estimation from hazy videos. Motion estimation is the basic need for the success of many video analysis algorithms such as moving object detection, human activity recognition, etc. Most of the motion estimation algorithms are prone to weather conditions and thus, they fail to estimate the motion in degraded weather. Severe weather situations like snow, rain, haze, smog, etc., degrades the performance and reliability of video analysis algorithms. In this paper, we have analyzed the effect of the haze on motion estimation in hazy videos. We propose a cascaded architecture i.e. haze removal followed by optical flow for motion estimation in hazy videos. The proposed image de-hazing network is build upon the Res idual and I nception module concepts and named as ResINet. Further, an optical flow is utilized to estimate the motion information. We have carried out the visual analysis to validate the proposed approach for motion estimation in hazy videos. Also, to validate the proposed ResINet for de-hazing, we carried out the quantitative analysis on two benchmark image de-hazing datasets.

A Cross-Layer Gate-Level-to-Application Co-Simulation for Design Space Exploration of Approximate Circuits in HEVC Video Encoders

A cross-layer design space exploration (DSE) method based on a proposed co-simulation technique is presented herein. The proposed method is demonstrated evaluating the impacts on both coding efficiency and power dissipation of applying distinct approximate logic operators in a s $\mu {\mathrm{ m}}$ of absolute differences (SAD) kernel that accelerates an H.265/HEVC (high-efficiency video coding) encoder. The proposed method simulates the gate-level circuit dynamically inside the application, with realistic results of the impact of the adder-tree approximate logic implementation on both quality and encoder bit-rate results. A comprehensive DSE is shown herein, with 13 types of 6 classes of approximate adders in the SAD accelerator hardware blocks. Over 3,000 logic variants of approximations at gate-level were developed. Actual video sequences as inputs to the x265 software encoder are co-simulated, to dynamically capture the video motion-estimation (ME) behavior in the presence of logic approximations. While the prior art that only estimates the impact of the approximate logic on power, area, and quality on static designs with statistical assumptions, which are agnostic to the actual algorithm data-dependent behavior in the application, our method explores accurately the trade-off between power dissipation and coding efficiency dynamically over the entire HEVC encoding. Our approach shows that the lower-part-or and error-tolerant adder I approximate adders, as well as truncation-to-zero deliver better compression-power trade-offs, with substantial differences from the static analysis.

Deep network for human action recognition using Weber motion

Effective motion estimation is one of the prime steps for any human action recognition (HAR) algorithm. Optical flow (OF) and motion history image (MHI) are two well-known methods for motion estimation in videos. OF has several advantages over MHI. But the major drawback with OF is that it is computationally very expensive as compared to the MHI. Therefore, in this paper, a new motion estimation technique named as Weber Motion History Image (WMHI) is proposed. Here, an extremely fast algorithm is proposed for HAR using WMHI, pose information, and convolutional neural network (CNN). In spite of being fast and less space consuming, the algorithm outperforms the existing pose based CNN results on five benchmark datasets namely JHMDB [1], sub-JHMDB [1], MPII [2] and HMDB51 [3] and UCF101 [4]. The work mainly focuses on a new efficient algorithm which can be implemented for real-time HAR in videos. For real-time implementation, the two basic criteria on which an algorithm can be analyzed are space and time complexity. The proposed algorithm is faster as compared to the existing OF based HAR systems. In terms of space complexity, the feature size of the proposed algorithm is almost 50% of the existing OF based algorithm. The recognition results still outperform the existing result by a significant margin.

Estimation of full‐field dynamic strains from digital video measurements of output‐only beam structures by video motion processing and modal superposition

Strain is an essential quantity to characterize local structural behaviors and directly correlates with structural damage initiation and development that is within local regions. Strain measurement at high spatial resolution (density) locations is thus required to characterize local structural behaviors and detect potential local damage. Traditional contact‐type strain gauges are mostly discrete point‐wise sensors that can only be placed in a limited number of positions. Distributed optical fiber sensing techniques can measure strains at spatially dense measurement points, but their instrumentation is a time‐ and labor‐intensive process associated with the issue of the fragility of fibers. Noncontact optical measurement techniques, such as a family of interferometry techniques using laser beams (e.g., laser Doppler vibrometers), can provide vibration measurement at high density spatial points without the need to install sensors on the structure. However, these measurement devices are active sensing methods that are relatively expensive and vulnerable to ambient motion. Photogrammetry is an alternative noncontact optical measurement method using (passive) white‐light imaging of digital video cameras that are relatively low‐cost, agile, and provides simultaneous measurements at high spatial density locations where every pixel becomes a measurement point. Among others, digital image correlation can achieve full‐field deformation measurements and subsequently estimate the full‐field strains. However, it is computationally extensive. This study develops a new efficient approach to estimate the full‐field (as many measurement points as the pixel number of the video frame on the structure) dynamic strains at high‐spatial (pixel)‐resolution/density location points from the digital video measurement of output‐only vibrating structures. The developed approach is based on phase‐based video motion estimation and modal superposition of structural dynamic response. Furthermore, the method is augmented by a high‐fidelity finite element model, which is updated with the full‐field experimental modal parameters “blindly” identified from the video measurement of the output‐only structure. Laboratory experiments are conducted to validate the method on a bench‐scale cantilever beam structure. Results demonstrate that the full‐field dynamic strain estimated by the developed approach from the video measurement of the output‐only vibrating beam match very well those directly measured by the strain gauges (at discrete measurement points). Some factors associated with the effectiveness of the method are experimentally studied and discussed.

Properties of a Variational Model for Video Inpainting

We consider a variational model analyzed in March and Riey (Inverse Probl Imag 11(6): 997–1025, 2017) for simultaneous video inpainting and motion estimation. The model has applications in the field of recovery of missing data in archive film materials. A gray-value video content is reconstructed in a spatiotemporal region where the video data is lost. A variational method for motion compensated video inpainting is used, which is based on the simultaneous estimation of apparent motion in the video data. Apparent motion is mathematically described by a vector field of velocity, denoted optical flow, which is estimated through gray-value variations of the video data. The functional to be minimized is defined on a space of vector valued functions of bounded variation and the relaxation method of the Calculus of Variations is used. We introduce in the functional analyzed in March and Riey(Inverse Probl Imag 11(6): 997–1025, 2017) a suitable positive weight, and we show that diagonal minimizing sequences of the functional converge, up to subsequences as the weight tends to infinity, to minimizers of an appropriate limit functional. Such a limit functional is the relaxed version of a functional, modified with suitable improvements, proposed by Lauze and Nielsen (2004) and which permits an accurate joint reconstruction both of the optical flow and of the video content.

A hybrid block-based motion estimation algorithm using JAYA for video coding techniques

In any video coding solution, motion estimation (ME) and motion compensation (MC) techniques are widely used and they play an inevitable role in reducing the temporal redundancies between successive frames. Block-based motion estimation (BME) is one of the widely utilized techniques in the recently developed video compression standards by JCT-VC including HEVC due to its efficacy and ease of implementation. In BME, the frames in a video sequence are partitioned into a number of non-overlapping blocks. Then, for each of the blocks, a best-matched block is obtained within a definite search region in the reference frame to minimize certain cost function or fitness value such as the sum of absolute difference (SAD), mean of absolute difference (MAD), or mean square error (MSE). However, the key challenge lies in the evaluation of these cost functions, since they are computationally expensive and involves the most time-taking operations in the BME process. Hence, BME-based approaches can be viewed as an optimization problem and meta-heuristic algorithms can be effectively exploited for the problem under consideration. In this paper, a hybrid BME technique using a recently developed optimization algorithm, namely, JAYA algorithm, is proposed. Besides this proposal, several other modules, namely, the fitness approximation technique, search history preservation, and early & adaptive termination criterion are utilized. The main objective behind the use of the aforementioned modules is to avoid the unnecessary evaluation of the fitness function. Exhaustive simulations are carried out to demonstrate the efficacy of the proposed method over that of the benchmark schemes with respect to different performance measures, namely, the peak signal-to-noise ratio (PSNR), PSNR degradation ratio (DPSNR), search efficiency, structural similarity index measure (SSIM), and computation time. Comparative analysis and quantitative evaluation clearly show that the present technique produces more reliable motion vectors with competitive time complexity as compared to that of the benchmark schemes.

A Residue Number System Hardware Design of Fast-Search Variable-Motion-Estimation Accelerator for HEVC/H.265

A residue number system (RNS) has an inherent parallel structure that can be utilized for improving computer hardware systems. An RNS represents large integer numbers as a smaller integer set, or residues of a modulo set, without carry propagation between them. Hence mathematical operations, such as addition or subtraction, can be performed on the residues independently. This paper proposes an RNS implementation of motion estimation for the latest video coding standard known as high-efficiency video coding (HEVC) or H.265. Since motion estimation is the most computationally intensive task in video coding, several simplified algorithms are proposed for mitigating the problem, but the majority of them result in a worsening peak signal-to-noise ratio (PSNR) or bit-rate performance, or sometimes both. This paper also proposes a modified algorithm based on a test-zone (TZ) search algorithm, a widely used fast-search algorithm with good rate-distortion performance, suitable for hardware implementation for encoding ultra-high-definition videos in real time. The results show that worst-case PSNR degradation and bit-rate increases compared with the TZ search in the HEVC reference software implementation are negligible, and the hardware gate count is less than for many other designs in the literature.

Real-time 3DRS motion estimation for frame-rate conversion

In this article we present an updated version of a block-wise video motion estimation (ME) algorithm that is intended to be part of real-time frame rate conversion (FRC) software system. As a baseline, we choose a variant of a 3D recursive search (3DRS) algorithm, due to its relative simplicity and known real-time performance potential. We introduce updating of this algorithm, which provides effective multithreaded parallelisation in multicore systems and decreases the number of computations without a noticeable decrease in quality.

The algorithm and VLSI architecture of a high efficient motion estimation with adaptive search range for HEVC systems

This paper presents a novel algorithm and VLSI architecture of motion estimation (ME) for high efficiency video coding systems. The proposed algorithm examines a much smaller set of search candidates and thus greatly reduces the computational complexity. Furthermore, in order to strike a balance between the quality of the Video and the efficiency of the system, this algorithm possesses the advantages that the number of search candidates is adaptive to the characteristic of the Video content. The simulation results show that, compared to the HM reference software, the proposed algorithm leads to a 96% reduction in search candidates with only 1.98% increment in average bitrate. Based on this algorithm, a hardware-efficient VLSI architecture of ME is designed and implemented with 90 nm technology. The experimental results show that, occupying the area complexity of 274.5 kGE, the presented design achieves 60 frames per second with resolution of 3840 × 2160 at the frequency of 201 MHz. The proposed ME system enhances the hardware efficiency by at least 50% compared to the prior works.

PIXEL-BY-PIXEL ESTIMATION OF SCENE MOTION IN VIDEO

Abstract. The paper considers the effectiveness of motion estimation in video using pixel-by-pixel recurrent algorithms. The algorithms use stochastic gradient decent to find inter-frame shifts of all pixels of a frame. These vectors form shift vectors’ field. As estimated parameters of the vectors the paper studies their projections and polar parameters. It considers two methods for estimating shift vectors’ field. The first method uses stochastic gradient descent algorithm to sequentially process all nodes of the image row-by-row. It processes each row bidirectionally i.e. from the left to the right and from the right to the left. Subsequent joint processing of the results allows compensating inertia of the recursive estimation. The second method uses correlation between rows to increase processing efficiency. It processes rows one after the other with the change in direction after each row and uses obtained values to form resulting estimate. The paper studies two criteria of its formation: gradient estimation minimum and correlation coefficient maximum. The paper gives examples of experimental results of pixel-by-pixel estimation for a video with a moving object and estimation of a moving object trajectory using shift vectors’ field.

Restricted affine motion compensation and estimation in video coding with particle filtering and importance sampling: a multi-resolution approach

In this paper, we propose a multi-resolution affine block-based tracker for motion estimation and compensation, compatible with existing video coding standards such as H.264 and HEVC. We propose three modifications to traditional motion compensation techniques in video coding standards such as H.264 and HEVC. First, we replace traditional search methods with an efficient particle filtering-based method, which incorporates information from both spatial and temporal continuity. Second, we use a higher order linear model in place of the traditional translation motion model in these standards to efficiently represent complex motions such as rotation and zoom. Third, we propose a multi-resolution framework that enables efficient parameter estimation. Results of extensive experimentation show reduced residual energy and better Peak Signal-to-Noise Ratio (PSNR, hereafter) as compared to H.264/HEVC for instance, especially in regions of complex motion such as zooming and rotation.

Analysis of a variational model for motion compensated inpainting

We study a variational problem for simultaneous video inpainting and motion estimation. We consider a functional proposed by Lauze and Nielsen [25] and we study, by means of the relaxation method of the Calculus of Variations, a slightly modified version of this functional. The domain of the relaxed functional is constituted of functions of bounded variation and we compute a representation formula of the relaxed functional. The representation formula shows the role of discontinuities of the various functions involved in the variational model. The present study clarifies the variational properties of the functional proposed in [25] for motion compensated video inpainting.

Energy Efficient Reconfigurable Architecture for Motion Estimation in Video Coding

Background/Objectives: Reconfigurable architecture has ability to dynamically allocate the hardware resources during runtime. It can be effectively used in computationally intensive application like media processing. As the motion estimation in video coding consumes large amount of computational time and resources, it can be mapped into reconfigurable architecture to effectively manage the power utilization by dynamic reconfiguration. Methods/Statistical Analysis: A systolic array based reconfigurable architecture for motion estimation which can be configured based on the properties of input video is proposed. A dynamically reconfigurable hardware is designed which can be worked on different search regions based on the level of motion in frames of input video. For the input video, the level of motion among the adjacent frames is determined by motion analyzer. Based on the level of motion between the frames of video, the search window size for block search is selected and this selection will enable the optimum number of processing elements for processing. This dynamic selection of hardware resources based on the search window reduces the power dissipation and computational complexity. Findings: Two search windows have been fixed for analysis 8 × 8 and 7 × 7. For power dissipation analysis, the total logic elements, total registers and fan-out for each design is taken. The performance is analysed by enabling selective number of processing elements for different size of search window. It is observed that power dissipation is high for the search window 8 × 8, because the resource utilization is higher than 7 × 7 search window. Instead of using the same fixed search window for performing block batching, different sized search windows can be used based on the level of motion of the video. After analysis, it is positively found that the proper selection of search window will lead to the optimum utilization in terms of power and resources. Application/Improvements: The context-aware reconfigurable hardware design for highly computationally intensive applications like video processing would be helpful in optimizing the power and resource utilization in hand held devices like smart phones, cam-coders etc.

Optimized EDDR Design for Motion estimation applications

Given the critical role of motion estimation (ME) in a video coder, testing such a module is of priority concern. While focusing on the testing of ME in a video coding system, this work presents an error detection and data recovery (EDDR) design, based on the residue-and-quotient (RQ) code, to embed into ME for video coding testing applications. An error in processing elements (PEs), i.e. key components of a ME, can be detected and recovered effectively by using the proposed EDDR design. Experimental results indicate that the proposed Error Detection and Data Recovery design for motion estimation testing can detect errors and recover data with an acceptable area overhead and timing penalty. The functional verification and synthesis can be done by using any version of Xilinx ISE

IJARCCE - Computer and Communication Engineering

This paper presents an error detection and data recovery (EDDR) design, based on the residue-and-quotient (RQ) code, to embed into motion estimation (ME) for video coding testing applications. An error in processing elements (PEs), i.e. key components of a ME, can be detected and recovered effectively by using the EDDR design. The proposed EDDR design for ME testing can detect errors and recover data with an acceptable area overhead and timing penalty. The functional verification and synthesis can be done by Xilinx ISE. That is when compare to the existing design the implemented design area and timing will be reduced.. The new Joint Video Team (JVT) video coding standard has garnered increased attention recently. Generally, motion estimation computing array (MECA) performs up to 50% of computations in the entire video coding system, and is typically considered the computationally most important part of video coding systems. Thus, integrating the MECA into a system-on-chip (SOC) design has become increasingly important for video coding applications. Although advances in VLSI technology allow integration of a large number of processing elements (PEs) in an MECA into an SOC, this increases the logic- per-pin ratio, thereby significantly decreasing the efficiency of chip logic testing. For a commercial chip, a video coding system must introduce design for testability (DFT), especially in an MECA. The objective of DFT is to increase the ease with which a device can be tested to guarantee high system reliability. Many DFT approaches have been developed. These approaches can be divided into three categories: ad hoc (problem oriented), structured, and built-in self-test (BIST). Among these techniques, BIST has an obvious advantage in that expensive test equipment is not needed and tests are low cost.

A fast motion estimation algorithm based on adaptive pattern and search priority

Block matching algorithm (BMA) has been widely used in motion estimation for various video coding standards since it can remove temporal redundancy effectively. However, motion estimation is the key problem in realizing real-time video coding due to the high computation complexity of BMA. In this manuscript, we present a fast motion estimation algorithm according to the adaptive pattern and search priority (APSP). Based on the distribution characteristics of motion vector (MV) that achieved by a series of experiments, the improved algorithm defines different efficient patterns and adopts the appropriate pattern adaptively. Firstly, the search can be stopped after checking one point by the features of the current block. And then the starting pattern is determined based on the motion vectors from the neighboring blocks. The subsequent pattern can be further adjusted according to the current best matching point. Furthermore, the proposed method assigns search priority to each point of every pattern. Therefore, the search is performed under the guidance of the search priority, with the result that each pattern can be interrupted in any position by using priority and threshold. Compared to conventional fast algorithms, the experimental results demonstrate that the proposed algorithm improves the performance of the search algorithm with significant reduction in computational complexity on the premise of ensuring the image quality and searching precision.

Motion Estimation for Omnidirectional Images using the Adapted Block-Matching

The Block-Matching (BM) method for motion estimation in most video coding is largely discussed in the case of perspective images. The omnidirectional cameras provide images with large field of view. These images contain global information about motion and permit to remove the ambiguity present with little camera motion in perspective case. Nevertheless, these images contain significant radial distortions. The Block- Matching in these catadioptric images is not a resolved problem, and still a challenging research field. A rectangular block representing the neighborhood in BM of a point and used in the perspective images is not appropriate for catadioptric cameras. The work presented in this article concerns the local motion estimation in catadioptric videos with the Adapted Block-Matching (ABM). The ABM based on an adapted neighborhood, the local motion estimation allows successful compensation prediction in catadioptric images. The Adapted Block-Matching is obtained from the equivalence between the omnidirectional image and the projection of scene points on a unit sphere. Index Terms—Catadioptric images, adapted neighborhood, motion estimation, Adapted Block- Matching.