Global Motion Estimation Research Articles

Analysis and Compensation of Rolling Shutter Effect

Due to the sequential-readout structure of complementary metal-oxide semiconductor image sensor array, each scanline of the acquired image is exposed at a different time, resulting in the so-called electronic rolling shutter that induces geometric image distortion when the object or the video camera moves during image capture. In this paper, we propose an image processing technique using a planar motion model to address the problem. Unlike previous methods that involve complex 3-D feature correspondences, a simple approach to the analysis of inter- and intraframe distortions is presented. The high-resolution velocity estimates used for restoring the image are obtained by global motion estimation, BEzier curve fitting, and local motion estimation without resort to correspondence identification. Experimental results demonstrate the effectiveness of the algorithm.

Robust Global Motion Estimation Oriented to Video Object Segmentation

Most global motion estimation (GME) methods are oriented to video coding while video object segmentation methods either assume no global motion (GM) or directly adopt a coding-oriented method to compensate for GM. This paper proposes a hierarchical differential GME method oriented to video object segmentation. A scheme which combines three-step search and motion parameters prediction is proposed for initial estimation to increase efficiency. A robust estimator that uses object information to reject outliers introduced by local motion is also proposed. For the first frame, when the object information is unavailable, a robust estimator is proposed which rejects outliers by examining their distribution in local neighborhoods of the error between the current and the motion-compensated previous frame. Subjective and objective results show that the proposed method is more robust, more oriented to video object segmentation, and faster than the referenced methods.

Fast and accurate global motion estimation algorithm using pixel subsampling

Global motion generally describes the motion of a camera, although it may comprise motions of large objects. Global motions are often modeled by parametric transformations of two-dimensional images. The process of estimating the motions parameters is called global motion estimation (GME). GME is widely employed in many applications such as video coding, image stabilization and super-resolution. To estimate global motion parameters, the Levenburg–Marquardt algorithm (LMA) is typically used to minimize an objective function iteratively. Since the region of support for the global motion representation consists of the entire image frame, the minimization process tends to be very expensive computationally by involving all the pixels within an image frame. In order to significantly reduce the computational complexity of the LMA, we proposed to select only a small subset of the pixels for estimating the motion parameters, based on several subsampling patterns and their combinations. Simulation results demonstrated that the proposed method could speed up the conventional GME approach by over ten times, with only a very slight loss (less than 0.1 dB) in estimation accuracy. The proposed method was also found to outperform several state-of-the-art fast GME methods in terms of the speed/accuracy tradeoffs.

A fast CIS still image stabilization method without parallax and moving object problems

In this paper, we propose a fast still image stabilization method for CMOS image sensors (CIS) which can solve the parallax and moving object problems in image registration. CIS still image stabilization is a software technique that prevents motion blurs caused by minute shaking of a CIS camera, and it is realized by seamlessly registering several rapidly exposed CIS images. Our method consists of two stages of image registration: a global registration and then, a local registration. Since a CIS image can be distorted by the motion of a CIS camera having a rolling shutter mechanism, we present a way of removing CIS distortions in the global registration. Also, the local misalignments are reduced using a mesh-based local registration. Since the parallax or moving object regions cannot be correctly aligned in the image registration process, we propose a two-stage post-processing method that removes the misaligned regions by measuring the registration quality based on the labeling method. The computational load in the image stabilization is reduced by selectively performing our local registration based on a real-time global motion estimation method. The experiment results show the necessity of CIS distortion analysis in the CIS image registration and the effectiveness of our stabilization method even for the moving object and parallax problems.

Analysis and Hardware Architecture Design of Global Motion Estimation

Global motion estimation and compensation (GME/GMC) is an important video processing technique and has been applied to many applications including video segmentation, sprite/mosaic generation, and video coding. In MPEG-4 Advanced Simple Profile (ASP), GME/GMC is adopted to compensate camera motions. Since GME is important, many GME algorithms have been proposed. These algorithms have two common characteristics, huge computation complexity and ultra large memory bandwidth. Hence for realtime applications, a hardware accelerator of GME is required. However, there are many hardware design challenges of GME like irregular memory access and huge memory bandwidth, and only few hardware architectures have been proposed. In this paper, we first analyzed three typical algorithms of GME, and a fast GME algorithm is proposed. By using temporal prediction and skipping the redundant computation, 91% memory bandwidth and 80% iterations are saved, while the performance is kept, compared to Gradient Descent in MPEG-4 Verification Model. Based on our proposed algorithm, a hardware architecture of GME is also presented. A new scheduling, Reference-Based Scheduling, is developed to solve the irregular memory access problem. An interleaved memory arrangement is applied to satisfy the memory access requirement of interpolation. The total gate count of hardware implementation is 131 K with Artisan 0.18 um cell library, and the internal memory size is about 7.9 Kb. Its processing ability is MPEG-4 ASP@L3, which is 352×288 with 30 fps, at 30 MHz.

Interlaced-to-progressive conversion using adaptive projection-based global and representative local motion estimation

We propose a motion-compensation-based deinterlacing algorithm using global and representative local motion estimation. The proposed algorithm first divides an entire image into five regions of interest (ROIs) according to the temporally predicted motion type (i.e., global or local) and the spatial position. One of them is for global motion estimation and the others are for local motion estimation. Then, dominant motions of respective ROIs are found by adaptive projection approach. The adaptive projection method not only estimates dominant local motions with low computational cost, but also ensures consistent global motion estimation. Using the estimated motion vectors, adaptive two-field bidirectional motion compensation is performed. The arbitration rules, measuring the reliability of motion compensation accurately, produce high-quality deinterlaced frames by effectively combining the results of motion compensation and the stable intrafield deinterlacing. Experimental results show that the proposed deinterlacing algorithm provides better image quality than the existing algorithms in both subjective and objective measures.

Electronic image stabilization system based on global feature tracking

A new robust electronic image stabilization system is presented, which involves feature-point, tracking based global motion estimation and Kalman filtering based motion compensation. First, global motion is estimated from the local motions of selected feature points. Considering the local moving objects or the inevitable mismatch, the matching validation, based on the stable relative distance between the points set is proposed, thus maintaining high accuracy and robustness. Next, the global motion parameters are accumulated for correction by Kalman filteration. The experimental result illustrates that the proposed system is effective to stabilize translational, rotational, and zooming jitter and robust to local motions.

Reducing the complexity of MPEG-4 global motion estimation using pixel subsampling

MPEG-4 uses a perspective model for global motion estimation. The region of support for global motion representation consists of the entire image frame. Estimating global motion parameters is very expensive computationally, since all the pixels in the frames are involved in the computation. Therefore, computationally efficient global motion estimation techniques are sought-after. It is shown that real-time global motion estimation in an MPEG-4 video codec can be achieved by using a pixel subsampling method based on regular subsampling patterns. Simulation results demonstrated that the proposed method was capable of speeding up the conventional all-pixel GME approach by more than 11 times, with a subsampling ratio of 1/16, without significant loss in estimation accuracy and compression efficiency.

Sequential Bayesian estimation of global motion governing a group of points from local point tracking

Sequential Bayesian estimation of global motion governing a group of points from local point tracking

Motion estimation of magnetic resonance cardiac images using the Wigner-Ville and hough transforms

Myocardial motion analysis and quantification is of utmost importance for analyzing contractile heart abnormalities and it can be a symptom of a coronary artery disease. A fundamental problem in processing sequences of images is the computation of the optical flow, which is an approximation of the real image motion. This paper presents a new algorithm for optical flow estimation based on a spatiotemporal-frequency (STF) approach. More specifically it relies on the computation of the Wigner-Ville distribution (WVD) and the Hough Transform (HT) of the motion sequences. The latter is a well-known line and shape detection method that is highly robust against incomplete data and noise. The rationale of using the HT in this context is that it provides a value of the displacement field from the STF representation. In addition, a probabilistic approach based on Gaussian mixtures has been implemented in order to improve the accuracy of the motion detection. Experimental results in the case of synthetic sequences are compared with an implementation of the variational technique for local and global motion estimation, where it is shown that the results are accurate and robust to noise degradations. Results obtained with real cardiac magnetic resonance images are presented.

A coding scheme using global motion estimation for AVS P-frame

Block matching has been used for motion estimation and motion compensation in the Audio and Video Standard (AVS) for years. While having an acceptable performance in describing motion between frames, it requires quite a few bits to represent the motion vectors. In certain circumstances, the use of global motion estimation and compensation would perform equally well or even better than the block matching in terms of motion accuracy, while it results in the coding of global motion model parameters. In this letter, we modify an AVS coder by adding (1) six global motion model parameters to the frame header, and (2) mode selection among INTRA, SKIP, INTER-16×16, INTER-16×8, INTER-8×16, INTER-8×8, and Global Motion Compensation (GMC) modes by Lagrange optimal rate-distortion criteria. Simulation results demonstrate that over 0.1dB improvement in PSNR is obtained compared to the AVS coder for an average coded P-frame with the same bitrate.

Affine Motion Based CMOS Distortion Analysis and CMOS Digital Image Stabilization

The CMOS image distortion is due to the rolling shutter in CMOS image sensors (CISs) and it can be more exaggerated when a CIS camera moves rapidly. Several methods have been proposed to remove CMOS distortions made by the translational motion. But, in this paper, we propose the affine motion based CMOS distortion correction method combined with digital image stabilization. To remove CMOS distortions due to the affine global image motion, CMOS distortion model is proposed to explain the effect of the affine global image motion on the CMOS distortion in CISs. To improve CMOS video's visuals, we propose CMOS digital image stabilization to remove the jittering motions in the new image sequence obtained by our correction method. In addition, to reduce the computational time and the outlier effect, a reliable feature selection method is proposed to be used in the affine global image motion estimation. In the experiment results, we show that the proposed CMOS distortion correction method is more general than previous ones. Also, we show that our stabilization method can improve CMOS video's visuals running in real-time.

Integration of Digital Stabilizer With Video Codec for Digital Video Cameras

This paper presents three novel schemes for integrating digital stabilizer with video codec of a digital video camera, each designed for a different application scenario. Scheme 1 confines the global motion estimation (ME) to within a small background region determined by clustering the motion vectors (MVs) generated by the video encoder. Scheme 2 performs the main ME task at the digital stabilizer and sends the resulting motion vectors to the video encoder, where the motion vectors are further refined to subpixel accuracy. Scheme 3, which is applied on the decoder side, acquires the motion information directly from the video decoder to perform digital stabilization. These integration schemes achieve computational efficiency without affecting the performance of video coding and digital stabilization.

Robust global motion estimation

A global motion estimation method based on robust statistics is presented in this paper. By using tracked feature points instead of whole image pixels to estimate parameters the process speeds up. To further speed up the process and avoid numerical instability, an alterative description of the problem is given, and three types of solution to the problem are compared. By using a two step process, the robustness of the estimator is also improved. Automatic initial value selection is an advantage of this method. The proposed approach is illustrated by a set of examples, which shows good results with high speed.

Global parametric image alignment via high-order approximation

The estimation of parametric global motion is one of the cornerstones of computer vision. Such schemes are able to estimate various motion models (translation, rotation, affine, projective) with subpixel accuracy. The parametric motion is computed using a first order Taylor expansions of the registered images. But, it is limited to the estimation of small motions, and while large translations and rotations can be coarsely estimated by Fourier domain algorithms, no such techniques exist for affine and projective motions. This paper offers two contributions: first, we improve both the convergence range and rate using a second order Taylor expansion and show first order methods to be a degenerate case of the proposed scheme. Second, we extend the scheme using a symmetrical formulation which further improves the convergence properties. The results are verified by rigorous analysis and experimental trials.

Global motion estimation from randomly selected motion vector groups and GM/LM based applications

Fast global motion estimation has been paid much attention in video compression and analysis. In this paper, a global motion estimation method is proposed by randomly selected motion vector groups in the compression domain directly. It is carried out by refining the centroid of the global motion parameters corresponding to the motion vector groups. Simulation results on different global motions show its effectiveness and robustness against noise and motion vector loss. Finally, two applications, namely the text occluded region recovery and the error concealment, are presented using the global motion/local motion information. Experimental results show the effectiveness of the proposed method.

Shadow resistant tracking using inertia constraints

In this paper, we present a new method for tracking objects with shadows. Traditional motion-based tracking schemes cannot usually distinguish the shadow from the object itself, and this results in a falsely captured object shape. If we want to utilize the object's shape information for a pattern recognition task, this poses a severe difficulty. In this paper we present a color processing scheme to project the image into an illumination invariant space such that the shadow's effect is greatly attenuated. The optical flow in this projected image together with the original image is used as a reference for object tracking so that we can extract the real object shape in the tracking process. We present a modified snake model for general video object tracking. Two new external forces are introduced into the snake equation based on the predictive contour and a new chordal string shape descriptor such that the active contour is attracted to a shape similar to the one in the previous video frame. The proposed method can deal with the problem of an object's ceasing movement temporarily, and can also avoid the problem of the snake tracking into the object interior. Global affine motion estimation is applied to mitigate the effect of camera motion, and hence the method can be applied in a general video environment. Experimental results show that the proposed method can track the real object even if there is strong shadow influence.

Vision-based motion estimation for interaction with mobile devices

This paper introduces a novel interaction technique for handheld mobile devices which enables the user interface to be controlled by the motion of the user’s hand. A feature-based approach is proposed for global motion estimation that exploits gradient measures for both feature selection and feature motion uncertainty analysis. A voting-based scheme is presented for outlier removal. A Kalman filter is applied for smoothing motion trajectories. A fixed-point implementation of the method was developed due to the lack of floating-point hardware. Experiments testify the effectiveness of the approach on a camera-enabled mobile phone.

Robust global motion estimation and novel updating strategy for sprite generation

A new global motion estimation technique for sprite coding systems is presented. The proposed system can accurately register frames to a sprite without referencing to the sprite. This allows the motion estimation process to be performed in an environment independent of the quality of the sprite. The frame having the highest resolution of the scene is determined and all other frames can be projected on to the space of this chosen frame such that information loss due to decimation can be avoided. The static sprite is strategically updated, according to the major camera motion during decoding, to alleviate the problem of error propagation from the sprite image. Experimental results indicate that the proposed technique is very accurate and robust, which makes it suitable for MPEG-4 sprite coding.

A HIGH VISUAL QUALITY SPRITE GENERATOR USING INTELLIGENT BLENDING WITHOUT SEGMENTATION MASKS

The sprite generator introduced in MPEG-4 blends frames by averaging, which will make places, that are always occupied by moving objects, look blurred. Thus, providing segmented masks for moving objects is suggested. Several researchers have employed automatic segmentation methods to produce moving object masks. Based on these masks, they used a reliability-based blending strategy to generate sprites. Since perfect segmentation is impossible, some ghost-like shadows will appear in the generated sprite. To treat this problem, in this paper, an intelligent blending strategy without needing segmentation masks is proposed. It is based on the fact that for each point in the generated sprite, the corresponding pixels in most frames belong to background and only few belong to moving objects. A counting schema is provided to make only background points participate in average blending. The experimental result shows that the visual quality of the generated sprite using the proposed blending strategy is close to that using manually segmented masks and is better than that generated by Lu-Gao-Wu method. No ghostlike shadows are produced. Furthermore, a uniform feature point extraction method is proposed to increase the precision of global motion estimation, the effectiveness of this part is presented by showing the comparison results with other existing method.