Real-time Video Stabilization Research Articles

Real-Time Video Stabilization Algorithm Based on SuperPoint

Real-Time Video Stabilization Algorithm Based on SuperPoint

Efficient online real-time video stabilization with a novel least squares formulation and parallel AC-RANSAC

A novel online real-time video stabilization algorithm (LSstab) that suppresses unwanted motion jitters based on cinematography principles is presented. LSstab features a parallel realization of the a-contrario RANSAC (AC-RANSAC) algorithm to estimate the inter-frame camera motion parameters. A novel least squares based smoothing cost function is then proposed to mitigate undesirable camera jitters according to cinematography principles. A recursive least square solver is derived to minimize the smoothing cost function with a linear computation complexity. LSstab is evaluated using a suite of publicly available videos against state-of-the-art video stabilization methods. Results show that LSstab achieves comparable or better performance, which attains real-time processing speed when a GPU is used.

Real-Time Digital Video Stabilization of Bioinspired Robotic Fish Using Estimation-and-Prediction Framework

The rhythmic movement of bioinspired robotic fish brings about undesirable visual jitter. Note that the unstable camera path of this kind of robot is characterized by obvious regularity and predictability, of which traditional stabilization methods have not made full advantage. This article proposes a novel estimation-and-prediction framework for real-time digital video stabilization of bioinspired robotic fish. First, based on the attitude information of an inertial measurement unit (IMU), a camera–IMU model is established, where the homography transformation with eight degrees of freedom (DOFs) is reduced to translation transformation with two DOFs. Second, traditional optical flow and gray projection methods as well as a novel translation estimation network are employed to estimate the translations between consecutive frames. Third, a lightweight long short-term memory (LSTM) network is constructed, allowing remarkable prediction and smoothing of the camera path. Finally, aquatic experiments under various scenarios are conducted on a manta-inspired robot, demonstrating the effectiveness of the proposed method. Specifically, compared with the state-of-the-art commercial offline stabilization software, our online stabilization algorithm achieves approximate visual stability and remarkably faster stabilization speed. The obtained results shed light on visual sensing and control applications of bioinspired underwater vehicles.

Gyroscope-Based Video Stabilization for Electro-Optical Long-Range Surveillance Systems.

Video stabilization is essential for long-range electro-optical systems, especially in situations when the field of view is narrow, since the system shake may produce highly deteriorating effects. It is important that the stabilization works for different camera types, i.e., different parts of the electromagnetic spectrum independently of the weather conditions and any form of image distortion. In this paper, we propose a method for real-time video stabilization that uses only gyroscope measurements, analyze its performance, and implement and validate it on a real-world professional electro-optical system developed at Vlatacom Institute. Camera movements are modeled with 3D rotations obtained by integration of MEMS gyroscope measurements. The 3D orientation estimation quality depends on the gyroscope characteristics; we provide a detailed discussion on the criteria for gyroscope selection in terms of the sensitivity, measurement noise, and drift stability. Furthermore, we propose a method for improving the unwanted motion estimation quality using interpolation in the quaternion domain. We also propose practical solutions for eliminating disturbances originating from gyro bias instability and noise. In order to evaluate the quality of our solution, we compared the performance of our implementation with two feature-based digital stabilization methods. The general advantage of the proposed methods is its drastically lower computational complexity; hence, it can be implemented for a low price independent of the used electro-optical sensor system.

Real-time video stabilization via camera path correction and its applications to augmented reality on edge devices

With the rapid development of edge devices such as mobile phones, in the past decade, many videos have appeared on the professional video website and they are easy to retrieve. However, most of the videos captured by mobile cameras are jittery, and even motion blurred. These low-quality videos may affect the experience of users. Thus, the problem that how to eliminate the jittery issues, and make unstable video became stable one is very urgent. To enhance the stability of low-quality video, in this paper, we propose a novel method for video stabilization. The proposed method is called SimpleStab, and consists of motion estimation, trajectory smoothing, and compositing image. The SimpleStab is not only able to process offline videos but also can deal with live video streaming due to the novel architecture. We conduct a comprehensive experiment on the benchmarking dataset and make comparison with the state-of-the-art approaches. Experimental results show that the performance of SimpleStab is superior to the state-of-the-art methods.

Real-time high-resolution video stabilization using high-frame-rate jitter sensing

In this study, the novel approach of real-time video stabilization system using a high-frame-rate (HFR) jitter sensing device is demonstrated to realize the computationally efficient technique of digital video stabilization for high-resolution image sequences. This system consists of a high-speed camera to extract and track feature points in gray-level 512times 496 image sequences at 1000 fps and a high-resolution CMOS camera to capture 2048times 2048 image sequences considering their hybridization to achieve real-time stabilization. The high-speed camera functions as a real-time HFR jitter sensing device to measure an apparent jitter movement of the system by considering two ways of computational acceleration; (1) feature point extraction with a parallel processing circuit module of the Harris corner detection and (2) corresponding hundreds of feature points at the current frame to those in the neighbor ranges at the previous frame on the assumption of small frame-to-frame displacement in high-speed vision. The proposed hybrid-camera system can digitally stabilize the 2048times 2048 images captured with the high-resolution CMOS camera by compensating the sensed jitter-displacement in real time for displaying to human eyes on a computer display. The experiments were conducted to demonstrate the effectiveness of hybrid-camera-based digital video stabilization such as (a) verification when the hybrid-camera system in the pan direction in front of a checkered pattern, (b) stabilization in video shooting a photographic pattern when the system moved with a mixed-displacement motion of jitter and constant low-velocity in the pan direction, and (c) stabilization in video shooting a real-world outdoor scene when an operator holding hand-held hybrid-camera module while walking on the stairs.

A novel camera path planning algorithm for real-time video stabilization

In video stabilization, a steady camera path plan is as important as accurate camera motion prediction. While several camera path planning algorithms have been studied, most algorithms are used for post-processing video stabilization. Since all of the frames are accessible in post-processing video stabilization, a camera path planning method can generate a good camera path, considering the camera movements at all of the frames. Meanwhile, the number of accessible frames in real-time video stabilization is limited. Thus, smooth camera path planning is a challenging issue in real-time video stabilization. For example, a camera path planner does not know in advance the locations of sudden camera motions, so it is not easy to compensate a sudden camera movement. Therefore, this paper proposes a novel camera path planning algorithm for real-time video stabilization. A camera path planning method should fully utilize the given image margin to provide steady camera paths. In contrast, it should retain the certain amount of an unused image margin for use in frames with dynamic or sudden camera movements. To resolve this problem, the proposed algorithm uses two terms related to the steady camera path and the amount of image margin, and it cross-optimizes the two terms to provide a new camera path on-the-fly. Experimental results show that the proposed algorithm provides excellent performance in real-time video stabilization while requiring negligible computation.

A Hardware–Software Co-Design Framework for Real-Time Video Stabilization

Live digital video is a valuable source of information in security, broadcast and industrial quality control applications. Motion jitter due to camera and platform instability is a common artefact found in captured video which renders it less effective for subsequent computer vision tasks such as detection and tracking of objects, background modeling, mosaicking, etc. The process of algorithmically compensating for the motion jitter is hence a mandatory pre-processing step in many applications. This process, called video stabilization, requires estimation of global motion from consecutive video frames and is constrainted by additional challenges such as preservation of intentional motion and native frame resolution. The problem is exacerbated in the presence of local motion of foreground objects and requires robust compensation of the same. As such achieving real-time performance for this computationally intensive operation is a difficult task for embedded processors with limited computational and memory resources. In this work, development of an optimized hardware–software co-design framework for video stabilization has been investigated. Efficient video stabilization depends on the identification of key points in the frame which in turn requires dense feature calculation at the pixel level. This task has been identified to be most suitable for offloading the pipelined hardware implemented in the FPGA fabric due to the involvement of complex memory and computation operations. Subsequent tasks to be performed for the overall stabilization algorithm utilize these sparse key points and have been found to be efficiently handled in the software. The proposed Hardware–Software (HW–SW) co-design framework has been implemented on Zedboard FPGA platform which houses Xilinx Zynq SOC equipped with ARM A9 processor. The proposed implementation scheme can process real-time video stream input at 28 frames per second and is at least twice faster than the corresponding software-only approach. Two different hardware accelerator designs have been implemented using different high-level synthesis tools using rapid prototyping principle and consume less than 50% of logic resources available on the host FPGA while being at least 30% faster than contemporary designs.

Video stabilization performance enhancement for low-texture videos

Digital video stabilization (DVS) aims to remove irregular global motion effects from an image sequence. This work aims at developing a real-time video stabilization algorithm for rectifying high-frequency jitter in marine surveillance applications. A DVS system consists of a global motion estimation system and motion correction system. The development of global motion estimation system resistant to failures in low texture videos is the primary goal. Due to the computational advantage and inherent properties, the phase correlation method is adopted as the basic global motion estimation algorithm. The basic algorithm is then modified to adapt to the varying texture content of the video sequences under consideration. An adaptive phase correlation-based global motion estimation is suggested and verified on the videos of varying textures.

High-Quality Real-Time Video Stabilization Using Trajectory Smoothing and Mesh-Based Warping

Some state-of-the-art video stabilization methods can achieve quite good visual effect, but they always cost a lot of time. On the other hand, current real-time video stabilization methods cannot generate satisfactory results. In this paper, we propose a novel trajectory-based video stabilization method which can generate high-quality results in real time. Our method runs very fast, because many techniques are proposed for acceleration. In the trajectory smoothing step, trajectories are extracted, pre-processed, and smoothed. A video splitting algorithm is proposed for pre-processing, and binomial filtering is used for smoothing. Both of them are simple and fast. In the frame warping step, we calculate a spatially varying warp that is directed by a grid mesh for each frame. Instead of solving time consuming global optimization problems, the transformation matrix of each grid is calculated using nearby trajectories in our method, leading to very high speed. We implement our method and run it on a variety of videos. Experiments show that while the stabilization effect is comparable with state-of-the-art methods, our algorithm can run in real time.

基于最小生成树与改进卡尔曼滤波器的实时电子稳像方法

基于最小生成树与改进卡尔曼滤波器的实时电子稳像方法

Real-time optical flow-based video stabilization for unmanned aerial vehicles

This paper describes the development of a novel algorithm to tackle the problem of real-time video stabilization for unmanned aerial vehicles (UAVs). There are two main components in the algorithm: (1) By designing a suitable model for the global motion of UAV, the proposed algorithm avoids the necessity of estimating the most general motion model, projective transformation, and considers simpler motion models, such as rigid transformation and similarity transformation; (2) to achieve a high processing speed, optical flow-based tracking is employed in lieu of conventional tracking and matching methods used by state-of-the-art algorithms. These two new ideas resulted in a real-time stabilization algorithm, developed over two phases. Stage I considers processing the whole sequence of frames in the video while achieving an average processing speed of 50 fps on several publicly available benchmark videos. Next, Stage II undertakes the task of real-time video stabilization using a multi-threading implementation of the algorithm designed in Stage I.

Video Stabilization for Strict Real-Time Applications

Offline or deferred solutions are frequently employed for high quality and reliable results in current video stabilization. However, neither of these solutions can be used for strict real-time applications. In this paper, we propose a practical and robust algorithm for real-time video stabilization. To achieve this, a novel and efficient motion model based on inter-frame homography estimation is proposed to represent the video motion. An important feature of the proposed motion model is that it updates at each frame input to reduce the accumulation errors caused by parallax or scene changes. We also propose a novel Kalman filter for the motion smoothing and a unique mosaic algorithm for the video completion. The proposed Kalman filter and mosaic algorithm enable the development of a practical real-time video stabilizer that not only produces steady video but also retains the full resolution of the original video. We verify the proposed algorithm through a broad range of video sequences that demonstrate that the proposed algorithm is computationally efficient while being able to robustly stabilize videos with various challenges.

Real-Time Feature-Based Video Stabilization on FPGA

Digital video stabilization is an important video enhancement technology that aims to remove unwanted camera vibrations from video sequences. Trading off between stabilization performance and real-time hardware implementation feasibility, this paper presents a feature-based full-frame video stabilization method and a novel complete fully pipelined architectural design to implement it on field-programmable gate array (FPGA). In the proposed method, feature points are first extracted with the oriented features from accelerated segment test and rotated binary robust independent elementary features algorithm and matched between consecutive frames. Next, the matched point pairs are fitted to the affine transformation model using a random-sample consensus-based approach to estimate inter-frame motion robustly. Then, the estimated results are accumulated to compute the cumulative motion parameters between the current and reference frames, and the translational components are smoothed by a Kalman filter representing intentional camera movement. Finally, a mosaicked image is constructed based on cumulative motion parameters using an image mosaicking technique, and then a display window is created with the desired frame size according to the computed intentional camera movement to obtain a full motion-compensated frame. Using pipelining and parallel processing strategies, the whole process has been designed using a novel complete fully pipelined architecture and implemented on Altera’s Cyclone III FPGA to build a real-time stabilization system. The experimental results have shown that the proposed system can deal with standard PAL video input including arbitrate translation and rotation and can produce full-frame stabilized output providing a better viewing experience at 22.37 ms/frame, thus achieving real-time processing performance.

Feature-based real-time video stabilization for vehicle video recorder system

This paper presents a fast and effective method based on features to obtain real-time video stabilization for vehicle video recorder system. The corresponding feature points are first obtained from two consecutive frames and then optical flows are calculated based on these points. Next, the obtained optical flows are mapped to polar coordinates to obtain clusters and remove incorrect optical flows. These obtained clusters are used to evaluate the global motion and rotation angle. Finally, the obtained global motion and rotation angle are smoothed and then compensated to obtain the stabilized video. Experimental results show that the proposed method has good performance for video stabilization.

Real-time video stabilization for fast-moving vehicle cameras

Most previous methods of real-time video stabilization are only effective for low-vibrating frames which are usually captured by in-vehicle camera at the low-speed moving. To overcome their ineffectiveness on high-vibrating frames, this paper presents a real-time video stabilization system for the video sequences captured by a fast-moving in-vehicle camera without additional sensors. The proposed method is composed of four parts: frame-shaking judgment, feature classification, evaluating global motion and rotation angle, and frame compensation. Feature points and their motion vectors are employed for judging whether the current frame is shaking or not, and then a conversion matrix is deduced through the perspective projection for classifying such feature points into background or foreground type. Next, the optical flows of background’s feature points are mapped to polar coordinates for obtaining the representative optical-flow cluster of the background. Finally, such a cluster is utilized to calculate the global motion and rotation angle for compensation followed by the Kalman filtering in order to provide the better video stabilization. Experimental results show that the proposed method has good real-time video stabilization for a vehicle camera moving at various speeds and better stabilization performance than other methods for high-vibrating frames when both real-time processing and acceptable stabilization result are considered.

Стабилизация видеоизображения в режиме реального времени с использованием MEMS-датчиков

This article describes our ongoing research on real-time digital video stabilization using MEMS-sensors. The authors propose to use the described method for stabilizing the video that is transmitted to the mobile robot operator who controls the vehicle remotely, as well as increasing the precision of video-based navigation for subminiature autonomous models. The article describes the general mathematical models needed to implement the video stabilization module based on the MEMS sensors readings. These models includes the camera motion model, frame transformation model and rolling-shutter model. The existing approaches to stabilization using sensors data were analyzed and considered from the point of view of the application in a real-time mode. This article considers the main problems that came up during the experiments that were not resolved in the previous research papers. Such problems include: calibration of the camera and sensors, synchronization of the camera and sensors, increasing the accuracy of determining the camera position from sensors data. The authors offer possible solutions to these problems that would help improve quality of the work of existing algorithms, such as a system for parallel synchronized recording of video and sensor data based on the Android operating system. As the main result, the authors represent a framework for implementing video stabilization algorithms based on MEMS sensors readings.

Real-Time SURF-Based Video Stabilization System for an FPGA-Driven Mobile Robot

Mobile robots are used for search and rescue purposes in emergency informatics. The robots typically carry a vision system to gather information about the environment and pass on to remotely located rescue teams. When the robot moves, in view of the uneven nature of the terrain, the camera is subjected to vibrations and as a result, the transmitted videos tend to be unclear. Further, real-time data collection and processing are critical for quick action by rescue personnel. In this paper, we explore a robust and high-performance approach for video stabilization. In particular, we develop a systolic array for interest point detection and description, key phases in the speeded-up robust features (SURF) approach for finding point correspondences between two images of a scene. The systolic array is then used as part of a pipelined architecture for video stabilization. The entire architecture is ported on to a field-programmable gate array (FPGA) for real-time video stabilization on a mobile robot. Experiments with the FPGA-driven robot in our laboratory and institute corridors validate the efficacy of the proposed strategy.

Real-Time Model-Based Video Stabilization for Microaerial Vehicles

The emerging branch of micro aerial vehicles (MAVs) has attracted a great interest for their indoor navigation capabilities, but they require a high quality video for tele-operated or autonomous tasks. A common problem of on-board video quality is the effect of undesired movements, so different approaches solve it with both mechanical stabilizers or video stabilizer software. Very few video stabilizer algorithms in the literature can be applied in real-time but they do not discriminate at all between intentional movements of the tele-operator and undesired ones. In this paper, a novel technique is introduced for real-time video stabilization with low computational cost, without generating false movements or decreasing the performance of the stabilized video sequence. Our proposal uses a combination of geometric transformations and outliers rejection to obtain a robust inter-frame motion estimation, and a Kalman filter based on an ANN learned model of the MAV that includes the control action for motion intention estimation.

A compact hardware architecture for digital image stabilization using integral projections

We present a hardware architecture for real-time digital video stabilization for high-performance embedded systems. The stabilization algorithm analyzes the current and past video frames and obtains a motion estimation vector, which is then filtered to isolate unwanted camera movements from intentional panning. The vector is then used to correct the output video frame. The paper describes our hardware architecture for motion estimation, filtering and correction and its implementation on a Xilinx Spartan-6 LX45 FPGA. We evaluate our results on several benchmark video sequences, both visually and using the Inter-frame Transformation Fidelity index (ITF). Running on the 640×480-pixel video output of an infrared camera, our FPGA implementation successfully compensates involuntary camera motion at a maximum throughput of 104.15 frames per second with a 100MHz clock. The power consumption added to the FPGA by the image stabilization core is only 24.16mW.