Video Stabilization Algorithm Research Articles

Dual-Modality Cross-Interaction-Based Hybrid Full-Frame Video Stabilization

This study aims to generate visually useful imagery by preventing cropping while maintaining resolution and minimizing the degradation of stability and distortion to enhance the stability of a video for Augmented Reality applications. The focus is placed on conducting research that balances maintaining execution speed with performance improvements. By processing Inertial Measurement Unit (IMU) sensor data using the Versatile Quaternion-based Filter algorithm and optical flow, our research first applies motion compensation to frames of input video. To address cropping, PCA-flow-based video stabilization is then performed. Furthermore, to mitigate distortion occurring during the full-frame video creation process, neural rendering is applied, resulting in the output of stabilized frames. The anticipated effect of using an IMU sensor is the production of full-frame videos that maintain visual quality while increasing the stability of a video. Our technique contributes to correcting video shakes and has the advantage of generating visually useful imagery at low cost. Thus, we propose a novel hybrid full-frame video stabilization algorithm that produces full-frame videos after motion compensation with an IMU sensor. Evaluating our method against three metrics, the Stability score, Distortion value, and Cropping ratio, results indicated that stabilization was more effectively achieved with robustness to flow inaccuracy when effectively using an IMU sensor. In particular, among the evaluation outcomes, within the “Turn” category, our method exhibited an 18% enhancement in the Stability score and a 3% improvement in the Distortion value compared to the average results of previously proposed full-frame video stabilization-based methods, including PCA flow, neural rendering, and DIFRINT.

Background separation network for video anomaly detection

Video anomaly detection refers to the automatic identification of abnormal behaviors, objects, or events in videos. However, current methods for anomaly detection based on original frames lack a comprehensive understanding of the importance of foreground information, making it challenging to efficiently address video anomaly detection in the presence of complex background interference. In this paper, we propose a video anomaly detection algorithm based on Background Separation Network (BSN) to address this issue. Firstly, we utilize a video stabilization algorithm to reduce video jitter and enhance the quality of input video frames. Secondly, BSN shifts the focus from the entire frame to the foreground region with higher anomaly detection value. BSN utilizes the motion pixel distribution of the video as the basis for foreground extraction, enabling pixel-level background separation to obtain more accurate and complete foreground targets. Lastly, a certain proportion of foreground targets in the foreground image are masked as background, reducing the interference caused by redundant targets on the detection results. The proposed method achieves an accuracy of 96.2% on the UCSD ped2 dataset, demonstrating its effectiveness. This method contributes to accurately detecting abnormal behaviors in real-world surveillance videos to protect the safety of public lives and assets.

Video stabilization based on low‐rank constraint and trajectory optimization

AbstractVideo stabilization plays a pivotal role in enhancing video quality by eliminating unwanted jitter in shaky videos. This paper introduces a novel video stabilization algorithm that leverages low‐rank constraint and trajectory optimization to effectively eliminate undesirable motion and generate stabilized videos. In the proposed algorithm, a low‐rank constraint regularization term is incorporated to enhance the smoothness of motion trajectories. Additionally, a predictive path smoothness term is integrated to ensure the consistency of motion across neighbouring frames. To address the problem of excessive cropping resulting from aggressive smoothing, a flexible local window strategy that emphasizes local motion relationships within the trajectories is introduced. The experimental results show that, compared to other excellent video stabilization algorithms, the proposed algorithm improves the stability metric by approximately 2.3%. Furthermore, in the stabilized videos generated by the algorithm, an approximate improvement of 2.18 dB in average image temporal fidelity and a 5.7% increase in average structural similarity between adjacent frames are achieved. The code that implements the proposed method is publicly accessible at https://github.com/CZS0319/VS_Low‐Rank_Constraint_Trajectory_Optimization.

Real-Time Video Stabilization Algorithm Based on SuperPoint

Real-Time Video Stabilization Algorithm Based on SuperPoint

Vehicle video stabilization algorithm based on grid motion statistics and adaptive Kalman filtering

Vehicle video stabilization algorithm based on grid motion statistics and adaptive Kalman filtering

Efficient online real-time video stabilization with a novel least squares formulation and parallel AC-RANSAC

A novel online real-time video stabilization algorithm (LSstab) that suppresses unwanted motion jitters based on cinematography principles is presented. LSstab features a parallel realization of the a-contrario RANSAC (AC-RANSAC) algorithm to estimate the inter-frame camera motion parameters. A novel least squares based smoothing cost function is then proposed to mitigate undesirable camera jitters according to cinematography principles. A recursive least square solver is derived to minimize the smoothing cost function with a linear computation complexity. LSstab is evaluated using a suite of publicly available videos against state-of-the-art video stabilization methods. Results show that LSstab achieves comparable or better performance, which attains real-time processing speed when a GPU is used.

Video stabilization algorithm for field robots in uneven terrain

Video stabilization algorithm for field robots in uneven terrain

An Unsupervised Video Stabilization Algorithm Based on Key Point Detection

In recent years, video stabilization has improved significantly in simple scenes, but is not as effective as it could be in complex scenes. In this study, we built an unsupervised video stabilization model. In order to improve the accurate distribution of key points in the full frame, a DNN-based key-point detector was introduced to generate rich key points and optimize the key points and the optical flow in the largest area of the untextured region. Furthermore, for complex scenes with moving foreground targets, we used a foreground and background separation-based approach to obtain unstable motion trajectories, which were then smoothed. For the generated frames, adaptive cropping was conducted to completely remove the black edges while maintaining the maximum detail of the original frame. The results of public benchmark tests showed that this method resulted in less visual distortion than current state-of-the-art video stabilization methods, while retaining greater detail in the original stable frames and completely removing black edges. It also outperformed current stabilization models in terms of both quantitative and operational speed.

Low Memory Access Video Stabilization for Low-Cost Camera SoC.

Video stabilization is one of the most important features in consumer cameras. Even simple video stabilization algorithms may need to access the frames several times to generate a stabilized output image, which places a significant burden on the camera hardware. This high-memory-access requirement makes it difficult to implement video stabilization in real time on low-cost camera SoC. Reduction of the memory usage is a critical issue in camera hardware. This paper presents a structure and layout method to efficiently implement video stabilization for low-end hardware devices in terms of shared memory access amount. The proposed method places sub-components of video stabilization in a parasitic form in other processing blocks, and the sub-components reuse data read from other processing blocks without directly accessing data in the shared memory. Although the proposed method is not superior to the state-of-the-art methods applied in post-processing in terms of video quality, it provides sufficient performance to lower the cost of camera hardware for the development of real-time devices. According to my analysis, the proposed one reduces the memory access amount by 21.1 times compared to the straightforward method.

Video Stabilization Algorithm for Vineyard Robots

Video Stabilization Algorithm for Vineyard Robots

Evaluation and Selection of Video Stabilization Techniques for UAV-Based Active Infrared Thermography Application.

Unmanned Aerial Vehicles (UAVs) that can fly around an aircraft carrying several sensors, e.g., thermal and optical cameras, to inspect the parts of interest without removing them can have significant impact in reducing inspection time and cost. One of the main challenges in the UAV based active InfraRed Thermography (IRT) inspection is the UAV’s unexpected motions. Since active thermography is mainly concerned with the analysis of thermal sequences, unexpected motions can disturb the thermal profiling and cause data misinterpretation especially for providing an automated process pipeline of such inspections. Additionally, in the scenarios where post-analysis is intended to be applied by an inspector, the UAV’s unexpected motions can increase the risk of human error, data misinterpretation, and incorrect characterization of possible defects. Therefore, post-processing is required to minimize/eliminate such undesired motions using digital video stabilization techniques. There are number of video stabilization algorithms that are readily available; however, selecting the best suited one is also challenging. Therefore, this paper evaluates video stabilization algorithms to minimize/mitigate undesired UAV motion and proposes a simple method to find the best suited stabilization algorithm as a fundamental first step towards a fully operational UAV-IRT inspection system.

Video stabilization algorithm based on virtual sphere model

We propose a video stabilization algorithm based on the rotation of a virtual sphere. Unlike traditional video stabilization algorithms relying on two-dimensional motion models or reconstruction of (3D) camera motions, the proposed virtual sphere model stabilizes video by projecting each frame onto the sphere and performing corrective rotations. Specifically, matching feature points between two adjacent frames are first projected onto two virtual spheres to obtain pairs of spherical points. Then, the rotation matrix between the previous and current frame is calculated. The resulting 3D rotation matrix sequence is used to represent the camera motion, and it is smoothed using the geodesic distance on a Riemannian manifold. Finally, the difference between the smoothed and original path allows obtaining the rotation matrix that causes camera jitter, and the virtual spheres are rotated reversely to suppress jitter. Experimental results show that the proposed algorithm can effectively reduce random jitter, outperforming state-of-the-art methods.

Video stabilization algorithm for tunnel robots based on improved Kalman filter

The application of tunnel robots in power tunnels is becoming more and more extensive. Due to various reasons, video jitter will inevitably occur during the inspection process, which will affect the real-time processing of subsequent images. In order to reduce the impact of video jitter and meet the requirements of real-time image processing, it is necessary to study fast video image stabilization methods. The traditional Kalman filtering algorithm has less computational cost, but it will cause a large estimation error when the system motion state changes. In this regard, the paper proposes an improved Kalman filtering algorithm, which changes the corresponding filter parameters by real-time estimation of a system in motion to reduce the estimation error. And this algorithm performs Harris corner extraction in the set feature dense area, and combines the PyrLK optical flow algorithm with the homography matrix to calculate the motion trajectory, which improves the calculation speed and accuracy of the algorithm. Experimental results show that this algorithm has better video stabilization effect and faster calculation speed compared with the traditional Kalman filtering algorithm, and can better meet the real-time image processing requirements in the inspection process.

Thermal Infrared Video Stabilization for Aerial Monitoring of Active Wildfires

Measuring wildland fire behavior is essential for fire science and fire management. Aerial thermal infrared (TIR) imaging provides outstanding opportunities to acquire such information remotely. Variables such as fire rate of spread (ROS), fire radiative power (FRP), and fireline intensity may be measured explicitly both in time and space, providing the necessary data to study the response of fire behavior to weather, vegetation, topography, and firefighting efforts. However, raw TIR imagery acquired by unmanned aerial vehicles (UAVs) requires stabilization and georeferencing before any other processing can be performed. Aerial video usually suffers from instabilities produced by sensor movement. This problem is especially acute near an active wildfire due to fire-generated turbulence. Furthermore, the nature of fire TIR video presents some specific challenges that hinder robust interframe registration. Therefore, this article presents a software-based video stabilization algorithm specifically designed for TIR imagery of forest fires. After a comparative analysis of existing image registration algorithms, the KAZE feature-matching method was selected and accompanied by pre- and postprocessing modules. These included foreground histogram equalization and a multireference framework designed to increase the algorithm's robustness in the presence of missing or faulty frames. The performance of the proposed algorithm was validated in a total of nine video sequences acquired during field fire experiments. The proposed algorithm yielded a registration accuracy between 10 and 1000× higher than other tested methods, returned 10× more meaningful feature matches, and proved robust in the presence of faulty video frames. The ability to automatically cancel camera movement for every frame in a video sequence solves a key limitation in data processing pipelines and opens the door to a number of systematic fire behavior experimental analyses. Moreover, a completely automated process supports the development of decision support tools that can operate in real time during an emergency.

Automated White Blood Cell Counting in Nailfold Capillary Using Deep Learning Segmentation and Video Stabilization

White blood cells (WBCs) are essential components of the immune system in the human body. Various invasive and noninvasive methods to monitor the condition of the WBCs have been developed. Among them, a noninvasive method exploits an optical characteristic of WBCs in a nailfold capillary image, as they appear as visual gaps. This method is inexpensive and could possibly be implemented on a portable device. However, recent studies on this method use a manual or semimanual image segmentation, which depends on recognizable features and the intervention of experts, hindering its scalability and applicability. We address and solve this problem with proposing an automated method for detecting and counting WBCs that appear as visual gaps on nailfold capillary images. The proposed method consists of an automatic capillary segmentation method using deep learning, video stabilization, and WBC event detection algorithms. Performances of the three segmentation algorithms (manual, conventional, and deep learning) with/without video stabilization were benchmarks. Experimental results demonstrate that the proposed method improves the performance of the WBC event counting and outperforms conventional approaches.

A Robust Video Stabilization Method Based on Feature Point Extraction and Warping

In recent technologies there are various applications which include a camera joined to a moving platform, for example, cars, drones and Unmanned Aerial Vehicles (UAV). The moving platform may suffer from vibrations which may cause unwanted motion in recordings that can cause degradation of performance in various applications like surveillance, tracking and detection of object. Stabilization of video in various applications is an emerging research area nowadays. To remove the unwanted motion from video, the stabilization is necessary to preserve the important content present in the video. In this paper the feature points from recorded videos are detected and then these feature points are extracted and matched. The obtained feature points are smoothed by K means clustering, a mesh grid on every video frame is set up and every grid is warped by matching and comparing the features points, from original video frame with the smoothed and stabilized feature points. The reduced distortions in the video are estimated from various parameters. The efficiency of algorithm is compared in which the robust video stabilization algorithm based on feature extraction and mesh grid warping obtains better improvement in Inter-frame Transform Fidelity (ITF) factor than the traditional video stabilization algorithm.

Facing Device Attribution Problem for Stabilized Video Sequences

A problem deeply investigated by multimedia forensics researchers is that of detecting which device has been used to capture a video. This enables us to trace down the owner of a video sequence, which proves extremely helpful to solve copyright infringement cases as well as to fight distribution of illicit material (e.g., child exploitation clips and terroristic threats). Currently, the most promising methods to tackle this task exploit unique noise traces left by camera sensors on acquired images. However, given the recent advancements in motion stabilization of video content, robustness of sensor pattern noise-based techniques is strongly hindered. Indeed, video stabilization introduces geometric transformations to video frames, thus making camera fingerprint estimation problematic with classical approaches. In this paper, we deal with the challenging problem of attributing stabilized videos to their recording device. Specifically, we propose: 1) a strategy to extract the characteristic fingerprint of a device, starting from either a set of images or stabilized video sequences and 2) a strategy to match a stabilized video sequence with a given fingerprint. The proposed methodology is tested on videos coming from a set of different smartphones, taken from the modern publicly available Vision Dataset. The conducted experiments also provide an interesting insight on the effect of modern smartphones video stabilization algorithms on specific video frames.

A novel accuracy assessment model for video stabilization approaches based on background motion

In this paper, we propose a new accuracy measurement model for the video stabilization method based on background motion that can accurately measure the performance of the video stabilization algorithm. Undesired residual motion present in the video can quantitatively be measured by the pixel by pixel background motion displacement between two consecutive background frames. First of all, foregrounds are removed from a stabilized video, and then we find the two-dimensional flow vectors for each pixel separately between two consecutive background frames. After that, we calculate a Euclidean distance between these two flow vectors for each pixel one by one, which is regarded as a displacement of each pixel. Then a total Euclidean distance of each frame is averaged to get a mean displacement for each pixel, which is called mean displacement error, and finally we calculate the average mean displacement error. Our experimental results show the effectiveness of our proposed method.

Video stabilization performance enhancement for low-texture videos

Digital video stabilization (DVS) aims to remove irregular global motion effects from an image sequence. This work aims at developing a real-time video stabilization algorithm for rectifying high-frequency jitter in marine surveillance applications. A DVS system consists of a global motion estimation system and motion correction system. The development of global motion estimation system resistant to failures in low texture videos is the primary goal. Due to the computational advantage and inherent properties, the phase correlation method is adopted as the basic global motion estimation algorithm. The basic algorithm is then modified to adapt to the varying texture content of the video sequences under consideration. An adaptive phase correlation-based global motion estimation is suggested and verified on the videos of varying textures.

A survey on image and video stitching

Image/video stitching is a technology for solving the field of view (FOV) limitation of images/ videos. It stitches multiple overlapping images/videos to generate a wide-FOV image/video, and has been used in various fields such as sports broadcasting, video surveillance, street view, and entertainment. This survey reviews image/video stitching algorithms, with a particular focus on those developed in recent years. Image stitching first calculates the corresponding relationships between multiple overlapping images, deforms and aligns the matched images, and then blends the aligned images to generate a wide-FOV image. A seamless method is always adopted to eliminate such potential flaws as ghosting and blurring caused by parallax or objects moving across the overlapping regions. Video stitching is the further extension of image stitching. It usually stitches selected frames of original videos to generate a stitching template by performing image stitching algorithms, and the subsequent frames can then be stitched according to the template. Video stitching is more complicated with moving objects or violent camera movement, because these factors introduce jitter, shakiness, ghosting, and blurring. Foreground detection technique is usually combined into stitching to eliminate ghosting and blurring, while video stabilization algorithms are adopted to solve the jitter and shakiness. This paper further discusses panoramic stitching as a special-extension of image / video stitching. Panoramic stitching is currently the most widely used application in stitching. This survey reviews the latest image/video stitching methods, and introduces the fundamental principles/advantages/weaknesses of image/video stitching algorithms. Image/video stitching faces long-term challenges such as wide baseline, large parallax, and low-texture problem in the overlapping region. New technologies may present new opportunities to address these issues, such as deep learning-based semantic correspondence, and 3D image stitching. Finally, this survey discusses the challenges of image/video stitching and proposes potential solutions.