Video Stabilization Research Articles

An Analytical Approach for Enhanced Video Stabilization Using a Bio-Inspired Neural Network

The number of videos and photos shot over the past decade has dramatically increased owing to the rapid developments of hand-held digital cameras. However, viewers often see their videos difficult to watch, particularly because of severe camera shake and abrupt movements. As a result, this paper proposes a significant algorithm which results in creation of high-quality video stabilisation, thereby eliminating distracting jitters from amateur recordings and make them appear as though they were shot along smooth, purposeful camera tracks. Modern video stabilization algorithms typically possess an architecture that enables them to realize several benefits: detecting motion in video, constructing a motion model, refining the motion, and ultimately generating stabilized output frames. The video stabilisation includes a feature extraction mechanism in which Genetic Algorithm (GA) is employed due to its characteristics like invariant to scale, rotation, translations, illumination, and blur. In this paper, Whale Optimization Algorithm (WOA) tuned Convolutional Neural Network (CNN) technique is suggested for performing accurate estimation of inliers and outliers. The interframe motion is then estimated by fitting a set of matched point pairs into a linear transform model. Experimental analysis states that proposed system outperforms better than other state-of-art models under various measures (Rotation:94.2, Blue:95.1, Wrap:96.3 and Time cost:97.5).

An unsupervised video stabilization algorithm based on gyroscope image fusion

An unsupervised video stabilization algorithm based on gyroscope image fusion

Minimum Latency Deep Online Video Stabilization and Its Extensions.

We present a novel deep camera path optimization framework for minimum latency online video stabilization. Typically, a stabilization pipeline consists of three steps: motion estimation, path smoothing, and novel view synthesis. Most previous methods concentrate on motion estimation while path optimization receives less attention, particularly in the crucial online setting where future frames are inaccessible. In this work, we adopt off-the-shelf high-quality deep motion models for motion estimation and focus only on the path optimization. Specifically, our camera path smoothing network takes a short 2D camera path in a sliding window as input and outputs the stabilizing warp field of the last frame, which warps the coming frame to its stabilized position. We explore three motion densities: a global single camera path, local mesh-based bundled paths, and dense flow paths. A hybrid loss and an efficient motion smoothing attention (EMSA) module are proposed for spatially and temporally consistent path smoothing. Moreover, we build a motion dataset that contains stable and unstable motion pairs for training. Extensive experiments demonstrate that our method surpasses state-of-the-art online stabilization methods and rivals the performance of offline methods, offering compelling advancements in the field of video stabilization. Code and dataset are available at https://github.com/liuzhen03/NNDVS.

Video Stabilization based on Sub-pixel Keypoints

Abstract With the continuous advancement of seeker technology and image processing techniques, the precision of guided weapons has increasingly improved. However, due to the rigidly fixed structure between the seeker and the guided weapon, the weapon is prone to experiencing disturbances and vibrations during flight, which can deteriorate the real-time video quality monitored by the seeker and, consequently reduce the precision of the guided weapon. To enhance the precision of guided weapons, it is necessary to mitigate the adverse effects of vibrations through video stabilization techniques. This paper proposes a video stabilization framework based on sub-pixel keypoints detection. It utilizes a lightweight network to detect keypoints, employs the Lucas-Kanade (LK) optical flow method for motion estimation, and smooths the camera’s motion path with an adaptive filter. Experimental results show that the proposed algorithm has an average processing time of approximately 0.2s per frame, achieving a stability index of 0.9262, a PSNR index of 22.3074, and an SSIM index of 0.7188. This demonstrates a balanced performance in terms of computational efficiency and stabilization, exhibiting excellent comprehensive performance.

Optical Flow as Spatial-Temporal Attention Learners.

Optical flow is an indispensable building block for various important computer vision tasks, including motion estimation, object tracking, and disparity measurement. To date, the dominant methods are CNN-based, leaving plenty of room for improvement. In this work, we propose TransFlow, a transformer architecture for optical flow estimation. Compared to dominant CNN-based methods, TransFlow demonstrates three advantages. First, it provides more accurate correlation and trustworthy matching in flow estimation by utilizing spatial self-attention and cross-attention mechanisms between adjacent frames to effectively capture global dependencies; Second, it recovers more compromised information (e.g., occlusion and motion blur) in flow estimation through long-range temporal association in dynamic scenes; Third, it introduces a concise self-learning paradigm, eliminating the need for complex and laborious multi-stage pre-training procedures. The versatility and superiority of TransFlow extend seamlessly to 3D scene motion, yielding competitive outcomes in 3D scene flow estimation. Our approach attains state-of-the-art results on benchmark datasets such as Sintel and KITTI-15, while also exhibiting exceptional performance on downstream tasks, including video object detection using the ImageNet VID dataset, video frame interpolation using the GoPro dataset, and video stabilization using the DeepStab dataset. We believe that the effectiveness of TransFlow positions it as a flexible baseline for both optical flow and scene flow estimation, offering promising avenues for future research and development.

Embedded deep learning video stabilization based on multiprocessing

Abstract With the gradual development of modern war in the direction of informationization and intelligentization, image guidance has established a prominent position in precision guidance technology with its high guidance accuracy, strong anti-jamming ability and precise hit probability, so the image seeker has become the focus of competitive development. However, missile jitter during flight, influenced by external disturbances and projectile rigidity, seriously affects the video quality captured by the camera in the image seeker, thereby reducing the detection accuracy of target recognition and tracking. In addition, the computational resources of the missile-mounted embedded platform are limited, and algorithms based on deep learning compute slowly. To address these issues, we have proposed a lightweight video stabilization framework and harnessed the potential of multi-core embedded devices through multiprocessing to achieve higher operating speeds. We tested our algorithm on an NVIDIA Jetson Orin Nano device and achieved a speed improvement of nearly 30% compared to an algorithm for serial computation.

NAFT and SynthStab: A RAFT-Based Network and a Synthetic Dataset for Digital Video Stabilization

NAFT and SynthStab: A RAFT-Based Network and a Synthetic Dataset for Digital Video Stabilization

Blind 3D Video Stabilization with Spatio-Temporally Varying Motion Blur

Video stabilization is a challenging task that attempts to compensate for the overall frame shake during video acquisition. Existing three-dimensional video stabilization methods aim at modeling camera perspective projection through either data-driven training or explicit motion estimation. However, the above methods are difficult to effectively solve the issue of shaky videos with abrupt object movements, resulting in local motion blur in the direction of the movement. This phenomenon is prevalent in real-world scenarios featuring foreground blind motion scenes. Unfortunately, directly combining stabilization and deblurring methods poses challenges when dealing with this situation. In the video, the intensity of motion blur undergoes continuous changes, and the direct combination method inadequately utilizes spatiotemporal information, providing insufficient clues for cross-frame compensation. To alleviate this problem, the Cross-frame-temporal Module framework is proposed to address blind motion blur induced by various conditions, which utilizes cross-frame temporal features to estimate depth maps and camera motion. In this framework, a Blur Transform Network (BTNet) is designed to adapt to spatially varying motion blur, which transforms local regions according to the impact of blur intensities to adapt to the effects of non-uniform motion blur; furthermore, our Temporal-Aware Network (TANet) further suppresses motion blur by leveraging cross-frame temporal features. In addition, the limited availability of pair-training video data containing motion blur limits the application of this approach in practice. The Cross-frame-temporal Module framework adopts an un-pretrained in-test training strategy. Extensive experimental results have demonstrated that our method outperforms state-of-the-art methods.

An approach for accurately extracting vehicle trajectory from aerial videos based on computer vision

An approach for accurately extracting vehicle trajectory from aerial videos based on computer vision

Research on Video Electronic Stabilization Technology Based on ShiTomasi Corner Detection

Video stabilization technology based on ShiTomasi corner detection has broad applications in intelligence surveillance, medical examination, and security monitoring. The research on video stabilization technology is crucial due to the inevitable mechanical vibrations affecting embedded cameras during use, leading to video jitter and compromising image quality. This paper proposes a video stabilization scheme based on the ShiTomasi corner detection method. The ShiTomasi algorithm is robust and accurate, effectively extracting corner features in videos. To further improve stabilization accuracy, a new weighted voting mechanism is designed, combining three-dimensional directional information to avoid mismatches in corner matching and enhance the accuracy of block matching. Specifically, this paper employs an adaptive search area and a multi-level feature detection method to process each video frame, ensuring the accuracy and robustness of corner detection. Additionally, an improved Kalman filter is introduced to reduce the impact of mismatched points on the stabilization results. In summary, the ShiTomasi algorithm-based video stabilization system designed in this paper not only enhances the speed and efficiency of video processing but also improves the precision and stability of stabilization.

Dual-Modality Cross-Interaction-Based Hybrid Full-Frame Video Stabilization

This study aims to generate visually useful imagery by preventing cropping while maintaining resolution and minimizing the degradation of stability and distortion to enhance the stability of a video for Augmented Reality applications. The focus is placed on conducting research that balances maintaining execution speed with performance improvements. By processing Inertial Measurement Unit (IMU) sensor data using the Versatile Quaternion-based Filter algorithm and optical flow, our research first applies motion compensation to frames of input video. To address cropping, PCA-flow-based video stabilization is then performed. Furthermore, to mitigate distortion occurring during the full-frame video creation process, neural rendering is applied, resulting in the output of stabilized frames. The anticipated effect of using an IMU sensor is the production of full-frame videos that maintain visual quality while increasing the stability of a video. Our technique contributes to correcting video shakes and has the advantage of generating visually useful imagery at low cost. Thus, we propose a novel hybrid full-frame video stabilization algorithm that produces full-frame videos after motion compensation with an IMU sensor. Evaluating our method against three metrics, the Stability score, Distortion value, and Cropping ratio, results indicated that stabilization was more effectively achieved with robustness to flow inaccuracy when effectively using an IMU sensor. In particular, among the evaluation outcomes, within the “Turn” category, our method exhibited an 18% enhancement in the Stability score and a 3% improvement in the Distortion value compared to the average results of previously proposed full-frame video stabilization-based methods, including PCA flow, neural rendering, and DIFRINT.

A decision model for OTT service users to adopt wireless D2D caching networks: Exploring the Korean case

A decision model for OTT service users to adopt wireless D2D caching networks: Exploring the Korean case

ДО ПИТАННЯ ПОБУДОВИ АЛГОРИТМІВ ПЕРЕДАЧІ ПОТОКОВОГО ВІДЕО ДЛЯ МЕРЕЖ З НЕСТАБІЛЬНОЮ ПРОПУСКНОЮ ЗДАТНІСТЮ

Some questions of development of advanced video transmission algorithms that could robustly operate in the electronic warfare conditions are considered. The technology of adaptation of MJPEG compressed video stream for the networks with variable and unstable bandwidth is proposed. The new approach where connection control is separated from encoder basing on man-in-the-middle scenario is developed. In this case it became possible to preserve not only standard decoders on the client side, but the encoder code on the server side as well. The performance and reliability of the proposed MJPEG transmission algorithm was demonstrated in the field tests. The MJPEG transcoder based on developed algorithm demonstrated sufficiently stable video stream from the drone via WiFi network in the presence of impulse signal jamming.

Fast and stable pedicel detection for robust visual servoing to harvest shaking fruits

Fast and stable pedicel detection for robust visual servoing to harvest shaking fruits

Digital Video Stabilization Method Based on Periodic Jitters of Airborne Vision of Large Flapping Wing Robots

Digital Video Stabilization Method Based on Periodic Jitters of Airborne Vision of Large Flapping Wing Robots

Adaptive Video Streaming in Multi-Tier Computing Networks: Joint Edge Transcoding and Client Enhancement

With the advancement of multimedia technology and wireless networks, there is a growing demand for high-quality video streaming. Delivering stable video streaming in extremely dynamic wireless networks, nevertheless, is still an open problem. Recent developments in client computing and mobile edge computing (MEC) technologies have both shown promise in enhancing the adaptive bitrate (ABR) streaming services. In this paper, we consider a video streaming system in multi-tier computing networks, enabled by joint edge-side video transcoding and client-side video enhancement. By “enhancement,” we mean that the client improves the video chunk quality via client-side image processing modules. In particular, we aim to design a joint bitrate adaptation, edge transcoding, and client image-processing algorithm, maximizing the quality of experience (QoE) of streaming services. The majority of the prior art has concentrated on super-resolution-enabled video streaming. Contrarily, we show that the video enhancement method outperforms the super-resolution approach in terms of signal-to-noise ratio and frames per second, implying a superior alternative for client-side processing in ABR streaming. We formulate the problem as an event-triggered Markov decision process (E-MDP), and propose a deep reinforcement learning (DRL)-based framework, named EDTEA. To deal with the delayed feedback induced by multi-tier computing, the entropy and the expected re-buffering terms are introduced to the objective and the reward, respectively. Extensive simulations based on real-world videos and bandwidth traces manifest that compared with state-of-the-art approaches, EDTEA provides <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$10.4\%\sim 78.4\%$</tex-math></inline-formula> extra QoE while reducing re-buffering time by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$85.5\%\sim 91.7\%$</tex-math></inline-formula> .

Exploring Temporal Feature Correlation for Efficient and Stable Video Semantic Segmentation

This paper tackles the problem of efficient and stable video semantic segmentation. While stability has been under-explored, prevalent work in efficient video semantic segmentation uses the keyframe paradigm. They efficiently process videos by only recomputing the low-level features and reusing high-level features computed at selected keyframes. In addition, the reused features stabilize the predictions across frames, thereby improving video consistency. However, dynamic scenes in the video can easily lead to misalignments between reused and recomputed features, which hampers performance. Moreover, relying on feature reuse to improve prediction consistency is brittle; an erroneous alignment of the features can easily lead to unstable predictions. Therefore, the keyframe paradigm exhibits a dilemma between stability and performance. We address this efficiency and stability challenge using a novel yet simple Temporal Feature Correlation (TFC) module. It uses the cosine similarity between two frames’ low-level features to inform the semantic label’s consistency across frames. Specifically, we selectively reuse label-consistent features across frames through linear interpolation and update others through sparse multi-scale deformable attention. As a result, we no longer directly reuse features to improve stability and thus effectively solve feature misalignment. This work provides a significant step towards efficient and stable video semantic segmentation. On the VSPW dataset, our method significantly improves the prediction consistency of image-based methods while being as fast and accurate.

Fast Coherent Video Style Transfer via Flow Errors Reduction

For video style transfer, naively applying still image techniques to process a video frame-by-frame independently often causes flickering artefacts. Some works adopt optical flow into the design of temporal constraint loss to secure temporal consistency. However, these works still suffer from incoherence (including ghosting artefacts) where large motions or occlusions occur, as optical flow fails to detect the boundaries of objects accurately. To address this problem, we propose a novel framework which consists of the following two stages: (1) creating new initialization images from proposed mask techniques, which are able to significantly reduce the flow errors; (2) process these initialized images iteratively with proposed losses to obtain stylized videos which are free of artefacts, which also increases the speed from over 3 min per frame to less than 2 s per frame for the gradient-based optimization methods. To be specific, we propose a multi-scale mask fusion scheme to reduce untraceable flow errors, and obtain an incremental mask to reduce ghosting artefacts. In addition, a multi-frame mask fusion scheme is designed to reduce traceable flow errors. In our proposed losses, the Sharpness Losses are used to deal with the potential image blurriness artefacts over long-range frames, and the Coherent Losses are performed to restrict the temporal consistency at both the multi-frame RGB level and Feature level. Overall, our approach produces stable video stylization outputs even in large motion or occlusion scenarios. The experiments demonstrate that the proposed method outperforms the state-of-the-art video style transfer methods qualitatively and quantitatively on the MPI Sintel dataset.

See farther and more: a master-slave UAVs based synthetic optical aperture imaging system with wide and dynamic baseline.

An open challenge remained in designing an optical system to capture the aerial image with a wide field of view (FoV) and high resolution. The optical system of one camera from a single unmanned aerial vehicle (UAV) can hardly promise the FoV and resolution. The conventional swarm UAVs can form the camera array with a short or fixed baseline. They can capture the images with a wide FoV and high resolution, but the cost is the requirement of many UAVs. We aim to design a camera array with a wide and dynamic baseline to reduce the demand for UAVs to organize a synthetic optical aperture. In this thought, we propose a master-slave UAVs-based synthetic optical aperture imaging system with a wide and dynamic baseline. The system consists of one master UAV and multiple slave UAVs. Master and slave UAVs provide the global and local FoVs, respectively, and improve the efficiency of image acquisition. In such a system, fusing UAV images becomes a new challenge due to two factors: (i) the small FoV overlap of slave UAVs and (ii) the gap in resolution scale from slave to master UAV images. To deal with it, a coarse-to-fine stitching method is proposed to stitch up the multi-view images into one to obtain a wide FoV with high resolution. A video stabilization method has also been designed for the proposed imaging system. Challenges caused by wide and dynamic baselines can thus be solved by the above methods. Actual data experiments demonstrate that the proposed imaging system achieves high-quality imaging results.

Background separation network for video anomaly detection

Video anomaly detection refers to the automatic identification of abnormal behaviors, objects, or events in videos. However, current methods for anomaly detection based on original frames lack a comprehensive understanding of the importance of foreground information, making it challenging to efficiently address video anomaly detection in the presence of complex background interference. In this paper, we propose a video anomaly detection algorithm based on Background Separation Network (BSN) to address this issue. Firstly, we utilize a video stabilization algorithm to reduce video jitter and enhance the quality of input video frames. Secondly, BSN shifts the focus from the entire frame to the foreground region with higher anomaly detection value. BSN utilizes the motion pixel distribution of the video as the basis for foreground extraction, enabling pixel-level background separation to obtain more accurate and complete foreground targets. Lastly, a certain proportion of foreground targets in the foreground image are masked as background, reducing the interference caused by redundant targets on the detection results. The proposed method achieves an accuracy of 96.2% on the UCSD ped2 dataset, demonstrating its effectiveness. This method contributes to accurately detecting abnormal behaviors in real-world surveillance videos to protect the safety of public lives and assets.