Video Stabilization Research Articles

Video Stabilization based on Sub-pixel Keypoints

Abstract With the continuous advancement of seeker technology and image processing techniques, the precision of guided weapons has increasingly improved. However, due to the rigidly fixed structure between the seeker and the guided weapon, the weapon is prone to experiencing disturbances and vibrations during flight, which can deteriorate the real-time video quality monitored by the seeker and, consequently reduce the precision of the guided weapon. To enhance the precision of guided weapons, it is necessary to mitigate the adverse effects of vibrations through video stabilization techniques. This paper proposes a video stabilization framework based on sub-pixel keypoints detection. It utilizes a lightweight network to detect keypoints, employs the Lucas-Kanade (LK) optical flow method for motion estimation, and smooths the camera’s motion path with an adaptive filter. Experimental results show that the proposed algorithm has an average processing time of approximately 0.2s per frame, achieving a stability index of 0.9262, a PSNR index of 22.3074, and an SSIM index of 0.7188. This demonstrates a balanced performance in terms of computational efficiency and stabilization, exhibiting excellent comprehensive performance.

Optical Flow as Spatial-Temporal Attention Learners.

Optical flow is an indispensable building block for various important computer vision tasks, including motion estimation, object tracking, and disparity measurement. To date, the dominant methods are CNN-based, leaving plenty of room for improvement. In this work, we propose TransFlow, a transformer architecture for optical flow estimation. Compared to dominant CNN-based methods, TransFlow demonstrates three advantages. First, it provides more accurate correlation and trustworthy matching in flow estimation by utilizing spatial self-attention and cross-attention mechanisms between adjacent frames to effectively capture global dependencies; Second, it recovers more compromised information (e.g., occlusion and motion blur) in flow estimation through long-range temporal association in dynamic scenes; Third, it introduces a concise self-learning paradigm, eliminating the need for complex and laborious multi-stage pre-training procedures. The versatility and superiority of TransFlow extend seamlessly to 3D scene motion, yielding competitive outcomes in 3D scene flow estimation. Our approach attains state-of-the-art results on benchmark datasets such as Sintel and KITTI-15, while also exhibiting exceptional performance on downstream tasks, including video object detection using the ImageNet VID dataset, video frame interpolation using the GoPro dataset, and video stabilization using the DeepStab dataset. We believe that the effectiveness of TransFlow positions it as a flexible baseline for both optical flow and scene flow estimation, offering promising avenues for future research and development.

Embedded deep learning video stabilization based on multiprocessing

Abstract With the gradual development of modern war in the direction of informationization and intelligentization, image guidance has established a prominent position in precision guidance technology with its high guidance accuracy, strong anti-jamming ability and precise hit probability, so the image seeker has become the focus of competitive development. However, missile jitter during flight, influenced by external disturbances and projectile rigidity, seriously affects the video quality captured by the camera in the image seeker, thereby reducing the detection accuracy of target recognition and tracking. In addition, the computational resources of the missile-mounted embedded platform are limited, and algorithms based on deep learning compute slowly. To address these issues, we have proposed a lightweight video stabilization framework and harnessed the potential of multi-core embedded devices through multiprocessing to achieve higher operating speeds. We tested our algorithm on an NVIDIA Jetson Orin Nano device and achieved a speed improvement of nearly 30% compared to an algorithm for serial computation.

NAFT and SynthStab: A RAFT-Based Network and a Synthetic Dataset for Digital Video Stabilization

NAFT and SynthStab: A RAFT-Based Network and a Synthetic Dataset for Digital Video Stabilization

Blind 3D Video Stabilization with Spatio-Temporally Varying Motion Blur

Video stabilization is a challenging task that attempts to compensate for the overall frame shake during video acquisition. Existing three-dimensional video stabilization methods aim at modeling camera perspective projection through either data-driven training or explicit motion estimation. However, the above methods are difficult to effectively solve the issue of shaky videos with abrupt object movements, resulting in local motion blur in the direction of the movement. This phenomenon is prevalent in real-world scenarios featuring foreground blind motion scenes. Unfortunately, directly combining stabilization and deblurring methods poses challenges when dealing with this situation. In the video, the intensity of motion blur undergoes continuous changes, and the direct combination method inadequately utilizes spatiotemporal information, providing insufficient clues for cross-frame compensation. To alleviate this problem, the Cross-frame-temporal Module framework is proposed to address blind motion blur induced by various conditions, which utilizes cross-frame temporal features to estimate depth maps and camera motion. In this framework, a Blur Transform Network (BTNet) is designed to adapt to spatially varying motion blur, which transforms local regions according to the impact of blur intensities to adapt to the effects of non-uniform motion blur; furthermore, our Temporal-Aware Network (TANet) further suppresses motion blur by leveraging cross-frame temporal features. In addition, the limited availability of pair-training video data containing motion blur limits the application of this approach in practice. The Cross-frame-temporal Module framework adopts an un-pretrained in-test training strategy. Extensive experimental results have demonstrated that our method outperforms state-of-the-art methods.

An approach for accurately extracting vehicle trajectory from aerial videos based on computer vision

Vehicle trajectory data holds valuable information for advanced driving development and traffic analysis. While unmanned aerial vehicle (UAV) offer a broader perspective, the detection of small-scale vehicles in video frames still suffers from low accuracy or is even missed. This study proposes a comprehensive technical framework for accurate vehicle trajectory extraction, encompassing six main components: video stabilization, vehicle detection, vehicle tracking, lane marking detection, coordinate transformation, and data denoising. To mitigate video jitter, the SURF and FLANN stabilization algorithms are utilized. An enhanced detector based on You Only Look Once X (YOLOX) is employed for multi-target vehicle detection, incorporating a shallow feature extraction module within the detection head to improve the performance for low-level and small-scale features. Efficient Channel Attention (ECA) modules are integrated before the neck to further boost the expressiveness. Additionally, a sliding window inference method is applied at the input stage to prevent compression of high-resolution video frames. The Savitzky-Golay filter is used for trajectory noise reduction. Verification results demonstrate that the improved YOLOX achieves a mean average precision (mAP) of 88.7 %, an enhancement of 5.6 % over the original model. When compared to advanced YOLOv7 and YOLOv8 models, the proposed method increases mAP@50 by 7.63 % and 1.07 %, respectively. The Mostly Tracked (MT) trajectories metric reaches 98.9 %, and the root-mean-square error of one-sided localization is approximately 0.05 m. These results confirm that the proposed framework is an effective tool for high-accuracy vehicle trajectory data collection in traffic studies. Additionally, a vehicle trajectory dataset has been developed and is publicly accessible at www.cqskyeyex.com.

Research on Video Electronic Stabilization Technology Based on ShiTomasi Corner Detection

Video stabilization technology based on ShiTomasi corner detection has broad applications in intelligence surveillance, medical examination, and security monitoring. The research on video stabilization technology is crucial due to the inevitable mechanical vibrations affecting embedded cameras during use, leading to video jitter and compromising image quality. This paper proposes a video stabilization scheme based on the ShiTomasi corner detection method. The ShiTomasi algorithm is robust and accurate, effectively extracting corner features in videos. To further improve stabilization accuracy, a new weighted voting mechanism is designed, combining three-dimensional directional information to avoid mismatches in corner matching and enhance the accuracy of block matching. Specifically, this paper employs an adaptive search area and a multi-level feature detection method to process each video frame, ensuring the accuracy and robustness of corner detection. Additionally, an improved Kalman filter is introduced to reduce the impact of mismatched points on the stabilization results. In summary, the ShiTomasi algorithm-based video stabilization system designed in this paper not only enhances the speed and efficiency of video processing but also improves the precision and stability of stabilization.

Dual-Modality Cross-Interaction-Based Hybrid Full-Frame Video Stabilization

This study aims to generate visually useful imagery by preventing cropping while maintaining resolution and minimizing the degradation of stability and distortion to enhance the stability of a video for Augmented Reality applications. The focus is placed on conducting research that balances maintaining execution speed with performance improvements. By processing Inertial Measurement Unit (IMU) sensor data using the Versatile Quaternion-based Filter algorithm and optical flow, our research first applies motion compensation to frames of input video. To address cropping, PCA-flow-based video stabilization is then performed. Furthermore, to mitigate distortion occurring during the full-frame video creation process, neural rendering is applied, resulting in the output of stabilized frames. The anticipated effect of using an IMU sensor is the production of full-frame videos that maintain visual quality while increasing the stability of a video. Our technique contributes to correcting video shakes and has the advantage of generating visually useful imagery at low cost. Thus, we propose a novel hybrid full-frame video stabilization algorithm that produces full-frame videos after motion compensation with an IMU sensor. Evaluating our method against three metrics, the Stability score, Distortion value, and Cropping ratio, results indicated that stabilization was more effectively achieved with robustness to flow inaccuracy when effectively using an IMU sensor. In particular, among the evaluation outcomes, within the “Turn” category, our method exhibited an 18% enhancement in the Stability score and a 3% improvement in the Distortion value compared to the average results of previously proposed full-frame video stabilization-based methods, including PCA flow, neural rendering, and DIFRINT.

A decision model for OTT service users to adopt wireless D2D caching networks: Exploring the Korean case

As many Over-The-Top (OTT) users opt for mobile devices, mobile network traffic surges. However, current mobile communication technology faces difficulties. To provide a stable video content consumption experience to users, reliable alternatives must be found to resolve the limitation of the conventional approach. Accordingly, wireless Device-to-Device (D2D) caching networks have gained attention as a reliable alternative. Considering it can reduce network-related overheads, save related costs, and improve OTT video consumption environment, there is a high possibility that telecommunication operators want to commercialize this technology. To commercialize wireless D2D caching networks, users need to share their resources, such as personal information, smartphone storage, and battery. This study explores the decision-making process of mobile OTT users regarding D2D caching adoption on their mobile devices in Korea. An online survey of 94 respondents was conducted in South Korea. By using the analytic hierarchy process (AHP) approach, this study devised an evaluation model that includes three major factors: perceived benefits, perceived costs, and perceived situational conditions. The results reveal that risks, rather than situational factors and benefits, predominantly influence user intentions and strategies. Streaming video content via mobile data or Wi-Fi is viewed as the most feasible choice. Moreover, users show a preference for pre-downloading video content over adopting wireless D2D caching networks on their mobile devices. The findings provide a guideline for understanding the responses of mobile OTT service users and their priorities in responding to adopting wireless D2D caching networks.

ДО ПИТАННЯ ПОБУДОВИ АЛГОРИТМІВ ПЕРЕДАЧІ ПОТОКОВОГО ВІДЕО ДЛЯ МЕРЕЖ З НЕСТАБІЛЬНОЮ ПРОПУСКНОЮ ЗДАТНІСТЮ

Some questions of development of advanced video transmission algorithms that could robustly operate in the electronic warfare conditions are considered. The technology of adaptation of MJPEG compressed video stream for the networks with variable and unstable bandwidth is proposed. The new approach where connection control is separated from encoder basing on man-in-the-middle scenario is developed. In this case it became possible to preserve not only standard decoders on the client side, but the encoder code on the server side as well. The performance and reliability of the proposed MJPEG transmission algorithm was demonstrated in the field tests. The MJPEG transcoder based on developed algorithm demonstrated sufficiently stable video stream from the drone via WiFi network in the presence of impulse signal jamming.

Fast and stable pedicel detection for robust visual servoing to harvest shaking fruits

This study introduces a fast and stable pedicel detection method for robust visual servoing (RVS), supplemented with video stabilization (ViS) and fast pedicel detection, to realize automated harvesting of shaking fruits. By incorporating ViS, the system effectively mitigates the effects of motion blur, thereby ensuring consistent and precise object detection. In addition, the Fourier spectrum-based band-stop filter (FSBF) is used to improve clarity. The proposed approach also leverages the fast point feature histogram (FPFH) for fast pedicel detection, achieving real-time detection rates of 15–37 fps. Furthermore, it incorporates 6D pose estimation, culminating in the implementation of a 6D pose-based robust visual servoing (6DRVS) system. The performance of this system is evaluated using standard metrics such as perception accuracy, approach accuracy, precision, recall, accuracy, and F1-score in both preliminary tests and on-site experiments at two cucumber farms in Korea. The 6DRVS, supplemented with fast pedicel detection and ViS, exhibited improvements across all evaluation metrics. It recorded 90.00% perception accuracy, 82.22% approach accuracy, 0.957 precision, 0.938 recall, 0.900 accuracy, and 0.947 F1-score, highlighting its essential role in ensuring precise and efficient harvesting.

Digital Video Stabilization Method Based on Periodic Jitters of Airborne Vision of Large Flapping Wing Robots

Digital Video Stabilization Method Based on Periodic Jitters of Airborne Vision of Large Flapping Wing Robots

Adaptive Video Streaming in Multi-Tier Computing Networks: Joint Edge Transcoding and Client Enhancement

With the advancement of multimedia technology and wireless networks, there is a growing demand for high-quality video streaming. Delivering stable video streaming in extremely dynamic wireless networks, nevertheless, is still an open problem. Recent developments in client computing and mobile edge computing (MEC) technologies have both shown promise in enhancing the adaptive bitrate (ABR) streaming services. In this paper, we consider a video streaming system in multi-tier computing networks, enabled by joint edge-side video transcoding and client-side video enhancement. By “enhancement,” we mean that the client improves the video chunk quality via client-side image processing modules. In particular, we aim to design a joint bitrate adaptation, edge transcoding, and client image-processing algorithm, maximizing the quality of experience (QoE) of streaming services. The majority of the prior art has concentrated on super-resolution-enabled video streaming. Contrarily, we show that the video enhancement method outperforms the super-resolution approach in terms of signal-to-noise ratio and frames per second, implying a superior alternative for client-side processing in ABR streaming. We formulate the problem as an event-triggered Markov decision process (E-MDP), and propose a deep reinforcement learning (DRL)-based framework, named EDTEA. To deal with the delayed feedback induced by multi-tier computing, the entropy and the expected re-buffering terms are introduced to the objective and the reward, respectively. Extensive simulations based on real-world videos and bandwidth traces manifest that compared with state-of-the-art approaches, EDTEA provides <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$10.4\%\sim 78.4\%$</tex-math></inline-formula> extra QoE while reducing re-buffering time by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$85.5\%\sim 91.7\%$</tex-math></inline-formula> .

Exploring Temporal Feature Correlation for Efficient and Stable Video Semantic Segmentation

This paper tackles the problem of efficient and stable video semantic segmentation. While stability has been under-explored, prevalent work in efficient video semantic segmentation uses the keyframe paradigm. They efficiently process videos by only recomputing the low-level features and reusing high-level features computed at selected keyframes. In addition, the reused features stabilize the predictions across frames, thereby improving video consistency. However, dynamic scenes in the video can easily lead to misalignments between reused and recomputed features, which hampers performance. Moreover, relying on feature reuse to improve prediction consistency is brittle; an erroneous alignment of the features can easily lead to unstable predictions. Therefore, the keyframe paradigm exhibits a dilemma between stability and performance. We address this efficiency and stability challenge using a novel yet simple Temporal Feature Correlation (TFC) module. It uses the cosine similarity between two frames’ low-level features to inform the semantic label’s consistency across frames. Specifically, we selectively reuse label-consistent features across frames through linear interpolation and update others through sparse multi-scale deformable attention. As a result, we no longer directly reuse features to improve stability and thus effectively solve feature misalignment. This work provides a significant step towards efficient and stable video semantic segmentation. On the VSPW dataset, our method significantly improves the prediction consistency of image-based methods while being as fast and accurate.

Fast Coherent Video Style Transfer via Flow Errors Reduction

For video style transfer, naively applying still image techniques to process a video frame-by-frame independently often causes flickering artefacts. Some works adopt optical flow into the design of temporal constraint loss to secure temporal consistency. However, these works still suffer from incoherence (including ghosting artefacts) where large motions or occlusions occur, as optical flow fails to detect the boundaries of objects accurately. To address this problem, we propose a novel framework which consists of the following two stages: (1) creating new initialization images from proposed mask techniques, which are able to significantly reduce the flow errors; (2) process these initialized images iteratively with proposed losses to obtain stylized videos which are free of artefacts, which also increases the speed from over 3 min per frame to less than 2 s per frame for the gradient-based optimization methods. To be specific, we propose a multi-scale mask fusion scheme to reduce untraceable flow errors, and obtain an incremental mask to reduce ghosting artefacts. In addition, a multi-frame mask fusion scheme is designed to reduce traceable flow errors. In our proposed losses, the Sharpness Losses are used to deal with the potential image blurriness artefacts over long-range frames, and the Coherent Losses are performed to restrict the temporal consistency at both the multi-frame RGB level and Feature level. Overall, our approach produces stable video stylization outputs even in large motion or occlusion scenarios. The experiments demonstrate that the proposed method outperforms the state-of-the-art video style transfer methods qualitatively and quantitatively on the MPI Sintel dataset.

See farther and more: a master-slave UAVs based synthetic optical aperture imaging system with wide and dynamic baseline.

An open challenge remained in designing an optical system to capture the aerial image with a wide field of view (FoV) and high resolution. The optical system of one camera from a single unmanned aerial vehicle (UAV) can hardly promise the FoV and resolution. The conventional swarm UAVs can form the camera array with a short or fixed baseline. They can capture the images with a wide FoV and high resolution, but the cost is the requirement of many UAVs. We aim to design a camera array with a wide and dynamic baseline to reduce the demand for UAVs to organize a synthetic optical aperture. In this thought, we propose a master-slave UAVs-based synthetic optical aperture imaging system with a wide and dynamic baseline. The system consists of one master UAV and multiple slave UAVs. Master and slave UAVs provide the global and local FoVs, respectively, and improve the efficiency of image acquisition. In such a system, fusing UAV images becomes a new challenge due to two factors: (i) the small FoV overlap of slave UAVs and (ii) the gap in resolution scale from slave to master UAV images. To deal with it, a coarse-to-fine stitching method is proposed to stitch up the multi-view images into one to obtain a wide FoV with high resolution. A video stabilization method has also been designed for the proposed imaging system. Challenges caused by wide and dynamic baselines can thus be solved by the above methods. Actual data experiments demonstrate that the proposed imaging system achieves high-quality imaging results.

Background separation network for video anomaly detection

Video anomaly detection refers to the automatic identification of abnormal behaviors, objects, or events in videos. However, current methods for anomaly detection based on original frames lack a comprehensive understanding of the importance of foreground information, making it challenging to efficiently address video anomaly detection in the presence of complex background interference. In this paper, we propose a video anomaly detection algorithm based on Background Separation Network (BSN) to address this issue. Firstly, we utilize a video stabilization algorithm to reduce video jitter and enhance the quality of input video frames. Secondly, BSN shifts the focus from the entire frame to the foreground region with higher anomaly detection value. BSN utilizes the motion pixel distribution of the video as the basis for foreground extraction, enabling pixel-level background separation to obtain more accurate and complete foreground targets. Lastly, a certain proportion of foreground targets in the foreground image are masked as background, reducing the interference caused by redundant targets on the detection results. The proposed method achieves an accuracy of 96.2% on the UCSD ped2 dataset, demonstrating its effectiveness. This method contributes to accurately detecting abnormal behaviors in real-world surveillance videos to protect the safety of public lives and assets.

Video stabilization based on low‐rank constraint and trajectory optimization

AbstractVideo stabilization plays a pivotal role in enhancing video quality by eliminating unwanted jitter in shaky videos. This paper introduces a novel video stabilization algorithm that leverages low‐rank constraint and trajectory optimization to effectively eliminate undesirable motion and generate stabilized videos. In the proposed algorithm, a low‐rank constraint regularization term is incorporated to enhance the smoothness of motion trajectories. Additionally, a predictive path smoothness term is integrated to ensure the consistency of motion across neighbouring frames. To address the problem of excessive cropping resulting from aggressive smoothing, a flexible local window strategy that emphasizes local motion relationships within the trajectories is introduced. The experimental results show that, compared to other excellent video stabilization algorithms, the proposed algorithm improves the stability metric by approximately 2.3%. Furthermore, in the stabilized videos generated by the algorithm, an approximate improvement of 2.18 dB in average image temporal fidelity and a 5.7% increase in average structural similarity between adjacent frames are achieved. The code that implements the proposed method is publicly accessible athttps://github.com/CZS0319/VS_Low‐Rank_Constraint_Trajectory_Optimization.

DMCVS: Decomposed motion compensation‐based video stabilization

AbstractWith the popularity of handheld devices, video stabilization is becoming increasingly important. In previous studies, many methods have been proposed to stabilize shaky videos. However, these methods fail to balance between image content integrity and stability. Some methods sacrifice image content for better stability. Other methods ignore the subtle jitters, which leads to poor stability. This work innovatively proposes a video stabilization method based on decomposed motion compensation. First, a grid‐based motion statistics method is adopted for motion estimation, which obtains more accurate motion vectors according to matched likelihood estimates. Then, the motion compensation is inherently decomposed into two parts: linear motion compensation and auxiliary motion compensation. Linear motion compensation removes complex jitter by constructing linear path constraints to obtain a more stable camera path. Auxiliary motion compensation uses a moving average filter to remove the high‐frequency jitter as a supplement and preserve more image content. The two components are combined with individual weights to derive the final transform matrix and warp the original frames. Experimental results show that our method outperforms the previous methods on NUS and DeepStab datasets qualitatively and quantitatively.

Mining the Micro-Trajectory of Two-Wheeled Non-Motorized Vehicles Based on the Improved YOLOx

Two-wheeled non-motorized vehicles (TNVs) have become the primary mode of transportation for short-distance travel among residents in many underdeveloped cities in China due to their convenience and low cost. However, this trend also brings corresponding risks of traffic accidents. Therefore, it is necessary to analyze the driving behavior characteristics of TNVs through their trajectory data in order to provide guidance for traffic safety. Nevertheless, the compact size, agile steering, and high maneuverability of these TNVs pose substantial challenges in acquiring high-precision trajectories. These characteristics complicate the tracking and analysis processes essential for understanding their movement patterns. To tackle this challenge, we propose an enhanced You Only Look Once Version X (YOLOx) model, which incorporates a median pooling-Convolutional Block Attention Mechanism (M-CBAM). This model is specifically designed for the detection of TNVs, and aims to improve accuracy and efficiency in trajectory tracking. Furthermore, based on this enhanced YOLOx model, we have developed a micro-trajectory data mining framework specifically for TNVs. Initially, the paper establishes an aerial dataset dedicated to the detection of TNVs, which then serves as a foundational resource for training the detection model. Subsequently, an augmentation of the Convolutional Block Attention Mechanism (CBAM) is introduced, integrating median pooling to amplify the model’s feature extraction capabilities. Subsequently, additional detection heads are integrated into the YOLOx model to elevate the detection rate of small-scale targets, particularly focusing on TNVs. Concurrently, the Deep Sort algorithm is utilized for the precise tracking of vehicle targets. The process culminates with the reconstruction of trajectories, which is achieved through a combination of video stabilization, coordinate mapping, and filtering denoising techniques. The experimental results derived from our self-constructed dataset reveal that the enhanced YOLOx model demonstrates superior detection performance in comparison to other analogous methods. The comprehensive framework accomplishes an average trajectory recall rate of 85% across three test videos. This significant achievement provides a reliable method for data acquisition, which is essential for investigating the micro-level operational mechanisms of TNVs. The results of this study can further contribute to the understanding and improvement of traffic safety on mixed-use roads.