Video Stabilization Research Articles

Optical Flow as Spatial-Temporal Attention Learners.

Optical flow is an indispensable building block for various important computer vision tasks, including motion estimation, object tracking, and disparity measurement. To date, the dominant methods are CNN-based, leaving plenty of room for improvement. In this work, we propose TransFlow, a transformer architecture for optical flow estimation. Compared to dominant CNN-based methods, TransFlow demonstrates three advantages. First, it provides more accurate correlation and trustworthy matching in flow estimation by utilizing spatial self-attention and cross-attention mechanisms between adjacent frames to effectively capture global dependencies; Second, it recovers more compromised information (e.g., occlusion and motion blur) in flow estimation through long-range temporal association in dynamic scenes; Third, it introduces a concise self-learning paradigm, eliminating the need for complex and laborious multi-stage pre-training procedures. The versatility and superiority of TransFlow extend seamlessly to 3D scene motion, yielding competitive outcomes in 3D scene flow estimation. Our approach attains state-of-the-art results on benchmark datasets such as Sintel and KITTI-15, while also exhibiting exceptional performance on downstream tasks, including video object detection using the ImageNet VID dataset, video frame interpolation using the GoPro dataset, and video stabilization using the DeepStab dataset. We believe that the effectiveness of TransFlow positions it as a flexible baseline for both optical flow and scene flow estimation, offering promising avenues for future research and development.

Blind 3D Video Stabilization with Spatio-Temporally Varying Motion Blur

Video stabilization is a challenging task that attempts to compensate for the overall frame shake during video acquisition. Existing three-dimensional video stabilization methods aim at modeling camera perspective projection through either data-driven training or explicit motion estimation. However, the above methods are difficult to effectively solve the issue of shaky videos with abrupt object movements, resulting in local motion blur in the direction of the movement. This phenomenon is prevalent in real-world scenarios featuring foreground blind motion scenes. Unfortunately, directly combining stabilization and deblurring methods poses challenges when dealing with this situation. In the video, the intensity of motion blur undergoes continuous changes, and the direct combination method inadequately utilizes spatiotemporal information, providing insufficient clues for cross-frame compensation. To alleviate this problem, the Cross-frame-temporal Module framework is proposed to address blind motion blur induced by various conditions, which utilizes cross-frame temporal features to estimate depth maps and camera motion. In this framework, a Blur Transform Network (BTNet) is designed to adapt to spatially varying motion blur, which transforms local regions according to the impact of blur intensities to adapt to the effects of non-uniform motion blur; furthermore, our Temporal-Aware Network (TANet) further suppresses motion blur by leveraging cross-frame temporal features. In addition, the limited availability of pair-training video data containing motion blur limits the application of this approach in practice. The Cross-frame-temporal Module framework adopts an un-pretrained in-test training strategy. Extensive experimental results have demonstrated that our method outperforms state-of-the-art methods.

Dual-Modality Cross-Interaction-Based Hybrid Full-Frame Video Stabilization

This study aims to generate visually useful imagery by preventing cropping while maintaining resolution and minimizing the degradation of stability and distortion to enhance the stability of a video for Augmented Reality applications. The focus is placed on conducting research that balances maintaining execution speed with performance improvements. By processing Inertial Measurement Unit (IMU) sensor data using the Versatile Quaternion-based Filter algorithm and optical flow, our research first applies motion compensation to frames of input video. To address cropping, PCA-flow-based video stabilization is then performed. Furthermore, to mitigate distortion occurring during the full-frame video creation process, neural rendering is applied, resulting in the output of stabilized frames. The anticipated effect of using an IMU sensor is the production of full-frame videos that maintain visual quality while increasing the stability of a video. Our technique contributes to correcting video shakes and has the advantage of generating visually useful imagery at low cost. Thus, we propose a novel hybrid full-frame video stabilization algorithm that produces full-frame videos after motion compensation with an IMU sensor. Evaluating our method against three metrics, the Stability score, Distortion value, and Cropping ratio, results indicated that stabilization was more effectively achieved with robustness to flow inaccuracy when effectively using an IMU sensor. In particular, among the evaluation outcomes, within the “Turn” category, our method exhibited an 18% enhancement in the Stability score and a 3% improvement in the Distortion value compared to the average results of previously proposed full-frame video stabilization-based methods, including PCA flow, neural rendering, and DIFRINT.

A decision model for OTT service users to adopt wireless D2D caching networks: Exploring the Korean case

As many Over-The-Top (OTT) users opt for mobile devices, mobile network traffic surges. However, current mobile communication technology faces difficulties. To provide a stable video content consumption experience to users, reliable alternatives must be found to resolve the limitation of the conventional approach. Accordingly, wireless Device-to-Device (D2D) caching networks have gained attention as a reliable alternative. Considering it can reduce network-related overheads, save related costs, and improve OTT video consumption environment, there is a high possibility that telecommunication operators want to commercialize this technology. To commercialize wireless D2D caching networks, users need to share their resources, such as personal information, smartphone storage, and battery. This study explores the decision-making process of mobile OTT users regarding D2D caching adoption on their mobile devices in Korea. An online survey of 94 respondents was conducted in South Korea. By using the analytic hierarchy process (AHP) approach, this study devised an evaluation model that includes three major factors: perceived benefits, perceived costs, and perceived situational conditions. The results reveal that risks, rather than situational factors and benefits, predominantly influence user intentions and strategies. Streaming video content via mobile data or Wi-Fi is viewed as the most feasible choice. Moreover, users show a preference for pre-downloading video content over adopting wireless D2D caching networks on their mobile devices. The findings provide a guideline for understanding the responses of mobile OTT service users and their priorities in responding to adopting wireless D2D caching networks.

ДО ПИТАННЯ ПОБУДОВИ АЛГОРИТМІВ ПЕРЕДАЧІ ПОТОКОВОГО ВІДЕО ДЛЯ МЕРЕЖ З НЕСТАБІЛЬНОЮ ПРОПУСКНОЮ ЗДАТНІСТЮ

Some questions of development of advanced video transmission algorithms that could robustly operate in the electronic warfare conditions are considered. The technology of adaptation of MJPEG compressed video stream for the networks with variable and unstable bandwidth is proposed. The new approach where connection control is separated from encoder basing on man-in-the-middle scenario is developed. In this case it became possible to preserve not only standard decoders on the client side, but the encoder code on the server side as well. The performance and reliability of the proposed MJPEG transmission algorithm was demonstrated in the field tests. The MJPEG transcoder based on developed algorithm demonstrated sufficiently stable video stream from the drone via WiFi network in the presence of impulse signal jamming.

Fast and stable pedicel detection for robust visual servoing to harvest shaking fruits

This study introduces a fast and stable pedicel detection method for robust visual servoing (RVS), supplemented with video stabilization (ViS) and fast pedicel detection, to realize automated harvesting of shaking fruits. By incorporating ViS, the system effectively mitigates the effects of motion blur, thereby ensuring consistent and precise object detection. In addition, the Fourier spectrum-based band-stop filter (FSBF) is used to improve clarity. The proposed approach also leverages the fast point feature histogram (FPFH) for fast pedicel detection, achieving real-time detection rates of 15–37 fps. Furthermore, it incorporates 6D pose estimation, culminating in the implementation of a 6D pose-based robust visual servoing (6DRVS) system. The performance of this system is evaluated using standard metrics such as perception accuracy, approach accuracy, precision, recall, accuracy, and F1-score in both preliminary tests and on-site experiments at two cucumber farms in Korea. The 6DRVS, supplemented with fast pedicel detection and ViS, exhibited improvements across all evaluation metrics. It recorded 90.00% perception accuracy, 82.22% approach accuracy, 0.957 precision, 0.938 recall, 0.900 accuracy, and 0.947 F1-score, highlighting its essential role in ensuring precise and efficient harvesting.

Digital Video Stabilization Method Based on Periodic Jitters of Airborne Vision of Large Flapping Wing Robots

Digital Video Stabilization Method Based on Periodic Jitters of Airborne Vision of Large Flapping Wing Robots

Adaptive Video Streaming in Multi-Tier Computing Networks: Joint Edge Transcoding and Client Enhancement

With the advancement of multimedia technology and wireless networks, there is a growing demand for high-quality video streaming. Delivering stable video streaming in extremely dynamic wireless networks, nevertheless, is still an open problem. Recent developments in client computing and mobile edge computing (MEC) technologies have both shown promise in enhancing the adaptive bitrate (ABR) streaming services. In this paper, we consider a video streaming system in multi-tier computing networks, enabled by joint edge-side video transcoding and client-side video enhancement. By “enhancement,” we mean that the client improves the video chunk quality via client-side image processing modules. In particular, we aim to design a joint bitrate adaptation, edge transcoding, and client image-processing algorithm, maximizing the quality of experience (QoE) of streaming services. The majority of the prior art has concentrated on super-resolution-enabled video streaming. Contrarily, we show that the video enhancement method outperforms the super-resolution approach in terms of signal-to-noise ratio and frames per second, implying a superior alternative for client-side processing in ABR streaming. We formulate the problem as an event-triggered Markov decision process (E-MDP), and propose a deep reinforcement learning (DRL)-based framework, named EDTEA. To deal with the delayed feedback induced by multi-tier computing, the entropy and the expected re-buffering terms are introduced to the objective and the reward, respectively. Extensive simulations based on real-world videos and bandwidth traces manifest that compared with state-of-the-art approaches, EDTEA provides <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$10.4\%\sim 78.4\%$</tex-math></inline-formula> extra QoE while reducing re-buffering time by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$85.5\%\sim 91.7\%$</tex-math></inline-formula> .

Exploring Temporal Feature Correlation for Efficient and Stable Video Semantic Segmentation

This paper tackles the problem of efficient and stable video semantic segmentation. While stability has been under-explored, prevalent work in efficient video semantic segmentation uses the keyframe paradigm. They efficiently process videos by only recomputing the low-level features and reusing high-level features computed at selected keyframes. In addition, the reused features stabilize the predictions across frames, thereby improving video consistency. However, dynamic scenes in the video can easily lead to misalignments between reused and recomputed features, which hampers performance. Moreover, relying on feature reuse to improve prediction consistency is brittle; an erroneous alignment of the features can easily lead to unstable predictions. Therefore, the keyframe paradigm exhibits a dilemma between stability and performance. We address this efficiency and stability challenge using a novel yet simple Temporal Feature Correlation (TFC) module. It uses the cosine similarity between two frames’ low-level features to inform the semantic label’s consistency across frames. Specifically, we selectively reuse label-consistent features across frames through linear interpolation and update others through sparse multi-scale deformable attention. As a result, we no longer directly reuse features to improve stability and thus effectively solve feature misalignment. This work provides a significant step towards efficient and stable video semantic segmentation. On the VSPW dataset, our method significantly improves the prediction consistency of image-based methods while being as fast and accurate.

Fast Coherent Video Style Transfer via Flow Errors Reduction

For video style transfer, naively applying still image techniques to process a video frame-by-frame independently often causes flickering artefacts. Some works adopt optical flow into the design of temporal constraint loss to secure temporal consistency. However, these works still suffer from incoherence (including ghosting artefacts) where large motions or occlusions occur, as optical flow fails to detect the boundaries of objects accurately. To address this problem, we propose a novel framework which consists of the following two stages: (1) creating new initialization images from proposed mask techniques, which are able to significantly reduce the flow errors; (2) process these initialized images iteratively with proposed losses to obtain stylized videos which are free of artefacts, which also increases the speed from over 3 min per frame to less than 2 s per frame for the gradient-based optimization methods. To be specific, we propose a multi-scale mask fusion scheme to reduce untraceable flow errors, and obtain an incremental mask to reduce ghosting artefacts. In addition, a multi-frame mask fusion scheme is designed to reduce traceable flow errors. In our proposed losses, the Sharpness Losses are used to deal with the potential image blurriness artefacts over long-range frames, and the Coherent Losses are performed to restrict the temporal consistency at both the multi-frame RGB level and Feature level. Overall, our approach produces stable video stylization outputs even in large motion or occlusion scenarios. The experiments demonstrate that the proposed method outperforms the state-of-the-art video style transfer methods qualitatively and quantitatively on the MPI Sintel dataset.

See farther and more: a master-slave UAVs based synthetic optical aperture imaging system with wide and dynamic baseline.

An open challenge remained in designing an optical system to capture the aerial image with a wide field of view (FoV) and high resolution. The optical system of one camera from a single unmanned aerial vehicle (UAV) can hardly promise the FoV and resolution. The conventional swarm UAVs can form the camera array with a short or fixed baseline. They can capture the images with a wide FoV and high resolution, but the cost is the requirement of many UAVs. We aim to design a camera array with a wide and dynamic baseline to reduce the demand for UAVs to organize a synthetic optical aperture. In this thought, we propose a master-slave UAVs-based synthetic optical aperture imaging system with a wide and dynamic baseline. The system consists of one master UAV and multiple slave UAVs. Master and slave UAVs provide the global and local FoVs, respectively, and improve the efficiency of image acquisition. In such a system, fusing UAV images becomes a new challenge due to two factors: (i) the small FoV overlap of slave UAVs and (ii) the gap in resolution scale from slave to master UAV images. To deal with it, a coarse-to-fine stitching method is proposed to stitch up the multi-view images into one to obtain a wide FoV with high resolution. A video stabilization method has also been designed for the proposed imaging system. Challenges caused by wide and dynamic baselines can thus be solved by the above methods. Actual data experiments demonstrate that the proposed imaging system achieves high-quality imaging results.

Background separation network for video anomaly detection

Video anomaly detection refers to the automatic identification of abnormal behaviors, objects, or events in videos. However, current methods for anomaly detection based on original frames lack a comprehensive understanding of the importance of foreground information, making it challenging to efficiently address video anomaly detection in the presence of complex background interference. In this paper, we propose a video anomaly detection algorithm based on Background Separation Network (BSN) to address this issue. Firstly, we utilize a video stabilization algorithm to reduce video jitter and enhance the quality of input video frames. Secondly, BSN shifts the focus from the entire frame to the foreground region with higher anomaly detection value. BSN utilizes the motion pixel distribution of the video as the basis for foreground extraction, enabling pixel-level background separation to obtain more accurate and complete foreground targets. Lastly, a certain proportion of foreground targets in the foreground image are masked as background, reducing the interference caused by redundant targets on the detection results. The proposed method achieves an accuracy of 96.2% on the UCSD ped2 dataset, demonstrating its effectiveness. This method contributes to accurately detecting abnormal behaviors in real-world surveillance videos to protect the safety of public lives and assets.

Video stabilization based on low‐rank constraint and trajectory optimization

AbstractVideo stabilization plays a pivotal role in enhancing video quality by eliminating unwanted jitter in shaky videos. This paper introduces a novel video stabilization algorithm that leverages low‐rank constraint and trajectory optimization to effectively eliminate undesirable motion and generate stabilized videos. In the proposed algorithm, a low‐rank constraint regularization term is incorporated to enhance the smoothness of motion trajectories. Additionally, a predictive path smoothness term is integrated to ensure the consistency of motion across neighbouring frames. To address the problem of excessive cropping resulting from aggressive smoothing, a flexible local window strategy that emphasizes local motion relationships within the trajectories is introduced. The experimental results show that, compared to other excellent video stabilization algorithms, the proposed algorithm improves the stability metric by approximately 2.3%. Furthermore, in the stabilized videos generated by the algorithm, an approximate improvement of 2.18 dB in average image temporal fidelity and a 5.7% increase in average structural similarity between adjacent frames are achieved. The code that implements the proposed method is publicly accessible athttps://github.com/CZS0319/VS_Low‐Rank_Constraint_Trajectory_Optimization.

DMCVS: Decomposed motion compensation‐based video stabilization

AbstractWith the popularity of handheld devices, video stabilization is becoming increasingly important. In previous studies, many methods have been proposed to stabilize shaky videos. However, these methods fail to balance between image content integrity and stability. Some methods sacrifice image content for better stability. Other methods ignore the subtle jitters, which leads to poor stability. This work innovatively proposes a video stabilization method based on decomposed motion compensation. First, a grid‐based motion statistics method is adopted for motion estimation, which obtains more accurate motion vectors according to matched likelihood estimates. Then, the motion compensation is inherently decomposed into two parts: linear motion compensation and auxiliary motion compensation. Linear motion compensation removes complex jitter by constructing linear path constraints to obtain a more stable camera path. Auxiliary motion compensation uses a moving average filter to remove the high‐frequency jitter as a supplement and preserve more image content. The two components are combined with individual weights to derive the final transform matrix and warp the original frames. Experimental results show that our method outperforms the previous methods on NUS and DeepStab datasets qualitatively and quantitatively.

Mining the Micro-Trajectory of Two-Wheeled Non-Motorized Vehicles Based on the Improved YOLOx

Two-wheeled non-motorized vehicles (TNVs) have become the primary mode of transportation for short-distance travel among residents in many underdeveloped cities in China due to their convenience and low cost. However, this trend also brings corresponding risks of traffic accidents. Therefore, it is necessary to analyze the driving behavior characteristics of TNVs through their trajectory data in order to provide guidance for traffic safety. Nevertheless, the compact size, agile steering, and high maneuverability of these TNVs pose substantial challenges in acquiring high-precision trajectories. These characteristics complicate the tracking and analysis processes essential for understanding their movement patterns. To tackle this challenge, we propose an enhanced You Only Look Once Version X (YOLOx) model, which incorporates a median pooling-Convolutional Block Attention Mechanism (M-CBAM). This model is specifically designed for the detection of TNVs, and aims to improve accuracy and efficiency in trajectory tracking. Furthermore, based on this enhanced YOLOx model, we have developed a micro-trajectory data mining framework specifically for TNVs. Initially, the paper establishes an aerial dataset dedicated to the detection of TNVs, which then serves as a foundational resource for training the detection model. Subsequently, an augmentation of the Convolutional Block Attention Mechanism (CBAM) is introduced, integrating median pooling to amplify the model’s feature extraction capabilities. Subsequently, additional detection heads are integrated into the YOLOx model to elevate the detection rate of small-scale targets, particularly focusing on TNVs. Concurrently, the Deep Sort algorithm is utilized for the precise tracking of vehicle targets. The process culminates with the reconstruction of trajectories, which is achieved through a combination of video stabilization, coordinate mapping, and filtering denoising techniques. The experimental results derived from our self-constructed dataset reveal that the enhanced YOLOx model demonstrates superior detection performance in comparison to other analogous methods. The comprehensive framework accomplishes an average trajectory recall rate of 85% across three test videos. This significant achievement provides a reliable method for data acquisition, which is essential for investigating the micro-level operational mechanisms of TNVs. The results of this study can further contribute to the understanding and improvement of traffic safety on mixed-use roads.

High interaction potential of online-only courses with breakout sessions-results of apilot study

The flipped classroom (FC) is nowadays apopular principle of blended student-centered learning. Students first prepare basic knowledge at home and subsequently meet for consolidation and amore in-depth look at acertain topic. During the COVID-19 pandemic, several groups developed approaches to also transform the characteristic FC second session into an online-only format. Herein, we present apilot study on establishing an online-only FC format with elements of collaborative learning and its evaluation by medical students. The FC design related to diseases of the salivary glands was transformed into an online-only event. After studying the basic information online, supported by self-made interactive videos and/or lecture recordings, students met in avideo conference enriched by breakout sessions, interactive demonstration of the related clinical examination including ultrasound, and aformative assessment. Aquestionnaire with 27items was answered by participants to evaluate the concept and the event. Use of common hard- and software systems led to atechnically stable video conference. Atotal of 55students completed the questionnaire and were included into data analysis. During the breakout sessions, lively interaction between participants was observed. The evaluation of both the event itself and the related learning progress showed good results despite the lack of preparation beforehand in 27% of participants. Online-only FC designs can result in high satisfaction. High quality of online preparation, asolid technical platform, accurate time management, and areasonable selection of topics are the main parameters contributing to successful course design. Nowadays, embedding medical imaging can be realized in appropriate quality for educational purposes. The implementation of breakout sessions and voting tools enables collaborative online learning with high levels of interaction and satisfaction for both teachers and students.

GLOCAL: A self-supervised learning framework for global and local motion estimation

Motions in videos are typically a mixture of local dynamic object motions and global camera motion, which are inconsistent in some cases, and even interfere with each other, causing difficulties in various downstream applications, such as video stabilization that requires the global motion, and action recognition that consumes local motions. Therefore, it is crucial to estimate them separately. Existing methods separate two motions from the mixed motion fields, such as optical flow. However, the quality of mixed motion determines the higher bounds of the performance. In this work, we propose a framework, GLOCAL, to directly estimate global and local motions simultaneously from adjacent frames in a self-supervised manner. Our GLOCAL consists of a Global Motion Estimation (GME) module and a Local Motion Estimation (LME) module. The GME module involves a mixed motion estimation backbone, an implicit bottleneck structure for feature dimension reduction, and an explicit bottleneck for global motion recovery based on the global motion bases with foreground mask under the training guidance of proposed global reconstruction loss. An attention U-Net is adopted for LME which produces local motions while excluding motion of irrelevant regions under the guidance of proposed local reconstruction loss. Our method can achieve better performance than the existing methods on the homography estimation dataset DHE and the action recognition dataset NCAA and UCF-101.

Boosting UAVs Live Uplink Streaming by Video Stabilization

Boosting UAVs Live Uplink Streaming by Video Stabilization

Minimum Latency Deep Online Video Stabilization and Its Extensions.

We present a novel deep camera path optimization framework for minimum latency online video stabilization. Typically, a stabilization pipeline consists of three steps: motion estimation, path smoothing, and novel view synthesis. Most previous methods concentrate on motion estimation while path optimization receives less attention, particularly in the crucial online setting where future frames are inaccessible. In this work, we adopt off-the-shelf high-quality deep motion models for motion estimation and focus only on the path optimization. Specifically, our camera path smoothing network takes a short 2D camera path in a sliding window as input and outputs the stabilizing warp field of the last frame, which warps the coming frame to its stabilized position. We explore three motion densities: a global single camera path, local mesh-based bundled paths, and dense flow paths. A hybrid loss and an efficient motion smoothing attention (EMSA) module are proposed for spatially and temporally consistent path smoothing. Moreover, we build a motion dataset that contains stable and unstable motion pairs for training. Extensive experiments demonstrate that our method surpasses state-of-the-art online stabilization methods and rivals the performance of offline methods, offering compelling advancements in the field of video stabilization. Code and dataset are available at https://github.com/liuzhen03/NNDVS.

A Fast Horizon Detector and a New Annotated Dataset for Maritime Video Processing

Accurate and fast sea horizon detection is vital for tasks in autonomous navigation and maritime security, such as video stabilization, target region reduction, precise tracking, and obstacle avoidance. This paper introduces a novel sea horizon detector from RGB videos, focusing on rapid and effective sea noise suppression while preserving weak horizon edges. Line fitting methods are subsequently employed on filtered edges for horizon detection. We address the filtering problem by extracting line segments with a very low edge threshold, ensuring the detection of line segments even in low-contrast horizon conditions. We show that horizon line segments have simple and relevant properties in RGB images, which we exploit to suppress noisy segments. Then we use the surviving segments to construct a filtered edge map and infer the horizon from the filtered edges. We propose a careful incorporation of temporal in- formation for horizon inference and experimentally show its effectiveness. We address the computational constraint by providing a vectorized implementation for efficient CPU execution, and leveraging image downsizing with minimal loss of accuracy on the original size. Moreover, we contribute a public horizon line dataset to enrich existing data resources. After extensive tests, we report the following major findings: 1) thanks to its filter, our algorithm accurately detects horizon lines with low or weak edge response, 2) the vectorized filter takes no more than 1.71% of the overall computations, while most of the computations are taken by the Line Segment Detection (LSD) algorithm we integrated into our pipeline, 3) our strategy of incorporating the temporal information avoids outlier detections, mitigates the effect of strong noisy lines, and exhibits high robustness when using incorrect detections as a temporal reference. Our algorithm’s performance is rigorously evaluated against state-of-the-art methods, and its core components are validated through ablation experiments.