Optical Flow Estimation Research Articles

Multi-objective optimization based robotic path planning for interpolation reconstruction model of CT data

Robotic path planning plays a pivotal role in computer-aided liver tumor thermal ablation surgery. However, traditional methods face challenges in terms of low reconstruction efficiency and planning safety. To address these issues, we propose a method for robotic path planning of liver tumor thermal ablation surgery. Firstly, an interlayer interpolation algorithm based on optical flow estimation is utilized to compensate for large interlayer spacing in computed tomography (CT) images by inserting predicted images between sequence images. Secondly, the voxel traversal strategy and patch intersection calculation strategy in the standard marching cube (MC) algorithm is optimized to improve the efficiency of abdominal tissues reconstruction. Finally, comprehensive clinical constraints are summarized to ensure the surgery safety and the strength pareto evolutionary algorithm II (SPEA-II) is leveraged to optimize surgery path, which can obtain even distribution of solutions in high-dimension optimization problems. Extensive experiments conducted on the 3Dircadb and SLIVER07 datasets revealed that our proposed method reduces reconstruction time by 21.5% compared to the standard MC algorithm, while achieving an average overlap rate of 88.25% and an average Hausdorff distance of 15.25 mm between Pareto front points and surgeon's recommended puncture points.

Implementation of an indoor optical camera communication and localization fusion system using infrared LED markers

Recently, with the development of communication and positioning technologies, progressively higher demands have been placed on the timeliness of indoor communication and the effectiveness of indoor positioning. This paper proposes a low-cost and full-time domain coverage indoor communication and localization fusion system that uses an infrared camera to identify mobile LEDs based on region of interest optical camera communication (RoI-OCC) and calculate three-dimensional (3D) world coordinates based on perspective-n-point (PnP) algorithm. An intermediate delimiter modulation-demodulation method is designed to achieve asynchronous OCC. In processing infrared images, a region-growth-based absolute-directional-mean-difference (RG-ADMD) algorithm is developed to detect infrared targets and recover the scale. Additionally, a local-contrast-method-based Lucas-Kanade (LCM-LK) optical flow estimation algorithm is designed to track infrared targets and determine the centroid coordinates of pixels. In a 5×5 m experimental area, the established system can achieve a target detection recall rate of over 85%, the asynchronous OCC with error-free transmission and an average 3D positioning accuracy of 2.01 cm under a LCM-LK tracking algorithm.

A Two-Stage Facial Kinematic Control Strategy for Humanoid Robots Based on Keyframe Detection and Keypoint Cubic Spline Interpolation

A plentiful number of facial expressions is the basis of natural human–robot interaction for high-fidelity humanoid robots. The facial expression imitation of humanoid robots involves the transmission of human facial expression data to servos situated within the robot’s head. These data drive the servos to manipulate the skin, thereby enabling the robot to exhibit various facial expressions. However, since the mechanical transmission rate cannot keep up with the data processing rate, humanoid robots often suffer from jitters in the imitation. We conducted a thorough analysis of the transmitted facial expression sequence data and discovered that they are extremely redundant. Therefore, we designed a two-stage strategy for humanoid robots based on facial keyframe detection and facial keypoint detection to achieve more natural and smooth expression imitation. We first built a facial keyframe detection model based on ResNet-50, combined with optical flow estimation, which can identify key expression frames in the sequence. Then, a facial keypoint detection model is used on the keyframes to obtain the facial keypoint coordinates. Based on the coordinates, the cubic spline interpolation method is used to obtain the motion trajectory parameters of the servos, thus realizing the robust control of the humanoid robot’s facial expression. Experiments show that, unlike before where the robot’s imitation would stutter at frame rates above 25 fps, our strategy allows the robot to maintain good facial expression imitation similarity (cosine similarity of 0.7226), even at higher frame rates.

Full-field displacement measurements of structural vibrations using a novel two-stage neural network

This paper presents a novel two-stage neural network for quantifying full-field displacements of vibrating structures. A graphics software was used to simulate structural vibrations with random displacement fields and generate image datasets through batch rendering. Subsequently, a two-stage neural network model comprising an object segmentation subnetwork for extracting structural shapes and an optical-flow subnetwork for identifying displacement fields was constructed. A coordinate convolution module was embedded in the optical-flow estimation model and the two subnetworks were integrated to develop a comprehensive model. An experiment was designed to monitor the vibration displacements in a cantilever column. The results showed that the proposed model achieved optimal recognition accuracy with a mean absolute error of 0.1477 pixels and root-mean-squared error of 0.2335 pixels. Additionally, it exhibited robustness to various lighting conditions. Furthermore, case studies on real-world structural vibration videos substantiated the practical engineering application value of the proposed model.

Optical Flow as Spatial-Temporal Attention Learners.

Optical flow is an indispensable building block for various important computer vision tasks, including motion estimation, object tracking, and disparity measurement. To date, the dominant methods are CNN-based, leaving plenty of room for improvement. Inthiswork, we propose TransFlow, a transformer architecture for optical flow estimation. Compared to dominant CNN-based methods, TransFlow demonstrates three advantages. First, it provides more accurate correlation and trustworthy matching in flow estimation by utilizing spatial self-attention and cross-attention mechanisms between adjacent frames to effectively capture global dependencies; Second, it recovers more compromised information (e.g. occlusion and motion blur) in flow estimation through long-range temporal association in dynamic scenes; Third, it introduces a concise self-learning paradigm, eliminating the need for complex and laborious multi-stage pre-training procedures. The versatility and superiority of TransFlow extend seamlessly to 3D scene motion, yielding competitive outcomes in 3D scene flow estimation. Our approach attains state-of-the-art results on benchmark datasets such as Sintel and KITTI-15, while also exhibiting exceptional performance on downstream tasks, including video object detection using the ImageNet VID dataset, video frame interpolation using the GoPro dataset, and video stabilization using the DeepStab dataset. We believe that the effectiveness of TransFlow positions it as a flexible baseline for both optical flow and scene flow estimation, offering promising avenues for future research and development.

Enhanced neural video compression for cloud gaming videos with aligned frame generation

The burgeoning popularity of cloud gaming makes it critical for efficient video compression to relieve the growing bandwidth pressure. While existing neural video coding approaches have demonstrated strong compression potential on natural videos, there is an absence of efficient neural codecs dedicated to gaming videos. To bridge this gap, in this paper, we propose an end-to-end neural video compression method designed specifically for cloud gaming videos. By effectively utilizing the unique camera motion information inherent to cloud gaming, the previous reconstructed frame is maximally aligned to the current frame through a learning-based module with multiple losses, which then replaces the previous reconstructed frame for optical flow estimation. By significantly reducing the displacement between two consecutive frames caused by camera motion, the motion estimation accuracy is enhanced, effectively handling the large and abrupt motion scenarios frequently present in gaming videos. Furthermore, the aligned tensor obtained in the previous step is used to enhance the latent prior of the entropy model, providing a superior temporal prior for coding. Extensive experimental results demonstrate the superior performance of our proposed method compared to one of the previous state-of-the-art approaches, DCVC-HEM, providing significant progress in end-to-end neural compression in cloud gaming videos.

Evolution of transformer-based optical flow estimation techniques: a survey

Evolution of transformer-based optical flow estimation techniques: a survey

Unsupervised Illumination Adaptation for Low-Light Vision.

Insufficient lighting poses challenges to both human and machine visual analytics. While existing low-light enhancement methods prioritize human visual perception, they often neglect machine vision and high-level semantics. In this paper, we make pioneering efforts to build an illumination enhancement model for high-level vision. Drawing inspiration from camera response functions, our model could enhance images from the machine vision perspective despite being lightweight in architecture and simple in formulation. We also introduce two approaches that leverage knowledge from base enhancement curves and self-supervised pretext tasks to train for different downstream normal-to-low-light adaptation scenarios. Our proposed framework overcomes the limitations of existing algorithms without requiring access to labeled data in low-light conditions. It facilitates more effective illumination restoration and feature alignment, significantly improving the performance of downstream tasks in a plug-and-play manner. This research advances the field of low-light machine analytics and broadly applies to various high-level vision tasks, including classification, face detection, optical flow estimation, and video action recognition.

An Autonomous Intelligent Liability Determination Method for Minor Accidents Based on Collision Detection and Large Language Models

With the rapid increase in the number of vehicles on the road, minor traffic accidents have become more frequent, contributing significantly to traffic congestion and disruptions. Traditional methods for determining responsibility in such accidents often require human intervention, leading to delays and inefficiencies. This study proposed a fully intelligent method for liability determination in minor accidents, utilizing collision detection and large language models. The approach integrated advanced vehicle recognition using the YOLOv8 algorithm coupled with a minimum mean square error filter for real-time target tracking. Additionally, an improved global optical flow estimation algorithm and support vector machines were employed to accurately detect traffic accidents. Key frames from accident scenes were extracted and analyzed using the GPT4-Vision-Preview model to determine liability. Simulation experiments demonstrated that the proposed method accurately and efficiently detected vehicle collisions, rapidly determined liability, and generated detailed accident reports. The method achieved the fully automated AI processing of minor traffic accidents without manual intervention, ensuring both objectivity and fairness.

Development of a body movement detection system to avoid re-exposure during radiography.

During the radiographic examination of the chest and bones in hospitals, communicating and maintaining posture is difficult for some patients, and movement before or during X-ray irradiation may necessitate re-exposure owing to body wobbling movements or breathing movements. To prevent the need for re-exposure during radiography and to determine the exposure timing, a body movement detection system that considers breathing movements was developed in this study. The posture of a patient was monitored using an RGB camera. The acquired video data was analyzed to detect body movement using either an inter-frame difference method or an optical flow estimation method. The performance of the system was evaluated by detecting the body and breathing movements during positioning. Consequently, the inter-frame difference method detected 179.8-1222.2pixels during body movements, and the optical flow estimation method confirmed that the feature points moved by 5.5-26.6mm (4.2-20.3pixels). When detecting breathing movements, 82-585pixels were detected by the inter-frame difference method, and the optical flow estimation method showed that the feature points moved by 5.2mm (2-4pixels). Therefore, the proposed method can detect body movements during radiography to prevent re-exposure due to body wobble and breathing movements. For healthcare providers, it will lead to reduce not only concerns about patient exposure but also unnecessary radiographic workload.

MREIFlow: Unsupervised dense and time-continuous optical flow estimation from image and event data

Objectives:We focus on exploring an unsupervised learning-based model which can take advantage of a single image and events to estimate dense and time-continuous optical flow. Methods:We propose a multi-scale optical flow recurrent estimation network, called MREIFlow, which mainly consists of a triplet feature encoder, a feature fusion subnetwork, and a flow iterative decoder. The triplet feature encoder is capable of extracting multi-scale features from a single image and events. The feature fusion subnetwork is designed to integrate spatial–temporal information in features and generate pseudo-features. The flow iterative decoder can perform feature correlation and estimate full-resolution optical flow iteratively in a coarse-to-fine way. We train the network in an unsupervised learning way to avoid using labeled data. In order to provide plentiful and reliable supervisory signals, we design a hierarchical unsupervised learning scheme. In the scheme, the multi-level optimization objectives are designed to supervise network training in different aspects. We also introduce a teacher model to provide auxiliary supervision. Besides, we apply a loss selection strategy to mine promising supervisory signals. Results:Experiments on the MVSEC dataset show that the proposed method achieves remarkable performance under indoor and outdoor circumstances. Compared with previous methods, the proposed method can provide dense and time-continuous optical flow estimation. Conclusion:The proposed network can take advantage of a single image and events to produce dense and time-continuous optical flow estimation, and can be trained through unsupervised learning techniques. However, it requires high computation and memory to guarantee the precision of optical flow. To address this problem, we will explore an efficient and lightweight network architecture in the future.

River Surface Velocity Measurement for Rapid Levee Breach Emergency Response Based on DFP-P-LK Algorithm.

In recent years, the increasing frequency of climate change and extreme weather events has significantly elevated the risk of levee breaches, potentially triggering large-scale floods that threaten surrounding environments and public safety. Rapid and accurate measurement of river surface velocities is crucial for developing effective emergency response plans. Video image velocimetry has emerged as a powerful new approach due to its non-invasive nature, ease of operation, and low cost. This paper introduces the Dynamic Feature Point Pyramid Lucas-Kanade (DFP-P-LK) optical flow algorithm, which employs a feature point dynamic update fusion strategy. The algorithm ensures accurate feature point extraction and reliable tracking through feature point fusion detection and dynamic update mechanisms, enhancing the robustness of optical flow estimation. Based on the DFP-P-LK, we propose a river surface velocity measurement model for rapid levee breach emergency response. This model converts acquired optical flow motion to actual flow velocities using an optical flow-velocity conversion model, providing critical data support for levee breach emergency response. Experimental results show that the method achieves an average measurement error below 15% within the velocity range of 0.43 m/s to 2.06 m/s, demonstrating high practical value and reliability.

Lightweight hashing with contrastive self-cross XLNet and combinational similarity matching for content-based video retrieval

ABSTRACT Text-to-video retrieval systems enable simple and natural video searches, but existing methods often employ single queries with noisy annotation and low-quality descriptions. A novel lightweight hashing with Contrastive self-cross XLNet and Combinational similarity matching is proposed to provide high-quality descriptions and clarity in annotation. Issues in existing text and video processing approaches, such as redundancy in keyword generation and extracting frames without considering optical flow estimation of keyframes, are solved using a novel Textual Labelling and Automated key frame segmentation. The 3D Cuboid Net with Lightweight Hashing is proposed, which selects features and converts them into hash codes using Lightweight LSS-Net based hashing, reducing memory utilization. Additionally, a novel Combinational Attentive Text-Video Retrieval is proposed, which eliminates multiple narrow extrema matching error surfaces and increases retrieval validity. The model efficiently solves video retrieval problems with high accuracy, recall, and sensitivity. This novel approach addresses the limitations of existing text-to-video retrieval methods and improves the overall quality of the retrieval process.

Recent Advances in Bio-Inspired Vision Sensor: A Review

Event-based cameras, also known as biologically inspired visual sensors, are capable of capturing real-time scene changes efficiently. Unlike traditional frame-based cameras, event cameras solely report triggered pixel-level brightness changes which are referred to as events. Event-based cameras show many advantages such as high temporal resolution, low latency, and high dynamic range, making them very attractive in robotics and computer vision, especially in challenging scenarios that are too demanding for traditional cameras. In this paper, we provide a comprehensive overview of the emerging field of event-based vision, focusing on the operation principle, sampling mechanisms, and algorithms that take advantage of their superior features. We also delve into the various tasks for which event cameras are utilized, such as object tracking, optical flow estimation, 3D reconstruction, SLAM, image reconstruction, and recognition. Additionally, we highlight the challenges and future opportunities for event cameras, seeking a more efficient way for machines to perceive and interact with the world.

Self-Supervised Deep Blind Video Super-Resolution.

Existing deep learning-based video super-resolution (SR) methods usually depend on the supervised learning approach, where the training data is usually generated by the blurring operation with known or predefined kernels (e.g., Bicubic kernel) followed by a decimation operation. However, this does not hold for real applications as the degradation process is complex and cannot be approximated by these idea cases well. Moreover, obtaining high-resolution (HR) videos and the corresponding low-resolution (LR) ones in real-world scenarios is difficult. To overcome these problems, we propose a self-supervised learning method to solve the blind video SR problem, which simultaneously estimates blur kernels and HR videos from the LR videos. As directly using LR videos as supervision usually leads to trivial solutions, we develop a simple and effective method to generate auxiliary paired data from original LR videos according to the image formation of video SR, so that the networks can be better constrained by the generated paired data for both blur kernel estimation and latent HR video restoration. In addition, we introduce an optical flow estimation module to exploit the information from adjacent frames for HR video restoration. Experiments show that our method performs favorably against state-of-the-art ones on benchmarks and real-world videos.

GLHDR: HDR video reconstruction driven by global to local alignment strategy

Reconstructing High Dynamic Range (HDR) video from alternating exposure Low Dynamic Range (LDR) sequence is an exceptionally challenging task. It not only demands the reliable reconstruction of missing information caused by occlusion or motion without introducing artifacts but also balances the exposure differences between frames to ensure a visually pleasing reconstructed HDR video. Unfortunately, existing methods are typically complex and struggle with unavoidable artifacts and noise, especially when dealing with low-exposed scenes. To tackle this formidable challenge, we propose a two-stage HDR video reconstruction method that employs a global to local alignment strategy. Firstly, we utilize iterative optical flow estimation and hybrid weighting to achieve global alignment, ensuring well-reconstructed in majority of areas. Secondly, the recursive refinement network further addresses locally misaligned areas, reconstructing HDR frames from bottom to top and recursively refining them to yield faithful reconstruction results. Extensive experimental results demonstrate that our method generates the HDR video with fine details and superior visually, surpassing the state-of-the-art method across diverse scenes.

Motion-Aware Video Frame Interpolation

Video frame interpolation methodologies endeavor to create novel frames betwixt extant ones, with the intent of augmenting the video’s frame frequency. However, current methods are prone to image blurring and spurious artifacts in challenging scenarios involving occlusions and discontinuous motion. Moreover, they typically rely on optical flow estimation, which adds complexity to modeling and computational costs. To address these issues, we introduce a Motion-Aware Video Frame Interpolation (MA-VFI) network, which directly estimates intermediate optical flow from consecutive frames by introducing a novel hierarchical pyramid module. It not only extracts global semantic relationships and spatial details from input frames with different receptive fields, enabling the model to capture intricate motion patterns, but also effectively reduces the required computational cost and complexity. Subsequently, a cross-scale motion structure is presented to estimate and refine intermediate flow maps by the extracted features. This approach facilitates the interplay between input frame features and flow maps during the frame interpolation process and markedly heightens the precision of the intervening flow delineations. Finally, a discerningly fashioned loss centered around an intermediate flow is meticulously contrived, serving as a deft rudder to skillfully guide the prognostication of said intermediate flow, thereby substantially refining the precision of the intervening flow mappings. Experiments illustrate that MA-VFI surpasses several representative VFI methods across various datasets, and can enhance efficiency while maintaining commendable efficacy.

Cross-modal learning for optical flow estimation with events

Benefiting from the low latency and high dynamic range, event cameras have recently been adopted for Optical Flow (OF) prediction under harsh environments with high-speed motion or extreme lighting conditions. However, the emitted events only respond to brightness changes, mostly at edges with high-intensity contrast, leading to the ill-posedness of OF prediction in non-edge or low-contrast regions. To address this problem, we propose a network to learn OF by exploring the complements between the frame and the event domains. In particular, we first extract cross-modal features from frames and events via a fusion-based encoder and then apply a self-attention module of cross-dimension interaction to suppress disturbances in noisy events. Furthermore, a long-term memory recurrent module is built for local and global temporal information combination and cross-modal feature enhancement. Finally, a CNN decoder is designed to upscale cross-modal features and provide pyramidal results for multi-scale loss computation. Extensive experiments demonstrate that the proposed method can achieve leading performance compared with the state-of-the-art OF models.

DSTAN: A Deformable Spatial-temporal Attention Network with Bidirectional Sequence Feature Refinement for Speckle Noise Removal in Thyroid Ultrasound Video.

Thyroid ultrasound video provides significant value for thyroid diseases diagnosis, but the ultrasound imaging process is often affected by the speckle noise, resulting in poor quality of the ultrasound video. Numerous video denoising methods have been proposed to remove noise while preserving texture details. However, existing methods still suffer from the following problems: (1) relevant temporal features in the low-contrast ultrasound video cannot be accurately aligned and effectively aggregated by simple optical flow or motion estimation, resulting in the artifacts and motion blur in the video; (2) fixed receptive field in spatial features integration lacks the flexibility of aggregating features in the global region of interest and is susceptible to interference from irrelevant noisy regions. In this work, we propose a deformable spatial-temporal attention denoising network to remove speckle noise in thyroid ultrasound video. The entire network follows the bidirectional feature propagation mechanism to efficiently exploit the spatial-temporal information of the whole video sequence. In this process, two modules are proposed to address the above problems: (1) a deformable temporal attention module (DTAM) is designed after optical flow pre-alignment to further capture and aggregate relevant temporal features according to the learned offsets between frames, so that inter-frame information can be better exploited even with the imprecise flow estimation under the low contrast of ultrasound video; (2) a deformable spatial attention module (DSAM) is proposed to flexibly integrate spatial features in the global region of interest through the learned intra-frame offsets, so that irrelevant noisy information can be ignored and essential information can be precisely exploited. Finally, all these refined features are rectified and merged through residual convolution blocks to recover the clean video frames. Experimental results on our thyroid ultrasound video (US-V) dataset and the DDTI dataset demonstrate that our proposed method exceeds 1.2 1.3dB on PSNR and has clearer texture detail compared to other state-of-the-art methods. In the meantime, the proposed model can also assist thyroid nodule segmentation methods to achieve more accurate segmentation effect, which provides an important basis for thyroid diagnosis. In the future, the proposed model can be improved and extended to other medical image sequence datasets, including CT and MRI slice denoising. The code and datasets are provided at https://github.com/Meta-MJ/DSTAN .

Motion segmentation with event camera: [formula omitted]-patches optical flow estimation and Pairwise Markov Random Fields

This paper proposes the N-patches optical flow estimation-based motion compensation for motion segmentation using event camera. The requirement for prior knowledge of the scenario to cluster events into different motion types is addressed by the N-patches optical flow estimation. The framework of motion segmentation combines motion compensation and Pairwise Markov Random Fields, which not only realize the alignment of events in space but also enhance the spatial correlation between events. The proposed approach aims to maximize the contrast of the images of warped events. It does so by minimizing a specially designed energy function that jointly estimates motion parameters and discrete label items. The problem of selecting the optimal value for N∗ when executing N-patches optical flow estimation is discussed in Section 4.2 of the experiments. Finally, the proposed approach is evaluated in several scenes involving high-speed motion, low-light conditions, occlusions, and multiple moving objects. The experimental results show that the proposed approach achieves better segmentation performance than the event-based motion segmentation by motion compensation.