Optical Flow Method Research Articles

DYS-SLAM: A real-time RGBD SLAM combined with optical flow and semantic information in a dynamic environment1

Traditional Simultaneous Localization and Mapping application in dynamic situations is constrained by static assumptions. However, the majority of well-known dynamic SLAM systems use deep learning to identify dynamic objects, which creates the issue of trade-offs between accuracy and real-time. To tackle this issue, this work suggests a unique dynamic semantics method(DYS-SLAM) for semantic simultaneous localization and mapping that strikes a trade-off between high accuracy and high real-time performance. We propose M-LK, an enhanced Lucas-Kanade(LK) optical flow method. This technique keeps the continuous motion and greyscale consistency assumptions from the original method while switching out the spatial consistency assumption for a motion consistency assumption, reducing sensitivity to image gradients to identify dynamic feature points and regions efficiently. In order to increase segmentation accuracy while maintaining real-time performance, we develop a segmentation refinement scheme that projects 3D point cloud segmentation results into 2D object detection zones. A dense semantic octree graph is built in the interim to expedite the high-level process. Compared to the four equivalent dynamic SLAM approaches, experiments on the publicly available TUM RGB-D dataset demonstrate that the DYS-SLAM method offers competitive localization accuracy and satisfactory real-time performance in both high and low-dynamic scenarios.

Study on video key frame extraction in different scenes based on optical flow

Key frame extraction is an important component of video analysis, and has gradually become a research hotspot in the computer vision community in recent years. Early key frame extraction algorithms were mostly based on fixed time intervals, and their accuracy could not meet practical application requirements. Thanks to the rapid development of machine learning technology, key frame extraction algorithms based on image quality, motion analysis, and deep learning are gradually becoming mature. Although the above methods significantly improve the accuracy of key frame extraction, few work pay attention to the extraction effect in different scenarios. In this article, selecting the key frame extraction algorithm based on optical flow method, we compared the key frame extraction algorithm based on optical flow method in detail for different video scenarios. Extensive experimental results demonstrate the robuteness of key frame extraction algorithm based on optical flow.

Polar optical flow network: extracting overlapped holograms of dynamic particles.

Digital holography with lensless in-line setup has been extensively used in particle field measurements. As particle concentration increases, the holograms of dynamic particles locating at different depths tend to superpose with each other with incoherent overlap, hampering effective measurement of individual particles with incomplete information. Drawing inspiration from suborbicular nature of the in-line holographic fringes, in this study, we propose an optical flow method in polar coordinates to mitigate the overlap issue. The approach employs a radial transformer-enhanced network that leverages both the radial and angular characteristics of the polar hologram. Through ablation tests and experimental results, we have demonstrated the effectiveness and superiority of our proposed method.

Nondestructive Analysis On 4D-Printed Hygroscopic Actuators Through Optical Flow-Based Displacement Measurements

The many advantages of additive manufacturing are particularly noticeable in the fabrication of 4D-printed actuators. Through the selection of specific printing properties and materials, hygroscopic wood polymer composites (WPCs) can be produced and their reaction to humidity can be preprogrammed to achieve the greatest deformations in the shortest time. This responsive behavior makes 4D-printed WPCs suitable for architectural applications, where they can act as passive airflow controllers to improve hygrometric conditions in indoor environments. Image analysis methods have been proven to be reliable to select the best combinations of materials and properties for hygroscopic WPCs but, in some cases, they provide only information on the curvature angles and the instrumentation and software can be expensive. This paper presents an optical flow method for tracking the displacements through a free and open-source software. Starting from a time-lapse video of the sample immersed in water, the analysis returns a matrix composed of the 3D displacement values for each pixel in consecutive frames and the velocity of the displacement, with their visual representation. Such image analysis techniques proved to be suitable to assess the different hygro-responsive behavior under water of 4D-printed WPCs through low-cost equipment without altering their configuration.

Super-Resolution Imaging Enhancement through a 2D Scanning Galvanometer: Algorithm Formulation and Application in Aerial Optoelectronic Systems

As the fields of aviation and aerospace optics continue to evolve, there is an increasing demand for enhanced detection capabilities in equipment. Nonetheless, in applications where both optical and mechanical constraints are stringent, the continuous expansion of optical aperture and focal length is impractical. Given the existing technological landscape, employing super-resolution algorithms to enhance the imaging capability of optical systems is both practical and highly relevant. This study capitalizes on using a 2D scanning galvanometer in optical systems to acquire micro-displacement information. Initially, an imaging model for optical systems equipped with a 2D scanning galvanometer was established, and the displacement vectors for both forward and sweep image motions were defined. On this foundation, we incorporated micro-displacement information that can induce high-frequency aliasing. Subsequently, the motion paths of the galvanometer were planned and modeled. To align image sequences with micro-displacement correlations, the Lucas–Kanade (L-K) optical flow method was employed with multi-layer pyramid iteration. Then, super-resolution reconstruction was performed using kernel regression techniques. Ultimately, we tested the algorithm on an aeronautical optoelectronic pod to evaluate its impact on optical resolution and imaging quality. Compared with the original images, the 16-frame image demonstrated a 39% improvement in optical resolution under laboratory conditions. Moreover, the algorithm exhibited satisfactory performance under both nighttime and daytime conditions, as well as during aerial tests.

A Robust and Integrated Visual Odometry Framework Exploiting the Optical Flow and Feature Point Method.

In this paper, we propose a robust and integrated visual odometry framework exploiting the optical flow and feature point method that achieves faster pose estimate and considerable accuracy and robustness during the odometry process. Our method utilizes optical flow tracking to accelerate the feature point matching process. In the odometry, two visual odometry methods are used: global feature point method and local feature point method. When there is good optical flow tracking and enough key points optical flow tracking matching is successful, the local feature point method utilizes prior information from the optical flow to estimate relative pose transformation information. In cases where there is poor optical flow tracking and only a small number of key points successfully match, the feature point method with a filtering mechanism is used for posing estimation. By coupling and correlating the two aforementioned methods, this visual odometry greatly accelerates the computation time for relative pose estimation. It reduces the computation time of relative pose estimation to 40% of that of the ORB_SLAM3 front-end odometry, while ensuring that it is not too different from the ORB_SLAM3 front-end odometry in terms of accuracy and robustness. The effectiveness of this method was validated and analyzed using the EUROC dataset within the ORB_SLAM3 open-source framework. The experimental results serve as supporting evidence for the efficacy of the proposed approach.

The integration of traditional lacquerware and modern art design in the context of artificial intelligence

Abstract This paper aims to promote the integration of traditional lacquerware and modern art design. AR technology is used for feature calculation of the appearance contour of traditional lacquerware. In this paper, the BRIEF algorithm is selected as the main algorithm for image feature calculation. The Steer BRIEF algorithm is used for key points, and the image feature points are extracted. The classical optical flow method of CMT is applied to track the appearance contour of traditional lacquerware, and the SSD and NCC are combined with filtering the measurement of tracking points. The results show that 90% of the traditional lacquerware can reflect the cultural connotation of the time, 80% of the traditional lacquerware can reflect the characteristics of the time, 80% of the traditional lacquerware has excellent craftsmanship, and the added value reflected by modern art design is much lower than that of the traditional lacquerware.

Improving Precipitation Nowcasting for High-Intensity Events Using Deep Generative Models with Balanced Loss and Temperature Data: A Case Study in the Netherlands

Abstract Precipitation nowcasting is essential for weather-dependent decision-making, but it remains a challenging problem despite active research. The combination of radar data and deep learning methods has opened a new avenue for research. Radar data are well suited for precipitation nowcasting due to the high space–time resolution of the precipitation field. On the other hand, deep learning methods allow the exploitation of possible nonlinearities in the precipitation process. Thus far, deep learning approaches have demonstrated equal or better performance than optical flow methods for low-intensity precipitation, but nowcasting high-intensity events remains a challenge. In this study, we have built a deep generative model with various extensions to improve nowcasting of heavy precipitation intensities. Specifically, we consider different loss functions and how the incorporation of temperature data as an additional feature affects the model’s performance. Using radar data from KNMI and 5–90-min lead times, we demonstrate that the deep generative model with the proposed loss function and temperature feature outperforms other state-of-the-art models and benchmarks. Our model, with both loss function and feature extensions, is skillful at nowcasting precipitation for high rainfall intensities, up to 60-min lead time.

Unsupervised Optical Flow Estimation for Differently Exposed Images in LDR Domain

Differently exposed low dynamic range (LDR) images are often captured sequentially using a smart phone or a digital camera with movements. Optical flow thus plays an important role in ghost removal for high dynamic range (HDR) imaging. The optical flow estimation is based on the theory of photometric consistency, which assumes that the corresponding pixels between two images have the same intensity. However, the assumption is no longer valid for the differently exposed LDR images since a pixel’s intensity changes significantly inter images. To address the problem, an unsupervised optical flow estimation framework, is presented in this study. Intensity mapping functions (IMFs) are first adopted to alleviate the intensity changes between the LDR images. Then a novel IMF-based unsupervised learning objective is proposed to circumvent the need for ground truth optical flows when training the deep network. Experimental results and ablation studies on publicly available datasets show that our framework outperforms the state-of-the-art unsupervised optical flow methods, demonstrating the effectiveness of the IMF and the learning objective. Our code is available at https://github.com/liuziyang123/LDRFlow.

Optical Flow Analysis of Paralaryngeal Muscle Movement.

The paralaryngeal muscles are thought to be hyperfunctional with phonation in patients with primary muscle tension dysphonia (pMTD). However, objective, quantitative tools to assess paralaryngeal movement patterns lack. The objectives of this study were to (1) validate the use of optical flow to characterize paralaryngeal movement patterns with phonation, (2) characterize phonatory optical flow velocities and variability of the paralaryngeal muscles before and after a vocal load challenge, and (3) compare phonatory optical flow measures to standard laryngoscopic, acoustic, and self-perceptual assessments. Phonatory movement velocities and variability of the paralaryngeal muscles at vocal onsets and offsets were quantified from ultrasound videos and optical flow methods across 42 subjects with and without a diagnosis of pMTD, before and after a vocal load challenge. Severity of laryngoscopic mediolateral supraglottic compression, acoustic perturbation, and ratings of vocal effort and discomfort were also obtained at both time points. There were no significant differences in optical flow measures of the paralaryngeal muscles with phonation between patients with pMTD and controls. Patients with pMTD had significantly more supraglottic compression, higher acoustic perturbations, and higher vocal effort and vocal tract discomfort ratings. Vocal load had a significant effect on vocal effort and discomfort but not on supraglottic compression, acoustics, or optical flow measures of the paralaryngeal muscles. Optical flow methods can be used to study paralaryngeal muscle movement velocity and variability patterns during vocal productions, although the role of the paralaryngeal in pMTD diagnostics (e.g., vocal hyperfunction) remains suspect. 2 Laryngoscope, 134:1792-1801, 2024.

MLAM: Multi-Layer Attention Module for Radar Extrapolation Based on Spatiotemporal Sequence Neural Network.

Precipitation nowcasting is mainly achieved by the radar echo extrapolation method. Due to the timing characteristics of radar echo extrapolation, convolutional recurrent neural networks (ConvRNNs) have been used to solve the task. Most ConvRNNs have been proven to perform far better than traditional optical flow methods, but they still have fatal problems. These models lack differentiation in the prediction of echoes of different intensities, which leads to the omission of responses from regions with high intensities. Moreover, because it is difficult for these models to capture long-term feature dependencies among multiple echo maps, the extrapolation effect declines sharply over time. This paper proposes an embedded multi-layer attention module (MLAM) to address the shortcomings of ConvRNNs. Specifically, an MLAM mainly enhances attention to critical regions in echo images and the processing of long-term spatiotemporal features through the interaction between input and memory features in the current moment. Comprehensive experiments were conducted on the radar dataset HKO-7 provided by the Hong Kong Observatory and the radar dataset HMB provided by the Hunan Meteorological Bureau. Experiments show that ConvRNNs embedded with MLAMs achieve more advanced results than standard ConvRNNs.

Friction measurement of aircraft wing based on optimized FlowNet2.0

Few studies have applied the deep optical flows model to global friction measurements of aircraft wing. This study used an optimized FlowNet2.0 model to measure friction of wing based on fluorescent oil film, which achieved the first integration of deep learning and friction measurement. Two input images of the traditional FlowNet2.0 model were extended to multiple images so that more flow features and details could be with them. It is the specific part of optimization that will also further improve the measurement accuracy of FlowNet2.0. Simulation experimental results show that the optimized FlowNet2.0 model reduces the Mean Absolute Percentage Error (MAPE) error by 8.51% and increases Root Mean Square Error (RMSE) by only 0.0138 when compared to the hybrid optical flow method, which indicate that the optimized FlowNet2.0 model has great potential for friction measurement. Measurements in continuous transonic wind tunnel tests demonstrate that FlowNet2.0 can calculate clearer and more accurate flow velocity than hybrid optical flow, and the solved friction magnitude distribution is consistent with the actual flow, which will be of great practical application in wind tunnel engineering.

Brain-like position measurement method based on improved optical flow algorithm

In this paper, a brain-like navigation scheme based on fuzzy kernel C-means (FKCM) clustering assisted pyramid Lucas Kanade (LK) optical flow algorithm is developed to measure the position of vehicle. The Speed Cell and Place Cell in animals’ brain are introduced to construct the brain-like navigation mechanism which involves the optical flow method and image template matching to imitate the cells above-mentioned separately. To eliminate the singular values during optical flow calculation, the output of pyramid LK algorithm is clustered by FKCM algorithm firstly. Then, the velocity is calculated and integrated to get the position of the vehicle, and the brain-like navigation scheme is introduced to correct the position measurement errors by eliminating the accumulated errors resulting from velocity integration. The prominent advantages of the presented method are: (i) a pure visual brain-like position measurement method based on the concept of speed cells and place cells is proposed, making visual navigation more accurate and intelligent; (ii) the FKCM algorithm is used to eliminate the singular value of the pyramid LK algorithm, which improves the calculated velocity accuracy. Also, experimental comparison with classical pyramid LK algorithm is given to illustrate the superiority of the proposed method in position measurement.

On the cardiorespiratory coordination assessed by the photoplethysmography imaging technique

Cardiorespiratory coordination (CRC) probes the interaction between cardiac and respiratory oscillators in which cardiac and respiratory activity are synchronized, with individual heartbeats occurring at approximately the same temporal positions during several breathing cycles. An increase of CRC has previously been related to pathological stressful states. We studied CRC employing coordigrams computed from non-contact photoplethysmography imaging (PPGI) and respiratory data using the optical flow method. In a blocked study design, we applied the cold pressure test (CPT), water at ambient temperature (AWT), and intermittent resting conditions. In controls (no intervention), CRC remained on initial low levels throughout measurements. In the experimental group (AWT and CPT intervention), CRC decreased during AWT and CPT. Following both interventions, CRC increased significantly, with a rebound effect following AWT. In controls, HR increased steadily over time. CPT evoked a significant HR increase which correlated with subjective stress/pain ratings. The CRC increase following AWT correlated significantly with subjective pain (r = .79) and stress (r = .63) ratings. Furthermore, we observed a significant correlation (r = − .80) between mean RMSSD and mean duration of CRC, which further supports an association between autonomic state and CRC level. CRC analysis obtained from cutaneous tissue perfusion data therefore appears to be a sensitive and useful method for the study of CRC and ANS activity. Future studies need to investigate the physiological principles and clinical significance of these findings.

SOFNet: Optical-flow based large-scale slice augmentation of brain MRI

Brain magnetic resonance imaging (MRI) data from multiple centers often exhibit variations in imaging conditions, such as the types of nuclear magnetic resonance instruments used and the presence of random noise. Additionally, discrepancies in the gap between MRI slices further complicate the usability of the data for advanced artificial intelligence (AI) analysis. Deep learning-based methods have emerged as practical solutions to address the challenge. However, existing research has largely overlooked the augmentation of brain MRI data, particularly when confronted with significant slice gaps, such as around 6 mm observed in our clinical brain MRI slices. In response to this research gap, we aim to develop novel approaches for augmenting brain MRI data, focusing on more significant slice gaps. To achieve this, we propose SOFNet, which utilizes the optical flow-based and encoder–decoder backbone. The primary objective of our model is to interpolate MRI slices while preserving feature consistency. Leveraging the optical flow method, which has exhibited exceptional performance compared to other super-resolution algorithms, our proposed approach has been evaluated on three distinct brain MRI datasets, explicitly addressing the gap between 4.2 mm and 6.0 mm. The experimental results highlight the significant enhancement in super-resolution quality achieved by SOFNet in generating adapted brain MRI data, surpassing other single-image super-resolution (SISR) methods in feature completion. To ensure the credibility of the interpolated brain MRI slices, we conducted experiments on three axes of MRI based on metrics such as peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM). These experiments demonstrate the effectiveness of our approach in transforming low-resolution MRI data into clear and reliable brain MRIs, thereby enabling improved analysis using AI techniques.

Rain Scanner Radar and Optical Flow Combination For Early Identification of The Incoming Bow Echo Heavy Precipitation

In this study, heavy precipitation in the form of bow echo is observed, identified, and tracked using a rain scanner and optical flow method. Two case studies of bow echoes with a range of properties from 15 to 33 km were observed in the afternoon on 1st February and 4th February 2021. The rain scanner developed based on X-band radar could obtain the first detection of the bow echo shape. The difference between these two bow echoes events is from their initiation. The first bow echo cases are initiated from weakly organized cells. Meanwhile, the second case of bow echo is initiated from a squall line event. Bow echo is initially observed in the western part of Bandung city. Approximately twenty minutes after the first detection, the bow echo passed the center of Bandung city. Image processing using edge detection could track the bow echo from the radar data. Applied Lucas-Kanade optical flow was then used to retrieve the bow echo movement. Vector flow from optical flow showed the dominant movement toward the east direction in both case studies. A simple method to predict the incoming time of the bow echo using the estimated speed from vector flow and the image translation is also presented.

Fuel/air mixing characteristics of a Micromix burner for hydrogen-rich gas turbine

As a promising hydrogen-rich gas turbine combustion technology, Micro-mixing combustion may produce a new flame stabilization mechanism due to the novel structure. To investigate the effects of nozzle structural parameters and inlet aerodynamic parameters on mixing characteristics, this paper firstly used the background oriented schlieren technique combined with the self-development program to obtain the mixing quality, and the program was based on the optical flow method and Gladstone-Dale equation. The fuel-air mixture process was predicted by three mass diffusion models, and the Kinetic Theory model matched the experimental results best compared with the Constant Dilute Approximation model and Dilute Approximation model. The results showed that the main mixing zone moved closer to the outer wall when the premixed channel diameter increased. And the vorticity decreased with the increased premixed length, leading to a decline in mixing efficiency. For inlet aerodynamic parameters, changes in equivalence ratios only affected the shear-layer vorticity. As the combustible component of syngas was in the range of 60%–40% (by vol.), the outer wall of the premixed channel could promote fuel-air mixture. Moreover, when the equivalence ratio ranged in 0.335–0.685, the heating value of fuel varied as 4.686–7.029 and 11.715 MJ/Nm3, fuel and air were premixed uniformly near the nozzle outlet.

Dynamic 3D phase-shifting profilometry based ona corner optical flow algorithm.

Real-time 3D reconstruction has been applied in many fields, calling for many ongoing efforts to improve the speed and accuracy of the used algorithms. Phase shifting profilometry based on the Lucas-Kanade optical flow method is a fast and highly precise method to construct and display the three-dimensional shape of objects. However, in this method, a dense optical flow calculation is required for the modulation image corresponding to the acquired deformed fringe pattern, which consumes a lot of time and affects the real-time performance of 3D reconstruction and display. Therefore, this paper proposes a dynamic 3D phase shifting profilometry based on a corner optical flow algorithm to mitigate this issue. Therein, the Harris corner algorithm is utilized to locate the feature points of the measured object, so that the optical flow needs to calculate for only the feature points which, greatly reduces the amount of calculation time. Both our experiments and simulations show that our method improves the efficiency of pixel matching by four times and 3D reconstruction by two times.

Individuality of a group: detailed walking ability analysis of broiler flocks using optical flow approach

Impaired walking ability is one of the most important factors affecting broiler welfare. Routine monitoring of walking ability provides insights in the welfare status of a flock and assists farmers in taking remedial measures at an early stage. Several computer vision techniques have been developed for automated assessment of walking ability, providing an objective and biosecure alternative to the currently more subjective and time-consuming manual assessment of walking ability. However, these techniques mainly focus on assessment of averages at flock level using pixel movement. Therefore, the aim of this study was to investigate the potential of optical flow algorithms to identify flock activity, distribution and walking ability in a commercial setting on levels close to individual monitoring. We used a combination of chicken segmentation and optical flow methods, where chicken contours were first detected and were then used to identify activity, spatial distribution, and gait score distribution (i.e. walking ability) of the flock via optical flow. This is a step towards focusing more on individual chickens in an image and its pixel representation. In addition, we predicted the gait score distribution of the flock, which is a more detailed assessment of broiler walking ability compared to average gait score of the flock, as slight changes in walking ability are more likely to be detected when using the distribution compared to the average score. We found a strong correlation between predicted and observed gait scores (R2 = 0.97), with separate gait scores all having R2 > 0.85. Thus, the algorithm used in this study is a first step to measure broiler walking ability automatically in a commercial setting on a levels close to individual monitoring. These validation results of the developed automatic monitoring of flock activity, distribution and gait score are promising, but further validation is required (e.g. for chickens at a younger age, with very low and very high gait scores).

A Semantic Information-Based Optimized vSLAM in Indoor Dynamic Environments

In unknown environments, mobile robots can use visual-based Simultaneous Localization and Mapping (vSLAM) to complete positioning tasks while building sparse feature maps and dense maps. However, the traditional vSLAM works in the hypothetical environment of static scenes and rarely considers the dynamic objects existing in the actual scenes. In addition, it is difficult for the robot to perform high-level semantic tasks due to its inability to obtain semantic information from sparse feature maps and dense maps. In order to improve the ability of environment perception and accuracy of mapping for mobile robots in dynamic indoor environments, we propose a semantic information-based optimized vSLAM algorithm. The optimized vSLAM algorithm adds the modules of dynamic region detection and semantic segmentation to ORB-SLAM2. First, a dynamic region detection module is added to the vision odometry. The dynamic region of the image is detected by combining single response matrix and dense optical flow method to improve the accuracy of pose estimation in dynamic environment. Secondly, the semantic segmentation of images is implemented based on BiSeNet V2 network. For the over-segmentation problem in semantic segmentation, a region growth algorithm combining depth information is proposed to optimize the 3D segmentation. In the process of map building, semantic information and dynamic regions are used to remove dynamic objects and build an indoor map containing semantic information. The system not only can effectively remove the effect of dynamic objects on the pose estimation, but also use the semantic information of images to build indoor maps containing semantic information. The proposed algorithm is evaluated and analyzed in TUM RGB-D dataset and real dynamic scenes. The results show that the accuracy of our algorithm outperforms that of ORB-SLAM2 and DS-SLAM in dynamic scenarios.