Use Of Optical Flow Research Articles

Flow-Edge-Net: Video Saliency Detection Based on Optical Flow and Edge-Weighted Balance Loss

Optical flow networks have been widely utilized for video saliency detection (VSD) due to their effective performance in capturing the motion of objects. However, the use of optical flow blurs the edges of salient objects and leads to the problems of poorly defined object boundaries. To address this issue, we propose an optical flow-based edge-weighted loss function, to train a network called Flow-Edge-Net, which can balance the weights of the foreground and background information at the edges of video frames. It has achieved superior performance in detecting salient boundaries. Specifically, we propose two complementary encoding and decoding networks based on the concept of decoupling. That is, the optical flow network focuses on moving objects, while the edge network, based on the encoder-decoder structure, focuses on edge information. As the two networks output features of the same dimension and are from the same input, our proposed self-designed adaptive weighted feature fusion module can compare and integrate the edge information and location information from the two networks through adaptive weighting. The proposed method has been evaluated on five widely used databases. Experiment results demonstrate the superior performance of the proposed Flow-Edge-Net in locating salient objects, with accurate and refined edges. The proposed method achieves superior performance over the state-of-the-art methods in detecting salient objects in videos.

Joint facial action unit recognition and self-supervised optical flow estimation

Facial action unit (AU) recognition and optical flow estimation are two highly correlated tasks, since optical flow can provide motion information of facial muscles to facilitate AU recognition. However, most existing AU recognition methods handle the two tasks independently by offline extracting optical flow as auxiliary information or directly ignoring the use of optical flow. In this paper, we propose a novel end-to-end joint framework of AU recognition and optical flow estimation, in which the two tasks contribute to each other. Moreover, due to the lack of optical flow annotations in AU datasets, we propose to estimate optical flow in a self-supervised manner. To regularize the self-supervised estimation of optical flow, we propose an identical mapping constraint for the optical flow guided image warping process, in which the estimated optical flow between two same images is required to not change the image during warping. Experiments demonstrate that our framework (i) outperforms most of the state-of-the-art AU recognition methods on the challenging BP4D and GFT benchmarks, and (ii) also achieves competitive self-supervised optical flow estimation performance.

Optical Flow Analysis of Paralaryngeal Muscle Movement.

The paralaryngeal muscles are thought to be hyperfunctional with phonation in patients with primary muscle tension dysphonia (pMTD). However, objective, quantitative tools to assess paralaryngeal movement patterns lack. The objectives of this study were to (1) validate the use of optical flow to characterize paralaryngeal movement patterns with phonation, (2) characterize phonatory optical flow velocities and variability of the paralaryngeal muscles before and after a vocal load challenge, and (3) compare phonatory optical flow measures to standard laryngoscopic, acoustic, and self-perceptual assessments. Phonatory movement velocities and variability of the paralaryngeal muscles at vocal onsets and offsets were quantified from ultrasound videos and optical flow methods across 42 subjects with and without a diagnosis of pMTD, before and after a vocal load challenge. Severity of laryngoscopic mediolateral supraglottic compression, acoustic perturbation, and ratings of vocal effort and discomfort were also obtained at both time points. There were no significant differences in optical flow measures of the paralaryngeal muscles with phonation between patients with pMTD and controls. Patients with pMTD had significantly more supraglottic compression, higher acoustic perturbations, and higher vocal effort and vocal tract discomfort ratings. Vocal load had a significant effect on vocal effort and discomfort but not on supraglottic compression, acoustics, or optical flow measures of the paralaryngeal muscles. Optical flow methods can be used to study paralaryngeal muscle movement velocity and variability patterns during vocal productions, although the role of the paralaryngeal in pMTD diagnostics (e.g., vocal hyperfunction) remains suspect. 2 Laryngoscope, 134:1792-1801, 2024.

I3D Light - A Simple Motion Information Stream for I3D

Vision-based Human Activity Recognition (HAR) aims to recognize human activities based on the analysis of video data, and has extensive applications in modern industry and human life. Inflated 3D (I3D) is a deep learning architecture that is commonly used for action recognition by using two-stream video data: RGB stream and optical flow stream. I3D achieved great success on a variety of action recognition benchmarks. However, the use of optical flow incurs high computational cost, making the approach unsuitable for real-time applications. We propose an alternative simple motion information extractor to replace the optical flow branch and reduce the computational cost. It is a modified I3D that uses 128 frames of 112x112 images as input. The low spatial resolution and long temporal range of the proposed I3D RGB stream can reduce the spatial information and enhance the motion information. Experiments show that this simple motion stream can increase the accuracy of the original I3D spatial stream by 4.09% on the Kinetics 400 dataset.

Multi-Scale correlation module for video-based facial expression recognition in the wild

The detection of facial muscle movements (e.g., mouth opening) is crucial for facial expression recognition (FER). However, extracting these facial motion features is challenging for a deep-learning recognition system for the following reasons: (1) without explicit labels of motion for training, there is no guarantee that convolutional neural networks (CNNs) can extract motions effectively; (2) compared to human action recognition (e.g., the object moving from left to right), some facial motions (e.g., raising eyebrows) are more subtle and thus harder to extract; and (3) the use of optical flow to extract motion features is time-consuming when using a commonly-used camera. In this work, we propose a Multi-Scale Correlation Module (MSCM) together with an adaptive fusion. Firstly, large as well as small facial motions are extracted by MSCM and encoded by CNNs. Then, an adaptive fusion module is used to aggregate motion features. With these modules, our recognition network is able to model both subtle and large motion features for video-based FER with only the RGB image frames as input. Experiments on two datasets, AFEW and DFEW, show that the network achieves state-of-art performances on the benchmarks.

Fast and High-Quality Monocular Depth Estimation with Optical Flow for Autonomous Drones

Recent years, autonomous drones have attracted attention in many fields due to their convenience. Autonomous drones require precise depth information so as to avoid collision to fly fast and both of RGB image and LiDAR point cloud are often employed in applications based on Convolutional Neural Networks (CNNs) to estimate the distance to obstacles. Such applications are implemented onboard embedded systems. In order to precisely estimate the depth, such CNN models are in general so complex to extract many features that the computational complexity increases, requiring long inference time. In order to solve the issue, we employ optical flow to aid in-depth estimation. In addition, we propose a new attention structure that makes maximum use of optical flow without complicating the network. Furthermore, we achieve improved performance without modifying the depth estimator by adding a perceptual discriminator in training. The proposed model is evaluated through accuracy, error, and inference time on the KITTI dataset. In the experiments, we have demonstrated the proposed method achieves better performance by up to 34% accuracy, 55% error reduction and 66% faster inference time on Jetson nano compared to previous methods. The proposed method is also evaluated through a collision avoidance in simulated drone flight and achieves the lowest collision rate of all estimation methods. These experimental results show the potential of proposed method to be used in real-world autonomous drone flight applications.

Key frames assisted hybrid encoding for high-quality compressive video sensing.

Snapshot compressive imaging (SCI) encodes high-speed scene video into a snapshot measurement and then computationally makes reconstructions, allowing for efficient high-dimensional data acquisition. Numerous algorithms, ranging from regularization-based optimization and deep learning, are being investigated to improve reconstruction quality, but they are still limited by the ill-posed and information-deficient nature of the standard SCI paradigm. To overcome these drawbacks, we propose a new key frames assisted hybrid encoding paradigm for compressive video sensing, termed KH-CVS, that alternatively captures short-exposure key frames without coding and long-exposure encoded compressive frames to jointly reconstruct high-quality video. With the use of optical flow and spatial warping, a deep convolutional neural network framework is constructed to integrate the benefits of these two types of frames. Extensive experiments on both simulations and real data from the prototype we developed verify the superiority of the proposed method.

Implementation of VBM3D and Some Variants

VBM3D is an extension to video of the well-known image denoising algorithm BM3D, which takes advantage of the sparse representation of stacks of similar patches in a transform domain. The extension is rather straightforward: the similar 2D patches are taken from a spatio-temporal neighborhood which includes neighboring frames. In spite of its simplicity, the algorithm offers a good trade-off between denoising performance and computational complexity. In this work we revisit this method, providing an open-source C++ implementation reproducing the results. A detailed description is given and the choice of parameters is thoroughly discussed. Furthermore, we study and compare several extensions of the original algorithm: (1) a multiscale implementation, (2) the use of 3D patches, (3) the use of optical flow to guide the patch search. These extensions yield results which are competitive with the most recent state of the art.

Endoscopic Image-Based Skill Assessment in Robot-Assisted Minimally Invasive Surgery.

Objective skill assessment-based personal performance feedback is a vital part of surgical training. Either kinematic—acquired through surgical robotic systems, mounted sensors on tooltips or wearable sensors—or visual input data can be employed to perform objective algorithm-driven skill assessment. Kinematic data have been successfully linked with the expertise of surgeons performing Robot-Assisted Minimally Invasive Surgery (RAMIS) procedures, but for traditional, manual Minimally Invasive Surgery (MIS), they are not readily available as a method. 3D visual features-based evaluation methods tend to outperform 2D methods, but their utility is limited and not suited to MIS training, therefore our proposed solution relies on 2D features. The application of additional sensors potentially enhances the performance of either approach. This paper introduces a general 2D image-based solution that enables the creation and application of surgical skill assessment in any training environment. The 2D features were processed using the feature extraction techniques of a previously published benchmark to assess the attainable accuracy. We relied on the JHU–ISI Gesture and Skill Assessment Working Set dataset—co-developed by the Johns Hopkins University and Intuitive Surgical Inc. Using this well-established set gives us the opportunity to comparatively evaluate different feature extraction techniques. The algorithm reached up to 95.74% accuracy in individual trials. The highest mean accuracy—averaged over five cross-validation trials—for the surgical subtask of Knot-Tying was 83.54%, for Needle-Passing 84.23% and for Suturing 81.58%. The proposed method measured well against the state of the art in 2D visual-based skill assessment, with more than 80% accuracy for all three surgical subtasks available in JIGSAWS (Knot-Tying, Suturing and Needle-Passing). By introducing new visual features—such as image-based orientation and image-based collision detection—or, from the evaluation side, utilising other Support Vector Machine kernel methods, tuning the hyperparameters or using other classification methods (e.g., the boosted trees algorithm) instead, classification accuracy can be further improved. We showed the potential use of optical flow as an input for RAMIS skill assessment, highlighting the maximum accuracy achievable with these data by evaluating it with an established skill assessment benchmark, by evaluating its methods independently. The highest performing method, the Residual Neural Network, reached means of 81.89%, 84.23% and 83.54% accuracy for the skills of Suturing, Needle-Passing and Knot-Tying, respectively.

Optimising the Workflow for Fish Detection in DIDSON (Dual-Frequency IDentification SONar) Data with the Use of Optical Flow and a Genetic Algorithm

DIDSON acoustic cameras provide a way to collect temporally dense, high-resolution imaging data, similar to videos. Detection of fish targets on those videos takes place in a manual or semi-automated manner, typically assisted by specialised software. Exploiting the visual nature of the recordings, tools and techniques from the field of computer vision can be applied in order to facilitate the relatively involved workflows. Furthermore, machine learning techniques can be used to minimise user intervention and optimise for specific detection and tracking scenarios. This study explored the feasibility of combining optical flow with a genetic algorithm, with the aim of automating motion detection and optimising target-to-background segmentation (masking) under custom criteria, expressed in terms of the result. A 1000-frame video sequence sample with sparse, smoothly moving targets, reconstructed from a 125 s DIDSON recording, was analysed under two distinct scenarios, and an elementary detection method was used to assess and compare the resulting foreground (target) masks. The results indicate a high sensitivity to motion, as well as to the visual characteristics of targets, with the resulting foreground masks generally capturing fish targets on the majority of frames, potentially with small gaps of undetected targets, lasting for no more than a few frames. Despite the high computational overhead, implementation refinements could increase computational feasibility, while an extension of the algorithms, in order to include the steps of target detection and tracking, could further improve automation and potentially provide an efficient tool for the automated preliminary assessment of voluminous DIDSON data recordings.

Adapting a method for tracking the movement of the camera in the visualization of augmented reality

This paper presents an adapted method for tracking camera motion in the visualization of augmented reality. A tracking algorithm based on significant points and descriptors without the use of optical flow is described. The tracking module is written in C++. Experiments have been carried out of the precision of the system using the camera motion-tracking module created.

Animating Cloud Images With Flow Style Transfer

We propose a method for animating static images using a generative adversarial network (GAN). Given a source image depicting a cloud image and a driving video sequence depicting a moving cloud image, our framework generates a video in which the source image is animated according to the driving sequence. By inputting the source image and optical flow of the driving video into the generator, a video is generated that is conditioned by the optical flow. The optical flow enables the application of the captured motion of clouds in the source image. Further, we experimentally show that the proposed method is more effective than the existing methods for animating a keypoint-less video (in which the keypoints cannot be explicitly determined) such as a moving cloud image. Furthermore, we show an improvement in the quality of the generated video due to the use of optical flow in the video reconstruction.

The Dominant Role of Visual Motion Cues in Bumblebee Flight Control Revealed Through Virtual Reality.

Flying bees make extensive use of optic flow: the apparent motion in the visual scene generated by their own movement. Much of what is known about bees' visually-guided flight comes from experiments employing real physical objects, which constrains the types of cues that can be presented. Here we implement a virtual reality system allowing us to create the visual illusion of objects in 3D space. We trained bumblebees, Bombus ignitus, to feed from a static target displayed on the floor of a flight arena, and then observed their responses to various interposing virtual objects. When a virtual floor was presented above the physical floor, bees were reluctant to descend through it, indicating that they perceived the virtual floor as a real surface. To reach a target at ground level, they flew through a hole in a virtual surface above the ground, and around an elevated virtual platform, despite receiving no reward for avoiding the virtual obstacles. These behaviors persisted even when the target was made (unrealistically) visible through the obstructing object. Finally, we challenged the bees with physically impossible ambiguous stimuli, which give conflicting motion and occlusion cues. In such cases, they behaved in accordance with the motion information, seemingly ignoring occlusion.

Differences in spatial resolution and contrast sensitivity of flight control in the honeybees Apis cerana and Apis mellifera

Visually guided behaviour is constrained by the capacity of the visual system to resolve detail. This, in turn, is limited by the spatial resolution and contrast sensitivity of the underlying visual system. Because these properties are interdependent and vary non-uniformly, it is only possible to fully understand the limits of a specific visually guided behaviour when they are investigated in combination. To understand the visual limits of flight control in bees, which rely heavily on vision to control flight, and to explore whether they vary between species, we tested how changes in spatial resolution and contrast sensitivity affect the speed and position control of the Asian and European honeybees (Apis cerana and Apis mellifera). Despite the apparent similarity of these species, we found some interesting and surprising differences between their visual limits. While the effect of spatial frequency and contrast on position control is similar between the species, ground speed is differently affected by these variables. A comparison with published data from the bumblebee Bombus terrestris revealed further differences. The visual resolution that limits the detection and use of optic flow for flight control in both species of honeybee is lower than the previously anatomically determined resolution and differs from object detection limits of A. mellifera, providing evidence that the limits of spatial resolution and contrast sensitivity are highly tuned to the particular behavioural task of a species.

Investigating the use of motion-based features from optical flow for gait recognition

Although numerous research studies have confirmed the potentials of using gait for people identification in surveillance and forensic scenarios, only a few studies have investigated the contribution of motion-based features on the recognition process. In this research paper, we explore the use of optical flow estimated from consecutive frames to construct a discriminative biometric signature for gait recognition. A set of experiments are carried out using the CASIA-B dataset to assess the discriminatory potency of motion-based features for gait identification subjected to different covariate factors including clothing and carrying conditions. Further experiments are conducted to explore the effects of the dataset size, the number of frames and viewpoint on the classification process. Based on a dataset containing 1000 video sequences for 100 individuals, higher recognition rates are achieved using the Knn and neural network classifiers without incorporating static and anthropometric measurements. This confirms that gait identification using motion-based features is perceivable with acceptable recognition rates even under different covariate factors. As such, this is a major milestone in translating gait research to surveillance and forensic scenarios.

Automatic Intruder Combat System: A way to Smart Border Surveillance

Security and safeguard of international borders have always been a dominant issue for every nation. A large part of a nation’s budget is provided to its defense system. Besides wars, illegal intrusion in terms of terrorism is a critical matter that causes severe harm to nation’s property. In India’s perspective, border patrolling by Border Security Forces (BSF) has already been practiced from a long time for surveillance. The patrolling parties are equipped with high-end surveillance equipments but yet an alternative to the ply of huge manpower and that too in harsh environmental conditions hasn’t been in existence. An automatic mechanism for smart surveillance and combat is proposed in this paper as a solution to the above-discussed problems. Smart surveillance requires automatic intrusion detection in the surveillance video, which is achieved by using optical flow information as motion features for intruder/human in the scene. The use of optical flow in the proposed smart surveillance makes it robust and more accurate. Use of a simple horizontal feature for fence detection makes system simple and faster to work in real-time. System is also designed to respond against the activities of intruders, of which auto combat is one kind of response.

Low-altitude horizon-based aircraft attitude estimation using UV-filtered panoramic images and optic flow

In this paper, a novel method for real-time horizon-based attitude estimation using panoramic images is presented with results from real flight tests showing accurate attitude estimation for a rotary-wing aircraft in cluttered environments. The vision system is biologically inspired from the function of the Ocelli organ present in some insects and the fact that ultraviolet images provide better sky/ground contrast enhancement. A new method for panoramic sky/ground thresholding is proposed, consisting of a sky/ground masking and a sun-tracking system that works effectively even when the horizon line is difficult to detect by normal thresholding methods due to flares and other effects from the presence of the sun in the image. The use of optic flow to determine body rates is investigated using the panoramic image and the image interpolation algorithm. A Kalman filter is used to fuse the unfiltered measurements from inertial sensors and the vision system.

Echo-acoustic flow affects flight in bats.

Flying animals need to react fast to rapid changes in their environment. Visually guided animals use optic flow, generated by their movement through structured environments. Nocturnal bats cannot make use of optic flow, but rely mostly on echolocation. Here, we show that bats exploit echo-acoustic flow to negotiate flight through narrow passages. Specifically, bats' flight between lateral structures is significantly affected by the echo-acoustic salience of those structures, independent of their physical distance. This is true even though echolocation, unlike vision, provides explicit distance cues. Moreover, the bats reduced the echolocation sound levels in stronger flow, probably to compensate for the increased summary target strength of the lateral reflectors. However, bats did not reduce flight velocity under stronger echo-acoustic flow. Our results demonstrate that sensory flow is a ubiquitous principle for flight guidance, independent of the fundamentally different peripheral representation of flow across the senses of vision and echolocation.

발달성협응장애아동의 자세조절에 있어 지지면의 견고성과 시각정보의 영향

Purpose: Previous study (Chung & Stoffregen, 2011) has shown that the coupling of DCD children with optic tic flow differed from that of typically developing (TD) children. The study further investigated the differentia l sensitivity of TD and DCD children to imposed optic flow using the moving room paradigm. Methods: A total of twenty children, ten with DCD ( 5th percentile in M-ABC test) and ten matched controls, in the age of 10-11 years (mean age = 9.5 ± 0.6 yr) participated in this study. We evaluated the postural sway of TD and DCD childrenwhile providing and reducing support rigidity with the vision or without during the cyclic oscillation of visual stimulus. The room was oscillated with magnitude 2 cm at three different frequencies (0.1 Hz, 0.2 Hz, 0.3 Hz). Variation in the frequency of room oscillation was crossed with variation in the support surface (foam vs. no foam), for both TD and DCD children. We measured the motion of the body in the anterior-posterior axis of center of pressure(COPap), that is, along the line of sight. Results: The postural sway of DCD children was less stable than TD children without room motion. In addition, DCD children displayed greater body sway (high mean gain, high max spectrum ) during periodic oscillation of room motion. The group differences were large during supported surface. Conclusion: We concluded that DCD children may have problem in integrating multi-sensory information and use of optic flow differently from TD in controlling their posture.

Virtual Reality and Musculoskeletal Pain: Manipulating Sensory Cues to Improve Motor Performance During Walking

Musculoskeletal pain (MSP) is the most expensive nonmalignant health problem and the most common reason for activity limitation. Treatment approaches to improve movement without aggravating pain are urgently needed. Virtual reality (VR) can decrease acute pain, as well as influence movement speed. It is not clear whether VR can improve movement speed in individuals with MSP without aggravating pain. This study investigated the extent to which different audio and optic flow cues in a VR environment influenced walking speed in people with and without MSP. A total of 36 subjects participated, 19 with MSP and 17 controls. All walked on a motorized self-paced treadmill interfaced with a three-dimensional virtual walkway. The audio tempo was scaled (75%, 100%, and 125%) from baseline cadence, and optic flow was either absent, or scaled to 50% or 100% of preferred walking speed. Gait speed was measured during each condition, and pain was measured before and after the experiment. Repeated measures analysis of variance showed that audio tempo above baseline cadence significantly increased walking speed in both groups, F(3, 99)=10.41, p<0.001. Walking speed increases of more than 25% occurred in both groups in the 125% audio tempo condition, without any significant increase in pain. There was also a trend toward increased walking speeds with the use of optic flow, but the results in this study did not achieve significance at the p<0.05 level, F(2, 66)=2.01, p=0.14. Further research is needed to establish the generalizability of increasing movement speed across different physical performance tasks in VR.