Optical Flow Field Research Articles

Human action recognition using high-order feature of optical flows

Optical flow is widely used in human action recognition. However, the influence of complex background on optical flow often leads to low recognition efficiency. To deal with this issue, an optical flow-based physical feature-driven action recognition framework is proposed in this paper. We first calculate the original dense optical flow field. Then, for reducing computational burden, joint action relevance that can eliminate the pseudo-optical flow in complex background is developed. After that, a more state flow field is obtained by local spatial–temporal thermal diffusion processing. On this basis, we design a feature descriptor that takes the divergence, curl and gradient features of flow field into consideration. Finally, we adopt Fisher vector to encode descriptors for classification. Experimental on HMDB51, KTH and UCF101 datasets proves that actions in complex background can be recognized accurately by the proposed framework, which outperforms the already developed methods.

Segmentation of the Ventricle Membranes in Short‐Axis Sequences by Optical Flow Base on DLSRE Model

Recent studies have pointed out that the boundary of the extracted ventricle membranes is unsmooth, and the segmentation of the cardiac papillary muscle and trabecular muscle do inconformity the clinical requirements. To address these issues, this paper proposes an automatic segment algorithm for continuously extracting ventricle membranes boundary, which adopts optical flow field information and sequential images information. The images are cropped by frame difference method, which according to the continuity of adjacent slices of cardiac MRI images. The roughly boundary of epicardium is extracted by the Double level set region evolution (DLSRE) model, which combines image global information, local information and edge information. The ventricle endocardium and epicardial contours are tracked according to the optical flow field information between image sequences. The segmentation results are optimized by Delaunay triangulation algorithm. The experimental results demonstrate that the proposed method can improve the accuracy of segmenting the ventricle endocardium and epicardium contours, and segment the contour of the smooth ventricle membrane edge that meets the clinical definition.

Deep learning based torsional nystagmus detection for dizziness and vertigo diagnosis

Dizziness and vertigo are common clinical symptoms and typical complaints of many vestibular diseases. In the bedside examination of dizziness and vertigo, nystagmus is the most sensitive and specific sign of vestibular lesions. The measurement of nystagmus pattern by infrared video goggle collected in clinic can provide a valuable diagnostic information for dizziness and vertigo. This paper mainly studies the automatic detection of torsional BPPV nystagmus based on deep learning, thus assisting clinicians diagnose dizziness and vertigo conveniently. In order to eliminate the invalid frames from the blinking of patients under observation, a convolutional neural network(ConvNet) based eye movement video condensation approach is proposed. When calibrating the moving pupil in the captured frame sequence, the Hough transform and trajectory tracking based on template matching are well combined to improve the robustness to eyelash occlusion and pupil deformation. In addition, the optical flow field of moving eyeball is exploited to characterize the torsion motion of torsional nystagmus, based on which a Torsion-aware Bi-Stream Identification Network (TBSIN) model is proposed. Furthermore, through label-error correction based on temporal consistency, we can merge multiple continuous torsional frames into torsional nystagmus segments for clinical diagnosis. Experiments are conducted on a clinically collected torsional nystagmus video dataset and promising experimental results show the effectiveness of the proposed approach. In particular, we achieve 85.73% and 81.00% in view of Accuracy and F1 measurements for frame-level identification, as well as IOU performance 67.45% for final torsional nystagmus segment localization.

Reference-Free DIBR-Synthesized Video Quality Metric in Spatial and Temporal Domains

Depth image-based rendering (DIBR) techniques play an important role in free viewpoint videos (FVVs), which have a wide range of applications including immersive entertainment, remote monitoring, education, etc. FVVs are usually synthesized by DIBR techniques in a &#x201C;blind&#x201D; environment (without a reference video). Thus, an effective reference-free synthesized video quality assessment (VQA) metric is vital. At present, many image quality assessment (IQA) algorithms for DIBR-synthesized images have been proposed, but limited researches have been concerned about the quality assessment of DIBR-synthesized videos. To this end, this paper proposes a novel reference-free VQA method for synthesized videos, which operates in Spatial and Temporal Domains, dubbed as STD. The design fundamental of the proposed STD metric considers the effects of two major distortions introduced by DIBR techniques on the visual quality of synthesized videos. First, considering the geometric distortion introduced by DIBR technologies can increase high-frequency contents of the synthesized frame, the influence of the geometric distortion on the visual quality of a synthesized video can be effectively evaluated by estimating high-frequency energies of each synthesized frame in spatial domain. Second, temporal inconsistency caused by DIBR techniques brings the temporal flicker distortion, which is one of the most annoying artifacts in DIBR-synthesized videos. In temporal domain, we quantify temporal inconsistency by measuring motion differences between consecutive frames. Specifically, optical flow method is first used to estimate the motion field between adjacent frames. Then, we calculate the structural similarity of adjacent optical flow fields and further adopt the structural similarity value to weight the pixel differences of adjacent optical flow fields. Experiments show that the above two features are able to well perceive the visual quality of DIBR-synthesized videos. Furthermore, since the two features are extracted from spatial and temporal domains, respectively, we integrate them using a linear weighting strategy to obtain our STD metric, which proves advantageous over two components and the competing state-of-the-art I/VQA methods. The source code is available at <uri>https://github.com/wgc-vsfm/DIBR-video-quality-assessment</uri>.

High-Resolution 3-Dimensional Contact Deformation Tracking for FingerVision Sensor With Dense Random Color Pattern

Recent studies on vision-based tactile sensing have shown promising results on the perception of contact information, which could improve the performance of dexterous manipulations. However, 3-dimensional contact deformation tracking at a higher resolution is desired and remains a challenge for vision-based tactile sensors with monocular configurations. In this work, a similar hardware structure to our previous FingerVision sensor is adopted. The dot markers are replaced with a novel random color pattern as the sensor's tracking target and a dense optical flow algorithm is used to track the deformation of its elastic contact interface. This results in a more accurate 2-dimensional deformation field estimation at a higher resolution in comparison with that obtained using sparse dot markers. Additionally, the denser and more accurate deformation field enables depth estimation with better fidelity. To achieve depth estimation purely from the optical flow field, Gaussian density feature extraction and processing framework are proposed. The resulting depth map can be used independently as a tactile sensing modality, or jointly with the accurate in-plane displacement field as a more complete deformation tracking of contact interfaces for vision-based tactile sensors.

Whole-pixel registration of non-rigid images using correspondences interpolation on sparse feature seeds

Whole pixel registration of non-rigid images with high accuracy and efficiency is a challenging problem in computer vision. To address this issue, we propose a correspondence vector field (CVF) Interpolation approach based on sparse matching of feature seeds. First, we detect and match two types of feature seeds to improve the accuracy of the later dense CVF interpolation. The first type of feature seeds is to guarantee the accuracy of the motion boundary, while the second one is to achieve the uniform distribution of seeds, which is helpful to improve the effect of interpolation. Second, we regionally estimate the dense CVF using the proposed interpolation approach on this basis. At last, we realize the whole-pixel registration of non-rigid images to yield the image alignment. Different from the traditional CVF interpolation approaches based on optical flow field, ours is based on the sparse matching of feature seeds. Thus, it is not limited to the large displacements and tends to achieve the accurate matching of certain key points easily, which is critical to the final interpolation result. Qualitative and quantitative experimental results on several internationally used datasets demonstrate that our approach outperforms the state-of-the-art ones.

Optical Flow based CNN for detection of unlearnt deepfake manipulations

Abstract A new phenomenon named Deepfakes constitutes a serious threat in video manipulation. AI-based technologies have provided easy-to-use methods to create extremely realistic videos. On the side of multimedia forensics, being able to individuate this kind of fake contents becomes ever more crucial. In this work, a new forensic technique able to detect fake and original video sequences is proposed; it is based on the use of CNNs trained to distinguish possible motion dissimilarities in the temporal structure of a video sequence by exploiting optical flow fields. The results obtained highlight comparable performances with the state-of-the-art methods which, in general, only resort to single video frames. Furthermore, the proposed optical flow based detection scheme also provides a superior robustness in the more realistic cross-forgery operative scenario and can even be combined with frame-based approaches to improve their global effectiveness.

Sensory Input Modulates Microsaccades during Heading Perception.

Microsaccades are small eye movements produced during attempted fixation. During locomotion, the eyes scan the environment; the gaze is not always directed to the focus of expansion of the optic flow field. We sought to investigate whether the microsaccadic activity was modulated by eye position during the view of radial optic flow stimuli, and if the presence or lack of a proprioceptive input signal may influence the microsaccade characteristics during self-motion perception. We recorded the oculomotor activity when subjects were either standing or sitting in front of a screen during the view of optic flow stimuli that simulated specific heading directions with different gaze positions. We recorded five trials of each stimulus. Results showed that microsaccade duration, peak velocity, and rate were significantly modulated by optic flow stimuli and trial sequence. We found that the microsaccade rate increased in each condition from trial 1 to trial 5. Microsaccade peak velocity and duration were significantly different across trials. The analysis of the microsaccade directions showed that the different combinations of optic flow and eye position evoked non-uniform directions of microsaccades in standing condition with mean vectors in the upper-left quadrant of the visual field, uncorrelated with optic flow directions and eye positions. In sitting conditions, all stimuli evoked uniform directions of microsaccades. Present results indicate that the proprioceptive signals when the subjects stand up creates a different input that could alter the eye-movement characteristics during heading perceptions.

Flow parsing and biological motion

Flow parsing is a way to estimate the direction of scene-relative motion of independently moving objects during self-motion of the observer. So far, this has been tested for simple geometric shapes such as dots or bars. Whether further cues such as prior knowledge about typical directions of an object’s movement, e.g., typical human motion, are considered in the estimations is currently unclear. Here, we adjudicated between the theory that the direction of scene-relative motion of humans is estimated exclusively by flow parsing, just like for simple geometric objects, and the theory that prior knowledge about biological motion affects estimation of perceived direction of scene-relative motion of humans. We placed a human point-light walker in optic flow fields that simulated forward motion of the observer. We introduced conflicts between biological features of the walker (i.e., facing and articulation) and the direction of scene-relative motion. We investigated whether perceived direction of scene-relative motion was biased towards biological features and compared the results to perceived direction of scene-relative motion of scrambled walkers and dot clouds. We found that for humans the perceived direction of scene-relative motion was biased towards biological features. Additionally, we found larger flow parsing gain for humans compared to the other walker types. This indicates that flow parsing is not the only visual mechanism relevant for estimating the direction of scene-relative motion of independently moving objects during self-motion: observers also rely on prior knowledge about typical object motion, such as typical facing and articulation of humans.

Attentional capture from looming alters perception

ABSTRACT Studies suggest looming motion represents a special class of attentional capture stimulus due to behavioural urgency: the need to act upon objects moving toward us in an environment. In particular, one theory suggests that faster reaction times to targets cued by looming relative to receding motion are driven by post-attentional, motor-priming processes beyond the attentional capture effects seen with other stimulus qualities such as colour pop-out. The present study tested this theory using a relative size judgment task where targets were pre-cued by looming and receding optic flow fields. Results show systematic increases in the perceived size of targets that were cued by looming flow fields, consistent with previous attentional capture studies using onset cues. These results challenge theories attributing behavioural changes from looming motion to motor-priming alone.

Frame Deletion Detection Based on Optical Flow Orientation Variation

Frame deletion detection is an arrestive field for video forensics in recent years. This paper presents a frame deletion detection algorithm based on optical flow orientation variation. Optical flow field has been proved to be effective for the detection of the frame deletion operation. However, current optical-flow based methods have relatively poor detection performance for real-world videos. Our research aims to develop a new forensic method to detect frame deletion for real-world videos. The proposed algorithm firstly develops an analytical model for flow orientation variation between adjacent optical flow fields, and exhibits a novel forensic clue for frame deletion detection, under an ideal but reasonable assumption. Accordingly, an effective descriptor, namely pseudo flow orientation variation (PFOV), is created to approximate the flow orientation variation, as a discriminative description feature to capture the frame deletion trace. The Frobenius norm and a smooth function are also introduced to quantify each descriptor to form an one-dimensional time series, and further statistics technique is adopted to detect the anomalous points in the time series caused by frame deletion. In addition, to improve the descriptor performance, the Robust Principal Component Analysis (RPCA) is applied to extract moving objects from video sequences and compute the proposed descriptor on them. The tests are made on 324 real-world videos, and experiment results demonstrate that the true positive rate can reach 90.12%, meanwhile the false alarm rate is 7.71%, which indicates the superiority of the proposed method in the forensic algorithms for real-world video.

SDOF-GAN: Symmetric Dense Optical Flow Estimation With Generative Adversarial Networks

There is a growing consensus in computer vision that symmetric optical flow estimation constitutes a better model than a generic asymmetric one for its independence of the selection of source/target image. Yet, convolutional neural networks (CNNs), that are considered the de facto standard vision model, deal with the asymmetric case only in most cutting-edge CNNs-based optical flow techniques. We bridge this gap by introducing a novel model named SDOF-GAN: symmetric dense optical flow with generative adversarial networks (GANs). SDOF-GAN realizes a consistency between the forward mapping (source-to-target) and the backward one (target-to-source) by ensuring that they are inverse of each other with an inverse network. In addition, SDOF-GAN leverages a GAN model for which the generator estimates symmetric optical flow fields while the discriminator differentiates the "real" ground-truth flow field from a "fake" estimation by assessing the flow warping error. Finally, SDOF-GAN is trained in a semi-supervised fashion to enable both the precious labeled data and large amounts of unlabeled data to be fully-exploited. We demonstrate significant performance benefits of SDOF-GAN on five publicly-available datasets in contrast to several representative state-of-the-art models for optical flow estimation.

Application of Optical Flow Field in Out-of-plane Displacement Measurement for Optical Fiber Material

Application of Optical Flow Field in Out-of-plane Displacement Measurement for Optical Fiber Material

Image tracking and matching algorithm of semi-dense optical flow method

The traditional optical flow method is based on the assumption of spatial consistency of the optical flow field. It is easy to reduce the tracking quality and even lead to the loss of target tracking in the areas of image feature missing, boundary and occlusion. A semi-dense optical flow method is proposed to realise stable tracking of image features. Firstly, the feature points are preserved by calculating the pixel points with large change of pixel gradient in the image; Secondly, according to the principle of grey level invariance, the grey level difference function between the corresponding feature points of adjacent frames is constructed; Finally, the gradient descent principle is used to optimise the grey difference function and realise the accurate matching of feature points of adjacent frames. The results show that compared with the traditional LK optical flow method, this algorithm can effectively improve the feature tracking capability, and at the same time can effectively retain the useful information in the image. Compared with the traditional feature point matching method, the algorithm presented in this paper has an efficient operation rate.

Human Action Recognition Based on Motion Feature and Manifold Learning

Human action recognition is an important task in the fields of video content analysis and computer vision. Since the performance of most existing action recognition frameworks depends on the representation of features, many researches aim to construct more discriminative features. In this paper, we propose a manifold learning framework based on optical flow for action recognition. First, we calculate the dense optical flow field of the original video sequence, and the attention pooling layer (AP) is adopted to separate target area and background area to eliminate background interference. On this basis, motion features (MF) based on the physical characteristics of dense optical flow are developed to characterize human motion information. After that, manifold learning is introduced to calculate the motion variance features (MVF), which reflect the change rate of motion features and measure the spatial correlation between features in non-Euclidean space. Finally, fusing the MVF obtained by manifold learning and MF, feeding fusion features into two fully connected layers (FC) in series for action classification and recognition. Experiments on several classic datasets show that the proposed method achieves 0.98%, 1.86% and 0.99% performance improvement on UCF 101, HMDB51 and JHMDB.

Multi-channel video target tracking algorithm based on feature constraint and optical flow field model

Multi-channel video target tracking algorithm based on feature constraint and optical flow field model

Heading perception depends on time-varying evolution of optic flow

There is considerable support for the hypothesis that perception of heading in the presence of rotation is mediated by instantaneous optic flow. This hypothesis, however, has never been tested. We introduce a method, termed "nonvarying phase motion," for generating a stimulus that conveys a single instantaneous optic flow field, even though the stimulus is presented for an extended period of time. In this experiment, observers viewed stimulus videos and performed a forced-choice heading discrimination task. For nonvarying phase motion, observers made large errors in heading judgments. This suggests that instantaneous optic flow is insufficient for heading perception in the presence of rotation. These errors were mostly eliminated when the velocity of phase motion was varied over time to convey the evolving sequence of optic flow fields corresponding to a particular heading. This demonstrates that heading perception in the presence of rotation relies on the time-varying evolution of optic flow. We hypothesize that the visual system accurately computes heading, despite rotation, based on optic acceleration, the temporal derivative of optic flow.

Optical flow estimation combining with illumination adjustment and edge refinement in livestock UAV videos

The performance of optical flow estimation is affected by many factors. The impact of illumination changes and video quality degradation in unmanned aerial vehicles videos on optical flow estimation cannot be ignored. Inspired by the human retina’s visual adaptation mechanism, we propose a mechanism for illumination adjustment that imitates retinal processing in order to reduce illumination variation. We further introduce an edge refinement mechanism into optical flow estimation that is based on a weighted neighborhood filtering. The experimental results on public benchmarks inlcuding KITTI 2012, KITTI 2015, MPI Sintel and Middlebury show that the proposed approach is robust on illumination and preserves accurate motion details. Further, experiments on outdoor livestock UAV videos show that the proposed approach implements illumination robustness and preserves the accurate detection of motion edges in other types of video. The performance of the proposed method on public benchmarks and livestock UAV videos demonstrates that the proposed approach improves motion edge accuracy of optical flow fields in varying illumination.

Automatic layered RGB‐D scene flow estimation with optical flow field constraint

Automatic layered RGB‐D scene flow estimation with optical flow field constraint

Visualization by P-flow: gradient- and feature-based optical flow and vector fields extracted from image analysis.

We proposed a method for extracting the optical flow suitable for visualization, pseudo-flow (P-flow), from a natural movie [Exp. Brain Res.237, 3321 (2019)EXBRAP0014-481910.1007/s00221-019-05674-0]. The P-flow algorithm comprises two stages: (1) extraction of a local motion vector field from two successive frames and (2) tracking of vectors between two successive frame pairs. In this study, we show that while P-flow takes a feature (vector) tracking approach, it is also classified as a gradient-based approach that satisfies the brightness constancy constraint. We also incorporate interpolation and a corner detector to address the shortcomings associated with the two approaches.