Optical Flow Vectors Research Articles

Angle-Weighted Reverse Time Migration With Wavefield Decomposition Based on the Optical Flow Vector

Reverse time migration (RTM) is based on the two-way wave equation, so its imaging results obtained by conventional zero-lag cross-correlation imaging conditions contain a lot of low-wavenumber noises. So far, the wavefield decomposition method based on the Poynting vector has been developed to suppress these noises; however, this method also has some problems, such as unstable calculation of the Poynting vector, low accuracy of wavefield decomposition, and poor effect of large-angle migration artifacts suppression. This article introduces the optical flow vector method to RTM to realize high-precision wavefield decomposition for both the source and receiver wavefields and obtains four directions of wavefields: up-, down-, left-, and right-going. Then, the cross-correlation imaging sections of one-way propagation components of forward- and back-propagated wavefields are optimized and stacked. On this basis, the reflection angle of each imaging point is calculated based on the optical flow vector, and an attenuation factor related to the reflection angle is introduced as the weight to generate the optimal stack images. The tests of theoretical model and field marine seismic data illustrate that compared with the conventional RTM with wavefield decomposition based on the Poynting vector, the angle-weighted RTM with wavefield decomposition based on the optical flow vector proposed in this article can achieve wavefield decomposition for both the source and receiver wavefields and calculate the reflection angle of each imaging point more accurately and stably. Moreover, the proposed method adopts angle weighting processing, which can further eliminate large-angle migration artifacts and effectively improve the imaging accuracy of RTM.

Modeling Physiological Sources of Heading Bias from Optic Flow.

Human heading perception from optic flow is accurate for directions close to the straight-ahead and systematic biases emerge in the periphery (Cuturi and Macneilage, 2013; Sun et al., 2020). In pursuit of the underlying neural mechanisms, primate brain dorsal medial superior temporal (MSTd) area has been a focus because of its causal link with heading perception (Gu et al., 2012). Computational models generally explain heading sensitivity in individual MSTd neurons as a feedforward integration of motion signals from medial temporal (MT) area that resemble full-field optic flow patterns consistent with the preferred heading direction (Britten, 2008; Mineault et al., 2012). In the present simulation study, we quantified within the structure of this feedforward model how physiological properties of MT and MSTd shape heading signals. We found that known physiological tuning characteristics generally supported the accuracy of heading estimation, but not always. A weak-to-moderate overrepresentation of peripheral headings in MSTd garnered the highest accuracy and precision out of the models that we tested. The model also performed well when noise corrupted high proportions of the optic flow vectors. Such a peripheral MSTd model performed well when units possessed a range of receptive field (RF) sizes and were strongly direction tuned. Physiological biases in MT direction tuning toward the radial direction also supported heading estimation, but the tendency for MT preferred speed and RF size to scale with eccentricity did not. Our findings help elucidate the extent to which different physiological tuning properties influence the accuracy and precision of neural heading signals.

Rolling-Shutter-stereo-aware motion estimation and image correction

The vast majority of modern consumer-grade cameras employ the Rolling Shutter (RS) mechanism, which often produces geometrically distorted images due to inter-row delay during imaging, that needs to be properly handled. While several models and solutions have been proposed for the motion estimation problem of the monocular RS camera, the corresponding motion estimation model and algorithm have not been developed for the standard RS stereo setup. This is primarily due to the temporal-dynamic nature of the RS camera, which results in scanline-varying camera poses. This paper fills in this gap by developing two tractable RS stereo models under constant velocity motion and constant acceleration motion respectively, which only need to simply scale the optical flow vector. We further propose two corresponding RS stereo motion estimation algorithms to accurately compute the relative pose of the RS stereo camera between two consecutive frames. As a result of utilizing these RS stereo models associated with the parallel visual rays, our proposed methods offer better disambiguation of the translation and rotation, hence leading to significantly improved performance compared with the Global Shutter (GS) stereo model. Furthermore, we develop an RS stereo image correction method to remove RS distortions and recover high-quality GS images by conducting motion compensation. Experimental results on both simulated data and real RS stereo images show the superiority of our proposed models and methods.

An Improved Fractional-Order Variational Optical Flow Model Combining Structure Tensor

Dealing with problems of illumination changes in optical flow estimation, an improved variational optical flow model is proposed in this paper. The local structure constancy constraint (LSCC) is applied in the data term of the traditional HS (Horn & Schunck) optical flow model to substitute the brightness constancy constraint. The fractional-order smoothness constraint (FSC) is applied in the smoothness term of the HS model. Then, the detailed calculation processes from the optical flow model to the optical flow value are explained. The structure tensor in LSCC is an image feature that is constant in the illumination changes scene. The fractional differential coefficient in FSC can fuse the local neighborhood optical flow vector into the optical flow vector of the target pixel, which can improve the integrity of the motion region with the same motion speed. Combining LSCC with FSC, our improved optical flow model can obtain an accurate optical flow field with clear outline in the illumination abnormity scene. The experimental results show that, compared with other optical flow models, our model is more suitable for the illumination changes scene and can be employed in outdoor motion detection projects.

OFx: A method of 4D image construction from free-breathing non-gated MRI slice acquisitions of the thorax via optical flux.

Since real-time 4D dynamic magnetic resonance imaging (dMRI) methods with adequate spatial and temporal resolution for imaging the pediatric thorax are currently not available, free-breathing slice acquisitions followed by appropriate 4D construction methods are currently employed. Self-gating methods, which extract breathing signals only from image information without any external gating technology, have much potential for this purpose, such as for use in studying pediatric thoracic insufficiency syndrome (TIS). Patients with TIS frequently suffer from extreme malformations of the chest wall, diaphragm, and spine, leading to breathing that is very complex, including deep or shallow respiratory cycles. Existing 4D construction methods cannot perform satisfactorily in this scenario, and most are not fully automatic, requiring manual interactive operations. In this paper, we propose a novel fully automatic 4D image construction method based on an image-derived concept called flux to address these challenges. We utilized 25 dMRI data sets from 25 pediatric subjects with no known thoracic anomalies and 58 dMRI data sets from 29 patients with TIS where each patient had a dMRI scan before and after surgery. A time sequence of 80 slices are acquired at each sagittal location continuously at a rate of ~480ms per slice under free-breathing conditions, with 30-40 sagittal locations across the chest for each subject depending on the thoracic size. In our approach, we first extract the breathing signal for each sagittal location based on the flux of the optical flow vector field of the body region from the image time series. Here, for each time point of respiratory phase, the net flux of the body region can be regarded as the flux going into or out of the body region, which we term Optical Flux (OFx). OFx provides a very robust representation of the real breathing motion of the thorax. OFx allows us to perform a full analysis of all respiratory cycles, extract only normal cycles in a robust manner, and map all extracted normal cycles on to one cosine respiration model for each sagittal location. Subsequently, we re-sample one normal cycle from the respiration model for each location independently. The normal cycle models associated with the different sagittal locations are finally composited to form the final constructed 4D image. We employ several metrics to evaluate the quality of the 4D construction results: Eie - error in locating time instants corresponding to end inspiration and end expiration; Eto - deviation from correct temporal order in each detected normal cycle; Ess - deviation in spatial smoothness; and Esc - deviation from spatial continuity as scored by a reader. The means and standard deviations of these metrics for normal subjects and TIS patients are found to be, respectively: Eie: 0.25±0.05 and 0.38±0.16 in units of time instance (ideal value=0); Eto: 2.7% ± 2.3% and 1.8%±2% (ideal value=0%); Ess: 0.5±0.17 and 0.54±0.25 in pixel units (ideal value=0); Esc: 4.6±0.48 and 4.56±0.98 (score range: best=5, worst=1). The results show that the OFx method achieves excellent spatial and temporal continuity and its yield was 100% meaning that it successfully performed 4D construction on every data set tested. Compared to a recently published method, OFx is fully automatic requiring about 5min of computational time per study starting from acquired dMRI scans. The method achieves high temporal and spatial continuity even on complex TIS data sets that include many abnormal respiratory cycles. A new 4D dMRI construction method based on the concept of optical flux is presented which is fully automatic and very robust in deriving respiratory signals purely from dynamic image sequences even when presented with complex breathing patterns due to severe disease conditions like TIS. Evaluations show that its accuracy is comparable to the variations found in manual annotations. An important characteristic of the method is that it is independent of the number of sagittal locations used in the construction process, which suggests that it is applicable to imaging techniques where data are acquired at only a few sagittal locations instead of the full width of the thorax. The method is not tied to any specific imaging modality, as demonstrated in this paper on not just dMRI but dynamic computed tomography (CT) as well.

Deep Learning-Based Congestion Detection at Urban Intersections.

In this paper, a deep learning-based traffic state discrimination method is proposed to detect traffic congestion at urban intersections. The detection algorithm includes two parts, global speed detection and a traffic state discrimination algorithm. Firstly, the region of interest (ROI) is selected as the road intersection from the input image of the You Only Look Once (YOLO) v3 object detection algorithm for vehicle target detection. The Lucas-Kanade (LK) optical flow method is employed to calculate the vehicle speed. Then, the corresponding intersection state can be obtained based on the vehicle speed and the discrimination algorithm. The detection of the vehicle takes the position information obtained by YOLOv3 as the input of the LK optical flow algorithm and forms an optical flow vector to complete the vehicle speed detection. Experimental results show that the detection algorithm can detect the vehicle speed and traffic state discrimination method can judge the traffic state accurately, which has a strong anti-interference ability and meets the practical application requirements.

Elastic wave‐mode separation in 2D transversely isotropic media using optical flow

ABSTRACTWave‐mode separation is a critical step in anisotropic elastic‐wave imaging. To avoid high computational costs and additional corrections, we apply a separation formula in the space domain that projects Cartesian components of the elastic wavefield onto the polarization vectors in the orthogonal direction. By solving the Christoffel equation, we implement the conversion from the phase velocity direction to the polarization direction. We use similarity between the seismic wave propagation and the optical flow problem to calculate the phase angle based on the Horn–Schunck algorithm. Compared with the conventional Poynting vector, the optical flow vector is more robust, particularly for complex underground structures. We demonstrate the performance of the proposed approach for two‐dimensional transversely isotropic media with three numerical examples and discuss the potential extension to three‐dimensional media.

Design of a Facial Landmark Detection System Using a Dynamic Optical Flow Approach

Many facial landmark methods based on convolutional neural networks (CNN) have been proposed to achieve favorable detection results. However, the instability landmarks that occur in video frames due to CNNs are extremely sensitive to input image noise. To solve this problem of landmark shaking, this study proposes a simple and effective facial landmark detection method comprising a lightweight U-Net model and a dynamic optical flow (DOF). The DOF uses the fast optical flow to obtain the optical flow vector of the landmark and uses dynamic routing to improve landmark stabilization. A lightweight U-Net model is designed to predict facial landmarks with a smaller model size and less computational complexity. The predicted facial landmarks are further fed to the DOF approach to deal with the unstable shaking. Finally, a comparison of several common methods and the proposed detection method is made on several benchmark datasets. Experimental evaluations and analyses show that not only can the lightweight U-Net model achieve favorable landmark prediction but also the DOF stabilizing method can improve the robustness of landmark prediction in both static images and video frames. It should be emphasized that the proposed detection system exhibits better performance than others without requiring heavy computational loadings.

RETRACTED: FPGA-Based Motion Estimation Algorithm Optimization

The new international video compression standards (HEVC) are referred to as video motion detection and efficient physical education video encoding. HEVC is determined by calculating the increasing complexity of video compression efficiency of the remarkable possible. Motion estimation is the most complicated arithmetic unit video encoder. Integer motion estimation and HEVC 70%, accounted for, and the video encoder's complexity calculates fractional motion estimation. High-Level Synthesis (HLS) tools algorithms have been successfully used in digital signal processing on FPGA. According to the actual requirements of the project, it is set as sports video management, target detection, video acquisition and target tracking. In order to adapt the smoothing time, the hardware configure ration consists of two parts: the intermediate results of the final optical flow vector and the optical flow calculating module generates the derivative calculation to adapt the smoothing time. Run built processor, software. FPGA chip, designed for directing data flow management and hardware components. Recommend using a first interpolation HEVC FPGA implementation scores and Vivado HLS motion estimation. The sports video target detection and tracking system designed in this paper analyzes the scene to be detected, further analyzes the moving target trajectory, and tracks the target according to the trajectory.

Learnable spatiotemporal feature pyramid for prediction of future optical flow in videos

The success of deep learning-based techniques in solving various computer vision problems motivated the researchers to apply deep learning to predict the optical flow of a video in the next frame. However, the problem of predicting the motion of an object in the next few frames remains an unsolved and less explored problem. Given a sequence of frames, predicting the motion in the next few frames of the video becomes difficult in cases where the displacement of optical flow vector across frames is large. Traditional CNNs often fail to learn the dynamics of the objects across frames in case of large displacements of objects in consecutive frames. In this paper, we present an efficient CNN based on the concept of feature pyramid for extracting the spatial features from a few consecutive frames. The spatial features extracted from consecutive frames by a modified PWC-Net architecture are fed into a bidirectional LSTM for obtaining the temporal features. The proposed spatiotemporal feature pyramid is able to capture the abrupt motion of the moving objects in video, especially when displacement of the object is large across the consecutive frames. Further, the proposed spatiotemporal pyramidal feature can effectively predict the optical flow in next few frames, instead of predicting only the next frame. The proposed method of predicting optical flow outperforms the state of the art when applied on challenging datasets such as “MPI Sintel Final Pass,” “Monkaa” and “Flying Chairs” where abrupt and large displacement of the moving objects in consecutive frames is the main challenge.

Object tracking using motion flow projection for pan-tilt configuration

We propose a new object tracking model for two degrees of freedom mechanism. Our model uses a reverse projection from a camera plane to a world plane. Here, the model takes advantage of optic flow technique by re-projecting the flow vectors from the image space into world space. A pan-tilt (PT) mounting system is used to verify the performance of our model and maintain the tracked object within a region of interest (ROI). This system contains two servo motors to enable a webcam rotating along PT axes. The PT rotation angles are estimated based on a rigid transformation of the the optic flow vectors in which an idealized translation matrix followed by two rotational matrices around PT axes are used. Our model was tested and evaluated using different objects with different motions. The results reveal that our model can keep the target object within a certain region in the camera view.

FullStop: A Camera-Assisted System for Characterizing Unsafe Bus Stopping

Road safety is a critical issue worldwide. We believe that mobile devices can play a positive role in this context by detecting dangerous conditions and providing feedback. This paper focuses on a specific problem in developing countries: the stopping behaviour of buses in the vicinity of bus stops. For instance, buses could arrive at a bus stop but continue rolling forward instead of coming to a complete halt, or could stop some distance away from the bus stop, possibly even in the middle of a busy road. Such behaviors put at risk the passengers boarding or alighting the bus, and also the people waiting at a bus stop. We present FullStop, a smartphone-based system that detects safety risks emanating from stopping behavior like the ones listed above. We show that the GPS and inertial sensors are unable to perform the fine-grained detection needed. Therefore, our approach in FullStop is based on the view obtained from looking out to the front of the vehicle using the camera of a smartphone that is mounted on the front windshield. Using optical flow vectors, with several refinements, FullStop running on a smartphone is able to effectively detect various unsafe bus stopping behaviours.

Single-stream long-term optical flow convolution network for action recognition of lameness dairy cow

Typical dairy cow lameness actions are only several seconds in duration and exhibit characteristic spatiotemporal structures. Manual lameness detection poses several problems, such as difficulty in the detection of sudden, severe or early lameness behavior. In this paper, our proposed method attempts to capture this structure and to learn action representations with convolutional neural networks. However, such representations are typically learned at the level of a few video frames, thereby failing to model actions over their full temporal extent. In this study, we learn video representations using neural networks with single-stream long-term optical flow convolution. To evaluate the performance of the algorithm, 756 videos were randomly selected from 1080 dairy cow videos as the original training videos, and the remaining videos were used as test videos. The experimental results demonstrate that single-stream long-term optical flow convolution network models with increased temporal extents improve the accuracy of dairy cow lameness action recognition. We also study the impacts of various low-level representations, such as raw values of video pixels and optical flow vector fields, and we demonstrate the importance of high-quality optical flow estimation for learning accurate dairy cow lameness action models. We report state-of-the-art results on challenging benchmarks for the dairy cow lameness action video set (98.24%).

4D image construction from free-breathing MRI slice acquisitions of the thorax based on a concept of flux.

Retrospective 4D image construction from continuously acquired 2D slices is a necessary step to achieve high-quality 4D images. Self-gating methods, which extract breathing signals only from image information without any external gating technology, have much potential, such as in pediatric patients with thoracic insufficiency syndrome (TIS) who suffer from extreme malformations of the chest wall, diaphragm, and spine, leading to breathing that is very complex with lots of abnormal respiration cycles, including very deep or shallow cycles. Existing methods do not work well in this clinical scenario and most are not fully automatic, requiring some manual interactive operations. In this paper, we propose a fully automatic 4D dMRI construction method based on the concept of flux to address the 4D image construction from 2D slices of subjects with complex respiration. Firstly, we extract the breathing signal for each location based on the flux of the optical flow vector field of the body region from the image series. Then, we give a full analysis for all cycles and extract several normal ones and map them to one cosine respiration model for each location. After that, we re-sample one normal cycle from the respiration model for each location independently. All of these resampled normal cycles form the final constructed 4D image. Qualitative and quantitative evaluations on 25 subjects show that the proposed method can handle datasets from subjects with more complex respiration and achieves good self-consistency results while maintaining time and space continuity.

Explanation of the Sense of Visual Perception in Cornering Based on Gaze Position and Optical Flows

In the recently developed self-driving technologies, users often experience strangeness when performing cornering control such as in lane-keeping assist. To realize human-centered cornering control, we had proposed a human perception model that considers optical flow and gaze position for cornering. In this paper, we propose a divergence point detection method for identifying the target position on the spherical image corresponding to the gaze position. We first discuss the relationship between optical flow and gaze movement in cornering, and we identify optical flows radiating outward from the gaze position. Next, we explain a divergence point detection method that uses optical flow vectors based on particle swarm optimization. For verifying the proposed method, we perform an experiment using an actual vehicle; the results show that the divergence detection method can detect a divergence point.

Abnormal Crowd Behavior Detection Using Motion Information Images and Convolutional Neural Networks

We introduce a novel method for abnormal crowd event detection in surveillance videos. Particularly, our work focuses on panic and escape behavior detection that may appear because of violent events and natural disasters. First, optical flow vectors are computed to generate a motion information image (MII) for each frame, and then MIIs are used to train a convolutional neural network (CNN) for abnormal crowd event detection. The proposed MII is a new formulation that provides a visual appearance of crowd motion. The proposed MIIs make the discrimination between normal and abnormal behaviors easier. The MII is mainly based on the optical flow magnitude, and angle difference computed between the optical flow vectors in consecutive frames. A CNN is employed to learn normal and abnormal crowd behaviors using MIIs. The MII generation, and the combination with a CNN is a new approach in the context of abnormal crowd behavior detection. Experiments are performed on commonly used datasets such as UMN and PETS2009. Evaluation indicates that our method achieves the best results.

An Abnormal Behavior Detection Method using Optical Flow Model and OpenPose

Abnormal behavior detection and recognition of pedestrian in escalator has always been a challenging task in intelligent video surveillance system. To cope this problem, a method combining optical flow vector of passenger with human skeleton extraction is proposed. At first, adaptive dual fractional order optical flow model is used to estimate the optical flow field under scenes with illumination changes, low contrast and uneven illumination. At the same time, the OpenPose deep convolutional neural network is used to extract body skeleton and persons in image can be located. Then, the optical flow field and the human skeleton are combined to obtain the optical flow vector of the passenger head. After that the optical flow field of the passenger head and the step of escalator under the passenger foot are used for abnormal behavior detection and recognition, random forest is employed to behavior classifier. Experimental results show that our proposed method and its improvement strategy can accurately estimate the optical flow field in real time of low contrast outdoor videos with insufficient illumination, uneven brightness and illumination changes, the accuracy of abnormal action detection and recognition can reach to 97.98% and 92.28%.

Weakly supervised deep network for spatiotemporal localization and detection of human actions in wild conditions

Human action localization in any long, untrimmed video can be determined from where and what action takes place in a given video segment. The main hurdles in human action localization are the spatiotemporal randomnesses of their happening in a parallel mode which means the location (a particular set of frames containing action instances) and duration of any particular action in real-life video sequences generally are not fixed. At another end, the uncontrolled conditions such as occlusions, viewpoints and motions at the crisp boundary of the action sequences demand to develop a fast deep network which can be easily trained from unlabeled samples of complex video sequences. Motivated from the facts, we proposed a weakly supervised deep network model for human action localization. The model is trained from unlabeled action samples from UCF50 action benchmark. The five-channel data obtained from the concatenation of RGB (three-channel) and optical flow vectors (two-channel) are fed to the proposed convolutional neural network. LSTM network is used to yield the region of action happening area. The performance of the model is tested on UCF-sports dataset. The observation and comparative results reflect that our model can localize any action from annotation-free data samples captured in uncontrolled conditions.

A Method of Making Video Trailer Based on Artificial Intelligence

Based on the flownet2 optical flow algorithm, cascade CNN face detection and facenet face recognition algorithm, the method of automatically generating video trailers by using the sum of the optical flow vector sum per second and the absolute value of the audio loudness amplitude per second and the character duration information is proposed. It can effectively edit the visual essence clip, the video essence clip corresponding to the audio, the main character clip and the story outline to generate a trailer.

Identification of Photo-taking behaviors using Optical Flow Vector

Identification of Photo-taking behaviors using Optical Flow Vector