Optical Flow Information Research Articles

Visual-Cerebellar Pathways and Their Roles in the Control of Avian Flight.

In this paper, we review the connections and physiology of visual pathways to the cerebellum in birds and consider their role in flight. We emphasize that there are two visual pathways to the cerebellum. One is to the vestibulocerebellum (folia IXcd and X) that originates from two retinal-recipient nuclei that process optic flow: the nucleus of the basal optic root (nBOR) and the pretectal nucleus lentiformis mesencephali (LM). The second is to the oculomotor cerebellum (folia VI-VIII), which receives optic flow information, mainly from LM, but also local visual motion information from the optic tectum, and other visual information from the ventral lateral geniculate nucleus (Glv). The tectum, LM and Glv are all intimately connected with the pontine nuclei, which also project to the oculomotor cerebellum. We believe this rich integration of visual information in the cerebellum is important for analyzing motion parallax that occurs during flight. Finally, we extend upon a suggestion by Ibbotson (2017) that the hypertrophy that is observed in LM in hummingbirds might be due to an increase in the processing demands associated with the pathway to the oculomotor cerebellum as they fly through a cluttered environment while feeding.

Social targets improve body-based and environment-based strategies during spatial navigation

Encoding the position of another person in space is vital for everyday life. Nevertheless, little is known about the specific navigational strategies associated with encoding the position of another person in the wider spatial environment. We asked two groups of participants to learn the location of a target (person or object) during active navigation, while optic flow information, a landmark, or both optic flow information and a landmark were available in a virtual environment. Whereas optic flow information is used for body-based encoding, such as the simulation of motor movements, landmarks are used to form an abstract, disembodied representation of the environment. During testing, we passively moved participants through virtual space, and compared their abilities to correctly decide whether the non-visible target was before or behind them. Using psychometric functions and the Bayes Theorem, we show that both groups assigned similar weights to body-based and environment-based cues in the condition, where both cue types were available. However, the group who was provided with a person as target showed generally reduced position errors compared to the group who was provided with an object as target. We replicated this effect in a second study with novel participants. This indicates a social advantage in spatial encoding, with facilitated processing of both body-based and environment-based cues during spatial navigation when the position of a person is encoded. This may underlie our critical ability to make accurate distance judgments during social interactions, for example, during fight or flight responses.

Spatiotemporal Low-Rank Modeling for Complex Scene Background Initialization

Background modeling constitutes the building block of many computer-vision tasks. Traditional schemes model the background as a low rank matrix with corrupted entries. These schemes operate in batch mode and do not scale well with the data size. Moreover, without enforcing spatiotemporal information in the low-rank component, and because of occlusions by foreground objects and redundancy in video data, the design of a background initialization method robust against outliers is very challenging. To overcome these limitations, this paper presents a spatiotemporal low-rank modeling method on dynamic video clips for estimating the robust background model. The proposed method encodes spatiotemporal constraints by regularizing spectral graphs. Initially, a motion-compensated binary matrix is generated using optical flow information to remove redundant data and to create a set of dynamic frames from the input video sequence. Then two graphs are constructed, one between frames for temporal consistency and the other between features for spatial consistency, to encode the local structure for continuously promoting the intrinsic behavior of the low-rank model against outliers. These two terms are then incorporated in the iterative Matrix Completion framework for improved segmentation of background. Rigorous evaluation on severely occluded and dynamic background sequences demonstrates the superior performance of the proposed method over state-of-the-art approaches.

Research on HS Optical Flow Algorithm Based on Motion Estimation Optimization

The amount of computation for detecting moving objects by the optical flow algorithm is large. The optical flow information in the smooth region cannot be detected by the optical flow algorithm, and it is susceptible to noise in a complicated environment. In this study, an optimized Horn-Schunck (HS) optical flow algorithm based on motion estimation is proposed. To detect Harris corner in the image, the proposed algorithm is used in combination with the motion estimation algorithm based on macroblock to determine the region of interest (ROI) [1]. The ROI is then used as the initial motion vector for HS calculation to obtain the optical flow information. Filtering is conducted to eliminate the background noise. Experimental result shows that the application of the proposed algorithm improves the computational speed, avoids the interference of background noise, and enhances the robustness of HS. Moreover, the algorithm solves the problem rooted in the inability of the HS algorithm to detect the smooth part of optical flow information [2].

Strategy of landing behavior of the tree frog Hyla japonica.

Toads on the ground control landing behavior by a simple strategy in which they extend their elbows at roughly the same time after take-off and continue extending them at the same velocity until they land. However, this simple strategy does not work well in much more complicated arboreal habitat. We analyzed properties of jumping and timing of forelimb extension of the tree frog, Hyla japonica, to demonstrate how the arboreal species land safely. The results showed that distance, duration, and velocity of jump were not affected by explicit optic flow information, whereas the all three parameters decreased significantly in the blindfolded animals. This suggests that the frogs do not use optic flow information during flight but need visual information to plan a jump to reach the destination appropriately. Many animals generate prelanding behavior when time-to-contact reaches a threshold value. However, the results showed that a significant positive correlation was found between jump duration and time-to-contact both in normal and blindfolded animals. The slopes and the intercepts of the linear regression lines were about 0.8 and about 50msec, respectively, in both cases. These suggest that, without any visual inputs, the animal can make the decision for prelanding forelimb extension at 20% of jump duration and start extending the forelimb after a motor delay of about 50msec as in the normal condition. This strategy enables the tree frog to always maintain 80% of the jump duration to prepare for landing in their complex habitat.

SURF, CAMSHIFT와 광류를 이용한 모바일 로봇의 객체 추종에 관한 연구

This study proposed a novel tracking algorithm for a mobile robot using modified CAMSHIFT and optical flow. CAMSHIFT algorithm is vulnerable to the lighting, brightness and speed of the object because it is based on the HSV color space model. To overcome the difference of light condition, the searching window of CAMSHIFT is set by SURF algorithm. Optical flow information is used to predict the position and velocity of the fast-moving object. The CAMSHIFT searching window in an image can be adjusted quickly. Furthermore, an ultra-sonic sensor is used to calculate the distance to an object. The proposed fusion algorithm shows superior performance in terms of accuracy, speed and reliability, compared with original SURF and CAMSIFT.

Deep spatial-temporal feature fusion for facial expression recognition in static images

ABSTRACT Traditional methods of performing facial expression recognition commonly use hand-crafted spatial features. This paper proposes a multi-channel deep neural network that learns and fuses the spatial-temporal features for recognizing facial expressions in static images. The essential idea of this method is to extract optical flow from the changes between the peak expression face image (emotional-face) and the neutral face image (neutral-face) as the temporal information of a certain facial expression, and use the gray-level image of emotional-face as the spatial information. A Multi-channel Deep Spatial-Temporal feature Fusion neural Network (MDSTFN) is presented to perform the deep spatial-temporal feature extraction and fusion from static images. Each channel of the proposed method is fine-tuned from a pre-trained deep convolutional neural networks (CNN) instead of training a new CNN from scratch. In addition, average-face is used as a substitute for neutral-face in real-world applications. Extensive experiments are conducted to evaluate the proposed method on benchmarks databases including CK+, MMI, and RaFD. The results show that the optical flow information from emotional-face and neutral-face is a useful complement to spatial feature and can effectively improve the performance of facial expression recognition from static images. Compared with state-of-the-art methods, the proposed method can achieve better recognition accuracy, with rates of 98.38% on the CK+ database, 99.17% on the RaFD database, and 99.59% on the MMI database, respectively.

A dual fast and slow feature interaction in biologically inspired visual recognition of human action

Computational neuroscience studies have examined the human visual system through functional magnetic resonance imaging (fMRI) and identified a model where the mammalian brain pursues two independent pathways for recognizing biological movement tasks. On the one hand, the dorsal stream analyzes the motion information by applying optical flow, which considers the fast features. On the other hand, the ventral stream analyzes the form information with slow features. The proposed approach suggests that the motion perception of the human visual system comprises fast and slow feature interactions to identify biological movements. The form features in the visual system follow the application of the active basis model (ABM) with incremental slow feature analysis (IncSFA). Episodic observation is required to extract the slowest features, whereas the fast features update the processing of motion information in every frame. Applying IncSFA provides an opportunity to abstract human actions and use action prototypes. However, the fast features are obtained from the optical flow division, which gives an opportunity to interact with the system as the final recognition is performed through a combination of the optical flow and ABM-IncSFA information and through the application of kernel extreme learning machine. Applying IncSFA into the ventral stream and involving slow and fast features in the recognition mechanism are the major contributions of this research. The two human action datasets for benchmarking (KTH and Weizmann) and the results highlight the promising performance of this approach in model modification.

On-Board Detection of Pedestrian Intentions

Avoiding vehicle-to-pedestrian crashes is a critical requirement for nowadays advanced driver assistant systems (ADAS) and future self-driving vehicles. Accordingly, detecting pedestrians from raw sensor data has a history of more than 15 years of research, with vision playing a central role. During the last years, deep learning has boosted the accuracy of image-based pedestrian detectors. However, detection is just the first step towards answering the core question, namely is the vehicle going to crash with a pedestrian provided preventive actions are not taken? Therefore, knowing as soon as possible if a detected pedestrian has the intention of crossing the road ahead of the vehicle is essential for performing safe and comfortable maneuvers that prevent a crash. However, compared to pedestrian detection, there is relatively little literature on detecting pedestrian intentions. This paper aims to contribute along this line by presenting a new vision-based approach which analyzes the pose of a pedestrian along several frames to determine if he or she is going to enter the road or not. We present experiments showing 750 ms of anticipation for pedestrians crossing the road, which at a typical urban driving speed of 50 km/h can provide 15 additional meters (compared to a pure pedestrian detector) for vehicle automatic reactions or to warn the driver. Moreover, in contrast with state-of-the-art methods, our approach is monocular, neither requiring stereo nor optical flow information.

Real‐time AdaBoost cascade face tracker based on likelihood map and optical flow

The authors present a novel face tracking approach where optical flow information is incorporated into a modified version of the Viola Jones detection algorithm. In the original algorithm, detection is static, as information from previous frames is not considered. In addition, candidate windows have to pass all stages of the classification cascade, otherwise they are discarded as containing no face. In contrast, the proposed tracker preserves information about the number of classification stages passed by each window. Such information is used to build a likelihood map, which represents the probability of having a face located at that position. Tracking capabilities are provided by extrapolating the position of the likelihood map to the next frame by optical flow computation. The proposed algorithm works in real time on a standard laptop. The system is verified on the Boston Head Tracking Database, showing that the proposed algorithm outperforms the standard Viola Jones detector in terms of detection rate and stability of the output bounding box, as well as including the capability to deal with occlusions. The authors also evaluate two recently published face detectors based on convolutional networks and deformable part models with their algorithm showing a comparable accuracy at a fraction of the computation time.

Optic flow and self-motion information during real-world locomotion

Optic flow and self-motion information during real-world locomotion

Breaking camouflage and detecting targets require optic flow and image structure information.

Use of motion to break camouflage extends back to the Cambrian [In the Blink of an Eye: How Vision Sparked the Big Bang of Evolution (New York Basic Books, 2003)]. We investigated the ability to break camouflage and continue to see camouflaged targets after motion stops. This is crucial for the survival of hunting predators. With camouflage, visual targets and distracters cannot be distinguished using only static image structure (i.e., appearance). Motion generates another source of optical information, optic flow, which breaks camouflage and specifies target locations. Optic flow calibrates image structure with respect to spatial relations among targets and distracters, and calibrated image structure makes previously camouflaged targets perceptible in a temporally stable fashion after motion stops. We investigated this proposal using laboratory experiments and compared how many camouflaged targets were identified either with optic flow information alone or with combined optic flow and image structure information. Our results show that the combination of motion-generated optic flow and target-projected image structure information yielded efficient and stable perception of camouflaged targets.

Traffic flow detection and statistics via improved optical flow and connected region analysis

Moving vehicle detection plays an important role in intelligent transportation systems. One of the common methods used in moving vehicle detection is optical flow. However, conventional Horn–Schunck optical flow consumes too much time when calculating dense optical flows so that it cannot meet the real-time requirements. This paper proposes a novel improved Horn–Schunck optical flow algorithm based on inter-frame differential method. In our algorithm, optical flow field distribution is only calculated for pixels with larger gray values in the difference image, while for other pixels we applied the iterative smooth. The number of vehicles in the videos of traffic conditions is counted by setting the virtual loop and detecting optical flow information. To extract the moving vehicle as accurately as possible, we also propose a method to obtain moving vehicle minimum bounding rectangle based on the connected region analysis. Finally, we compare the improved optical flow with other four optical flow algorithms in moving vehicle extraction and vehicle flow detection, from which our method gives a much more accurate result.

Spatio-temporal feature using optical flow based distribution for violence detection

Violence detection in video surveillance is a challenging task in computer vision. In this paper, we propose a new feature that describes violence in video based on interest points, detected in both space and time domains, and optical flow information. The proposed feature estimates the bivariate distribution of the optical flow magnitude and orientation computed around spatio-temporal interest points (STIP). The estimation of the distribution is performed using the bivariate kernel density estimation method. Our proposed descriptor, named Distribution of Magnitude and Orientation of Local Interest Frame (DiMOLIF), is used to learn a SVM based binary classifier. Experiments are conducted on two well-known benchmark datasets for violence detection in both crowded and uncrowded scenes. Obtained results show that our proposed feature is efficient and outperforms the state of the art descriptors.

Histograms of Optical Flow Orientation and Magnitude and Entropy to Detect Anomalous Events in Videos

This paper presents an approach for detecting anomalous events in videos with crowds. The main goal is to recognize patterns that might lead to an anomalous event. An anomalous event might be characterized by the deviation from the normal or usual, but not necessarily in an undesirable manner, e.g., an anomalous event might just be different from normal but not a suspicious event from the surveillance point of view. One of the main challenges of detecting such events is the difficulty to create models due to their unpredictability and their dependency on the context of the scene. Based on these challenges, we present a model that uses general concepts, such as orientation, velocity, and entropy to capture anomalies. Using such a type of information, we can define models for different cases and environments. Assuming images captured from a single static camera, we propose a novel spatiotemporal feature descriptor, called histograms of optical flow orientation and magnitude and entropy , based on optical flow information. To determine the normality or abnormality of an event, the proposed model is composed of training and test steps. In the training, we learn the normal patterns. Then, during test, events are described and if they differ significantly from the normal patterns learned, they are considered as anomalous. The experimental results demonstrate that our model can handle different situations and is able to recognize anomalous events with success. We use the well-known UCSD and Subway data sets and introduce a new data set, namely, Badminton.

Embodied memory allows accurate and stable perception of hidden objects despite orientation change.

Rotating a scene in a frontoparallel plane (rolling) yields a change in orientation of constituent images. When using only information provided by static images to perceive a scene after orientation change, identification performance typically decreases (Rock & Heimer, 1957). However, rolling generates optic flow information that relates the discrete, static images (before and after the change) and forms an embodied memory that aids recognition. The embodied memory hypothesis predicts that upon detecting a continuous spatial transformation of image structure, or in other words, seeing the continuous rolling process and objects undergoing rolling observers should accurately perceive objects during and after motion. Thus, in this case, orientation change should not affect performance. We tested this hypothesis in three experiments and found that (a) using combined optic flow and image structure, participants identified locations of previously perceived but currently occluded targets with great accuracy and stability (Experiment 1); (b) using combined optic flow and image structure information, participants identified hidden targets equally well with or without 30° orientation changes (Experiment 2); and (c) when the rolling was unseen, identification of hidden targets after orientation change became worse (Experiment 3). Furthermore, when rolling was unseen, although target identification was better when participants were told about the orientation change than when they were not told, performance was still worse than when there was no orientation change. Therefore, combined optic flow and image structure information, not mere knowledge about the rolling, enables accurate and stable perception despite orientation change. (PsycINFO Database Record

Facial Expression Recognition From Depth Video With Patterns of Oriented Motion Flow

In this paper, we propose a novel feature representation method by a new feature descriptor, named patterns of oriented motion flow (POMF) from the optical flow information, to recognize the proper facial expression from the facial video. The POMF computes different directional motion information and encodes the directional flow information with enhanced local texture micro patterns. As it captures the spatial temporal changes of facial movements through optical flow and enables us to observe both local and global structures, it shows its robustness in recognizing facial information. Finally, the POMF histogram is used to train the expression model through the hidden Markov model (HMM). To train through the HMM, the objective sequences are produced by the generation of a codebook using the K-means clustering technique. The performance of the proposed method has been evaluated over the RGB and depth camera-based video. Experimental results demonstrate that the proposed POMF descriptor is more robust in extracting facial information and provides a higher classification rate than other existing promising methods.

Automatic Intruder Combat System: A way to Smart Border Surveillance

Security and safeguard of international borders have always been a dominant issue for every nation. A large part of a nation’s budget is provided to its defense system. Besides wars, illegal intrusion in terms of terrorism is a critical matter that causes severe harm to nation’s property. In India’s perspective, border patrolling by Border Security Forces (BSF) has already been practiced from a long time for surveillance. The patrolling parties are equipped with high-end surveillance equipments but yet an alternative to the ply of huge manpower and that too in harsh environmental conditions hasn’t been in existence. An automatic mechanism for smart surveillance and combat is proposed in this paper as a solution to the above-discussed problems. Smart surveillance requires automatic intrusion detection in the surveillance video, which is achieved by using optical flow information as motion features for intruder/human in the scene. The use of optical flow in the proposed smart surveillance makes it robust and more accurate. Use of a simple horizontal feature for fence detection makes system simple and faster to work in real-time. System is also designed to respond against the activities of intruders, of which auto combat is one kind of response.

A Height Estimation Approach for Terrain Following Flights from Monocular Vision.

In this paper, we present a monocular vision-based height estimation algorithm for terrain following flights. The impressive growth of Unmanned Aerial Vehicle (UAV) usage, notably in mapping applications, will soon require the creation of new technologies to enable these systems to better perceive their surroundings. Specifically, we chose to tackle the terrain following problem, as it is still unresolved for consumer available systems. Virtually every mapping aircraft carries a camera; therefore, we chose to exploit this in order to use presently available hardware to extract the height information toward performing terrain following flights. The proposed methodology consists of using optical flow to track features from videos obtained by the UAV, as well as its motion information to estimate the flying height. To determine if the height estimation is reliable, we trained a decision tree that takes the optical flow information as input and classifies whether the output is trustworthy or not. The classifier achieved accuracies of for positives and for negatives, while the height estimation algorithm presented good accuracy.

Flight-Test Evaluation of Navigation Information in Wide-Field Optical Flow

Wide-field optical flow information can benefit the navigation of small or micro unmanned aerial vehicles in GPS-degraded/denied environments, inspired by the study of insect/bird flights. This paper focuses on a flight-test evaluation of navigation information in wide-field optical flow, using flight data collected by a wide-angle camera installed on a fixed-wing unmanned aerial vehicle at altitudes ranging from 30 to 183 m above the ground. A comprehensive evaluation methodology is proposed for the comparison between optical flows computed from videos and the reference motion fields determined from conventional navigation sensors including GPS, a laser range finder, and inertial sensors. Seven sets of unmanned aerial vehicle flight data including a total of approximately 72,000 image frames are used for both full-flight temporal evaluation and representative interframe spatial evaluation of wide-field optical flow information for unmanned aerial vehicle navigation purposes. The evaluation results showed a good match between vision-derived optical flows and reference motion fields from conventional navigation sensors. Finally, one benchmark data set is shared at the authors’ website for open access from the research community.