Optical Flow Magnitude Research Articles

Enhanced visual SLAM for construction robots by efficient integration of dynamic object segmentation and scene semantics

With the increasing adoption of autonomous mobile robots in the construction industry, accurate localization and mapping in dynamic construction environments have become paramount. This is typically tackled via Simultaneous Localization and Mapping (SLAM) techniques. Primarily designed for static environments, traditional SLAM systems struggle to maintain robustness and accuracy in dynamic settings. To address this challenge, this study presents an enhanced visual SLAM system specifically tailored for dynamic construction environments. The proposed system, named vSLAM-Con, introduces an adaptive dynamic object segmentation method, utilizing an innovative AD-keyframes selection mechanism grounded on optical flow magnitude to diminish computational overhead while preserving competitive tracking accuracy. Additionally, a semantic-based feature update process is developed, leveraging scene understanding and continuous observation to augment the reliability of tracking features. This system's performance, evaluated on both an established public benchmark and a custom construction dataset, shows substantial improvements over the baseline and competitive results with the state-of-the-art algorithms. More importantly, it largely reduces the processing time compared to state-of-the-arts, demonstrating robust tracking performance even under highly dynamic conditions. The findings highlight the system's potential to contribute significantly to autonomous robotics in construction, offering more accurate navigation and interaction capabilities in complex, ever-changing environments.

Weakly-supervised anomaly detection with a Sub-Max strategy

We study weakly-supervised anomaly detection where only video-level “anomalous”/“normal” labels are available in training, while anomaly events should be temporally localized in testing. For this task, a commonly used framework is multiple instance learning (MIL), where clip instances are sampled from individual videos to form video-level bags. This sampling process arguably is a bottleneck of MIL. If too many instances are sampled, we not only encounter high computational overheads but also have many noisy instances in the bag. On the other hand, when too few instances are used, e.g., through enlarged grids, much background noise may be included in the anomaly instances. To resolve this dilemma, we propose a simple yet effective method named Sub-Max. In partitioned image regions, it identifies instances that are most probable candidates for anomaly events by selecting cuboids that have high optical flow magnitudes. We show that our method effectively brings down the computational cost of the baseline MIL and at the same time significantly filters out the influence of noise. Albeit simple, this strategy is shown to facilitate the learning of discriminative features and thus improve event classification and localization performance. For example, after annotating the event location ground truths of the UCF-Crime test set, we report very competitive accuracy compared with the state of the art on both frame-level and pixel-level metrics, corresponding to classification and localization, respectively.

Feature homogenization strategy using optical flow for vision-related simultaneous localization and mapping in a complex environment.

Vision-related state estimation usually extracts multiple feature points from images captured by the camera. In this paper, we propose a robust feature homogenization method for resolving the problem of feature clustering. The proposed method deduced the depth of feature points from optical flow magnitude, and the homogenization of feature points was acquired by adaptively enforcing the minimum distance between neighboring feature points. With the assistance of optical flow, the proposed method develops a preference for feature points with smaller depths in feature homogenization. Experimental results show that the proposed method improves the system's global consistency and tracking stability by using optical flow information.

Neural responses to natural visual motion are spatially selective across the visual field, with selectivity differing across brain areas and task.

It is well established that neural responses to visual stimuli are enhanced at select locations in the visual field. Although spatial selectivity and the effects of spatial attention are well understood for discrete tasks (e.g. visual cueing), little is known for naturalistic experience that involves continuous dynamic visual stimuli (e.g. driving). Here, we assess the strength of neural responses across the visual space during a kart‐race game. Given the varying relevance of visual location in this task, we hypothesized that the strength of neural responses to movement will vary across the visual field, and it would differ between active play and passive viewing. To test this, we measure the correlation strength of scalp‐evoked potentials with optical flow magnitude at individual locations on the screen. We find that neural responses are strongly correlated at task‐relevant locations in visual space, extending beyond the focus of overt attention. Although the driver's gaze is directed upon the heading direction at the centre of the screen, neural responses were robust at the peripheral areas (e.g. roads and surrounding buildings). Importantly, neural responses to visual movement are broadly distributed across the scalp, with visual spatial selectivity differing across electrode locations. Moreover, during active gameplay, neural responses are enhanced at select locations in the visual space. Conventionally, spatial selectivity of neural response has been interpreted as an attentional gain mechanism. In the present study, the data suggest that different brain areas focus attention on different portions of the visual field that are task‐relevant, beyond the focus of overt attention.

Optical Flow-Based Weighted Magnitude and Direction Histograms for the Detection of Abnormal Visual Events Using Combined Classifier

Movement information of persons is a very vital feature for abnormality detection in crowded scenes. In this paper, a new method for detection of crowd escape event in video surveillance system is proposed. The proposed method detects abnormalities based on crowd motion pattern, considering both crowd motion magnitude and direction. Motion features are described by weighted-oriented histogram of optical flow magnitude (WOHOFM) and weighted-oriented histogram of optical flow direction (WOHOFD), which describes local motion pattern. The proposed method uses semi-supervised learning approach using combined classifier (KNN and K-Means) framework to detect abnormalities in motion pattern. The authors validate the effectiveness of the proposed approach on publicly available UMN, PETS2009, and Avanue datasets consisting of events like gathering, splitting, and running. The technique reported here has been found to outperform the recent findings reported in the literature.

Human Activity Recognition Algorithm in Video Sequences Based on Integration of Magnitude and Orientation Information of Optical Flow

Human activity recognition from video sequences has emerged recently as pivotal research area due to its importance in a large number of applications such as real-time surveillance monitoring, healthcare, smart homes, security, behavior analysis, and many more. However, lots of challenges also exist such as intra-class variations, object occlusion, varying illumination condition, complex background, camera motion, etc. In this work, we introduce a novel feature descriptor based on the integration of magnitude and orientation information of optical flow and histogram of oriented gradients which gives an efficient and robust feature vector for the recognition of human activities for real-world environment. In the proposed approach first we computed magnitude and orientation of the optical flow separately then a local-oriented histogram of magnitude and orientation of motion flow vectors are computed using histogram of oriented gradients followed by linear combination feature fusion strategy. The resultant features are then processed by a multiclass Support Vector Machine (SVM) classifier for activity recognition. The experimental results are performed over different publically available benchmark video datasets such as UT interaction, CASIA, and HMDB51 datasets. The effectiveness of the proposed approach is evaluated in terms of six different performance parameters such as accuracy, precision, recall, specificity, [Formula: see text]-measure, and Matthew’s correlation coefficient (MCC). To show the significance of the proposed method, it is compared with the other state-of-the-art methods. The experimental result shows that the proposed method performs well in comparison to other state-of-the-art methods.

A Magnitude and Angle Combined Optical Flow Feature for Microexpression Spotting

Microexpression (ME) spotting is a crucial step for emotion analysis to detect people’s true emotions. However, the short duration, small motion amplitude, and limited sample number make accurate spotting and ME-locating challenging. To address these problems, we make two contributions in this article, including an ME dataset SDU2 and a spotting method. The dataset SDU2 contains hybrid expressions of 1602 video clips labeled by professional psychologists, covering six main categories of emotions with balanced distribution. Our ME spotting method is based on a magnitude- and angle-combined optical flow feature, exploiting the angle information, which has been overlooked by other spotting methods. In this method, we select four most discriminative regions of interest (ROIs) and determine the appearance of MEs using the proposed feature and a novel decision criterion based on the characteristic of the optical flow angle and magnitude. We have conducted experiments on SDU2 and CASME II datasets. The results demonstrate that our method achieves much better performance compared to other state-of-the-art methods in terms of spotting accuracy.

Video Anomaly Detection Using the Optimization-Enabled Deep Convolutional Neural Network

Abstract In video surveillance, automatic detection of the anomalies is the active research area in computer technology. Even though various video anomaly detection methods are introduced, detecting anomalous events, such as illegal actions and crimes, is a major challenging issue in video surveillance. Thus, an effective automatic video anomaly detection strategy based on the deep convolutional neural network (deep CNN) is developed in this research. Initially, the input video surveillance is passed into the spatiotemporal feature descriptor, named Histograms of Optical Flow Orientation and Magnitude. The features obtained from the descriptor provide the optical flow details with the aspect of normal patterns from the scene. These patterns are further subjected to the deep CNN, which is trained using the proposed dragonfly-rider optimization algorithm (DragROA) to assure the classification either as an anomalous activity or normal. The proposed DragROA is the combination of the standard dragonfly optimization algorithm and the standard rider optimization algorithm. The implementation of the proposed DragROA-based deep CNN is carried out using two datasets, namely anomaly detection dataset and UMN dataset; the performance is analyzed using the metrics, namely accuracy, sensitivity and specificity. From the analysis, it is depicted that the proposed method obtains the maximum accuracy, sensitivity and specificity of 0.9922, 0.9809 and 1, respectively, for the UCSD dataset.

Converting video classification problem to image classification with global descriptors and pre‐trained network

Motion history image (MHI) is a spatio‐temporal template that temporal motion information is collapsed into a single image where intensity is a function of recency of motion. Also, it consists of spatial information. Energy image (EI) based on the magnitude of optical flow is a temporal template that shows only temporal information of motion. Each video can be described in these templates. So, four new methods are introduced in this study. The first three methods are called basic methods. In method 1, each video splits into N groups of consecutive frames and MHI is calculated for each group. Transfer learning with fine‐tuning technique has been used for classifying these templates. EIs are used for classifying in method 2 similar to method 1. Fusing two streams of these templates is introduced as method 3. Finally, spatial information is added in method 4. Among these methods, method 4 outperforms others and it is called the proposed method. It achieves the recognition accuracy of 92.30 and 94.50% for UCF Sport and UCF‐11 action data sets, respectively. Also, the proposed method is compared with the state‐of‐the‐art approaches and the results show that it has the best performance.

Weakly-supervised action localization based on seed superpixels

In this paper, we present action localization based on weak supervision with seed superpixels. In order to benefit from the superpixel segmentation and to learn a priori knowledge we select the seed superpixels from the action and non-action areas of few video frames of an action sequence equally. We compute correlation, joint entropy and joint histogram as the features of the video frame superpixels based on the optical flow magnitudes and intensity information. An SVM is trained with the action and non-action seed superpixels features and is used to classify the video frame superpixels as action and non-action. The superpixels classified as action provide the action localization. The localized action superpixels are used to recognize the action class by the Dendrogram-SVM based on the already extracted features. We evaluate the performance of the proposed approach for action localization and recognition using UCF sports and UCF-101 actions datasets, which demonstrates that the seed superpixels provide effective action localization and in turn facilitates to recognize the action class.

DTFA-Net: Dynamic and Texture Features Fusion Attention Network for Face Antispoofing

For face recognition systems, liveness detection can effectively avoid illegal fraud and improve the safety of face recognition systems. Common face attacks include photo printing and video replay attacks. This paper studied the differences between photos, videos, and real faces in static texture and motion information and proposed a living detection structure based on feature fusion and attention mechanism, Dynamic and Texture Fusion Attention Network (DTFA-Net). We proposed a dynamic information fusion structure of an interchannel attention block to fuse the magnitude and direction of optical flow to extract facial motion features. In addition, for the face detection failure of HOG algorithm under complex illumination, we proposed an improved Gamma image preprocessing algorithm, which effectively improved the face detection ability. We conducted experiments on the CASIA-MFSD and Replay Attack Databases. According to experiments, the DTFA-Net proposed in this paper achieved 6.9% EER on CASIA and 2.2% HTER on Replay Attack that was comparable to other methods.

Abnormal Crowd Behavior Detection Using Motion Information Images and Convolutional Neural Networks

We introduce a novel method for abnormal crowd event detection in surveillance videos. Particularly, our work focuses on panic and escape behavior detection that may appear because of violent events and natural disasters. First, optical flow vectors are computed to generate a motion information image (MII) for each frame, and then MIIs are used to train a convolutional neural network (CNN) for abnormal crowd event detection. The proposed MII is a new formulation that provides a visual appearance of crowd motion. The proposed MIIs make the discrimination between normal and abnormal behaviors easier. The MII is mainly based on the optical flow magnitude, and angle difference computed between the optical flow vectors in consecutive frames. A CNN is employed to learn normal and abnormal crowd behaviors using MIIs. The MII generation, and the combination with a CNN is a new approach in the context of abnormal crowd behavior detection. Experiments are performed on commonly used datasets such as UMN and PETS2009. Evaluation indicates that our method achieves the best results.

UAV Flight and Landing Guidance System for Emergency Situations †.

Unmanned aerial vehicles (UAVs) with high mobility can perform various roles such as delivering goods, collecting information, recording videos and more. However, there are many elements in the city that disturb the flight of the UAVs, such as various obstacles and urban canyons which can cause a multi-path effect of GPS signals, which degrades the accuracy of GPS-based localization. In order to empower the safety of the UAVs flying in urban areas, UAVs should be guided to a safe area even in a GPS-denied or network-disconnected environment. Also, UAVs must be able to avoid obstacles while landing in an urban area. For this purpose, we present the UAV detour system for operating UAV in an urban area. The UAV detour system includes a highly reliable laser guidance system to guide the UAVs to a point where they can land, and optical flow magnitude map to avoid obstacles for a safe landing.

Magnitude-Orientation Stream network and depth information applied to activity recognition

The temporal component of videos provides an important clue for activity recognition, as a number of activities can be reliably recognized based on the motion information. In view of that, this work proposes a novel temporal stream for two-stream convolutional networks based on images computed from the optical flow magnitude and orientation, named Magnitude-Orientation Stream (MOS), to learn the motion in a better and richer manner. Our method applies simple non-linear transformations on the vertical and horizontal components of the optical flow to generate input images for the temporal stream. Moreover, we also employ depth information to use as a weighting scheme on the magnitude information to compensate the distance of the subjects performing the activity to the camera. Experimental results, carried on two well-known datasets (UCF101 and NTU), demonstrate that using our proposed temporal stream as input to existing neural network architectures can improve their performance for activity recognition. Results demonstrate that our temporal stream provides complementary information able to improve the classical two-stream methods, indicating the suitability of our approach to be used as a temporal video representation.

A classification method based on optical flow for violence detection

Violence detection is one of the substantial and challenging topics in intelligent video surveillance systems. As there is a growing demand on video surveillance systems with the capability of automatic violence detection, we focus on existing violence detection methods to improve them. In this paper, we introduce a new feature descriptor named Histogram of Optical flow Magnitude and Orientation (HOMO). First, the proposed method converts input frames to the grayscale format. Next, it computes the optical flow between two consequence frames. Then, the optical flow magnitude and orientation of each pixel in each frame are compared separately with its predecessor frame to obtain meaningful changes of magnitude and orientation. Subsequently, different threshold values are applied to the magnitude and orientation changes for obtaining six binary indicators. Finally, these binary indicators are analyzed to get the HOMO descriptor which is used to train a SVM classifier. The system has been implemented using MATLAB. To evaluate the proposed method, two benchmark datasets have been used. The comparison of HOMO and other descriptors on benchmark datasets demonstrates satisfactory performance.

The role of lateral optic flow cues in hawkmoth flight control.

Flying animals require sensory feedback on changes of their body position, as well as on their distance from nearby objects. The apparent image motion, or optic flow, which is generated as animals move through the air, can provide this information. Flight tunnel experiments have been crucial for our understanding of how insects use optic flow for flight control in confined spaces. However, previous work mainly focused on species from two insect orders: Hymenoptera and Diptera. We therefore set out to investigate whether the previously described control strategies to navigate enclosed environments are also used by insects with a different optical system, flight kinematics and phylogenetic background. We tested the role of lateral visual cues for forward flight control in the hummingbird hawkmoth Macroglossum stellatarum (Sphingidae, Lepidoptera), which possesses superposition compound eyes, and has the ability to hover in addition to its capacity for fast forward flight. Our results show that hawkmoths use a similar strategy for lateral position control to bees and flies in balancing the magnitude of translational optic flow perceived in both eyes. However, the influence of lateral optic flow on flight speed in hawkmoths differed from that in bees and flies. Moreover, hawkmoths showed individually attributable differences in position and speed control when the presented optic flow was unbalanced.

Autoencoder‐based abnormal activity detection using parallelepiped spatio‐temporal region

The spread of surveillance cameras has necessitated the monitoring of large quantities of surveillance video feeds. A manual monitoring system is near impossible due to the large man‐hour requirements. Recently, automatic abnormal activity detection has been an area of interest among researchers. A spatio‐temporal feature, histogram of optical flow orientation and magnitude (HOFM), has produced impressive ability in detecting abnormal activities. The authors propose a novel non‐uniform spatio‐temporal region resembling parallelepipeds, from which they extract the HOFM features. Autoencoders can be configured to detect abnormal patterns. The authors have used these abilities of the autoencoders to detect abnormalities in the HOFM features extracted from their novel spatio‐temporal regions of the video feeds. The autoencoders are trained on the HOFM features of the videos containing no abnormalities. The autoencoders are then fed with the HOFM features of the videos to be tested for abnormal activities, and these are detected based on the abilities of the autoencoders to reconstruct these features. The proposed method is tested on the standard abnormality detection datasets: UCSD Ped1, UCSD Ped2, Subway Entrance, Subway Exit, and UMN.

Gait representation and recognition from temporal co-occurrence of flow fields

This paper proposes a new gait representation that encodes the dynamics of a gait period through a 2D array of 17-bin histograms. Every histogram models the co-occurrence of optical flow states at every pixel of the normalized template that bounds the silhouette of a target subject. Five flow states (up, down, left, right, null) are considered. The first histogram bin counts the number of frames over the gait period in which the optical flow for the corresponding pixel is null. In turn, each of the remaining 16 bins represents a pair of flow states and counts the number of frames in which the optical flow vector has changed from one state to the other during the gait period. Experimental results show that this representation is significantly more discriminant than previous proposals that only consider the magnitude and instantaneous direction of optical flow, especially as the walking direction gets closer to the viewing direction, which is where state-of-the-art gait recognition methods yield the lowest performance. The dimensionality of that gait representation is reduced through principal component analysis. Finally, gait recognition is performed through supervised classification by means of support vector machines. Experimental results using the public CMU MoBo and AVAMVG datasets show that the proposed approach is advantageous over state-of-the-art gait representation methods.

Stacked sparse autoencoder and history of binary motion image for human activity recognition

The recognition of human actions in a video sequence still remains a challenging task in the computer vision community. Several techniques have been proposed until today such as silhouette detection, local space-time features and optical flow techniques. In this paper, a supervised way followed by an unsupervised learning using the principle of the auto-encoder is proposed to address the problem. We introduce a new foreground detection architecture based on information extracted from the Gaussian mixture model (GMM) incorporating with the uniform motion of Magnitude of Optical Flow (MOF). Thus, we use a fast dynamic frame skipping technique to avoid frames that contain irrelevant motion, making it possible to decrease the computational complexity of silhouette extraction. Furthermore a new technique of representations to construct an informative concept for human action recognition based on the superposition of human silhouettes is presented. We called this approach history of binary motion image (HBMI).Our method has been evaluated by a classification on the Ixmas, Weizmann, and KTH datasets, the Sparce Stacked Auto-encoder (SSAE), an instance of a deep learning strategy, is presented for efficient human activities detection and the Softmax (SMC) for the classification. The objective of this classifier in deep learning is the learning of function hierarchies with higher-level functions at lower-level functions of the hierarchy to provide an agile, robust and simple method. The results prove the efficiency of our proposed approach with respect to the irregularity in the performance of an action shape distortion, change of point of view as well as significant changes of scale.

Abnormal events’ detection in crowded scenes

In this paper, two new methods are developed in order to detect and track unexpected events in scenes. The process of detecting people may face some difficulties due to poor contrast, noise and the small size of the defects. For this purpose,the perfect knowledge of the geometry of these defects is an essential step in assessing the quality of detection. First, we collected statistical models of the element for each individual for time tracking of different people using the technique of Gaussian mixture model (GMM). Then we improved this method to detect and track the crowd(IGMM). Thereafter, we adopted two methods: the differential method of Lucas and Kanade(LK) and the method of optical flow estimation of Horn Schunck(HS) for optical flow representation. Then, we proposed a novel descriptor, named the Distribution of Magnitude of Optical Flow (DMOF) for anomalous events’ detection in the surveillance video. This descriptor represents an algorithm whose aim is to accelerate the action of abnormal events’ detection based on a local adjustment of the velocity field by manipulating the light intensity.