Optical Flow Extraction Research Articles

An Overview of Behavioral Recognition

Human behavior recognition has become a popular research topic in the field of computer vision. With the introduction of deep learning and attention mechanisms, this field has been further promoted. However, issues such as dataset acquisition and preprocessing operations on multimodal datasets, modeling of long time information in videos, and fusion of temporal and spatial information still exist. In this paper, we first outline some video action recognition datasets and related preprocessing techniques, including frame extraction, optical flow extraction, and skeletal feature acquisition. Then, the relevant models are classified and parsed according to their characteristics and the types of input data modalities. In addition, we evaluate the performance of the models on several benchmark datasets to gain a deeper understanding of the model development process. Finally, we summarized the current challenges faced in the field of video behavior recognition, including model timeliness, data set subjectivity and effective fusion of multi-modal features, and proposed possible future improvement directions in order to provide more ideas and methods for subsequent research.

Gas–Liquid Two-Phase Flow Measurement Based on Optical Flow Method with Machine Learning Optimization Model

Gas–Liquid two-phase flows are a common flow in industrial production processes. Since these flows inherently consist of discrete phases, it is challenging to accurately measure the flow parameters. In this context, a novel approach is proposed that combines the pyramidal Lucas-Kanade (L–K) optical flow method with the Split Comparison (SC) model measurement method. In the proposed approach, videos of gas–liquid two-phase flows are captured using a camera, and optical flow data are acquired from the flow videos using the pyramid L–K optical flow detection method. To address the issue of data clutter in optical flow extraction, a dynamic median value screening method is introduced to optimize the corner point for optical flow calculations. Machine learning algorithms are employed for the prediction model, yielding high flow prediction accuracy in experimental tests. Results demonstrate that the gradient boosted regression (GBR) model is the most effective among the five preset models, and the optimized SC model significantly improves measurement accuracy compared to the GBR model, achieving an R2 value of 0.97, RMSE of 0.74 m3/h, MAE of 0.52 m3/h, and MAPE of 8.0%. This method offers a new approach for monitoring flows in industrial production processes such as oil and gas.

Adaptive Fractional-Order Multi-Scale Optimization TV-L1 Optical Flow Algorithm

We propose an adaptive fractional multi-scale optimization optical flow algorithm, which for the first time improves the over-smoothing of optical flow estimation under the total variation model from the perspective of global feature and local texture balance, and solves the problem that the convergence of fractional optical flow algorithms depends on the order parameter. Specifically, a fractional-order discrete L1-regularization Total Variational Optical Flow model is constructed. On this basis, the Ant Lion algorithm is innovatively used to realize the iterative calculation of the optical flow equation, and the fractional order is dynamically adjusted to obtain an adaptive optimization algorithm with strong search accuracy and high efficiency. In this paper, the flexibility of optical flow estimation in weak gradient texture scenes is increased, and the optical flow extraction rate of target features at multiple scales is greatly improved. We show excellent recognition performance and stability under the MPI_Sintel and Middlebury benchmarks.

Graph Convolutional Neural Networks for Micro-Expression Recognition—Fusion of Facial Action Units for Optical Flow Extraction

Graph Convolutional Neural Networks for Micro-Expression Recognition—Fusion of Facial Action Units for Optical Flow Extraction

Towards efficient video-based action recognition: context-aware memory attention network

Abstract Given the prevalence of surveillance cameras in our daily lives, human action recognition from videos holds significant practical applications. A persistent challenge in this field is to develop more efficient models capable of real-time recognition with high accuracy for widespread implementation. In this research paper, we introduce a novel human action recognition model named Context-Aware Memory Attention Network (CAMA-Net), which eliminates the need for optical flow extraction and 3D convolution which are computationally intensive. By removing these components, CAMA-Net achieves superior efficiency compared to many existing approaches in terms of computation efficiency. A pivotal component of CAMA-Net is the Context-Aware Memory Attention Module, an attention module that computes the relevance score between key-value pairs obtained from the 2D ResNet backbone. This process establishes correspondences between video frames. To validate our method, we conduct experiments on four well-known action recognition datasets: ActivityNet, Diving48, HMDB51 and UCF101. The experimental results convincingly demonstrate the effectiveness of our proposed model, surpassing the performance of existing 2D-CNN based baseline models. Article Highlights Recent human action recognition models are not yet ready for practical applications due to high computation needs. We propose a 2D CNN-based human action recognition method to reduce the computation load. The proposed method achieves competitive performance compared to most SOTA 2D CNN-based methods on public datasets.

Joint Measurement of Thermal Discomfort by Occupant Pose, Motion and Appearance in Indoor Surveillance Videos for Building Energy Saving

To accurately describe the thermal comfort of indoor occupant used to regulate heating, ventilation and air conditioning (HVAC) system is a key goal to reduce energy consumption in intelligent buildings. In this paper, we propose a noncontact measurement of occupant thermal discomfort behavior as an index of thermal comfort. The method takes the existing monitoring image data in the building as the input to infer the occupant thermal discomfort directly, which saves the cost because there is no need to install new sensors in the building and the occupant does not need to wear additional equipment. The framework combined three channels of body posture, motion and performance information to infer occupant thermal discomfort behavior, it consists of a human detection and crop module, a posture analysis module, an optical flow extraction module and a 3D convolutional neural network module. The three channels describe the actions from different perspectives, and the proposed method makes full use of the complementarity of the three modalities to identify the occupant’s thermal discomfort behaviors. Sixteen postures related to thermal discomfort were identified through a questionnaire, and 14,800 video clips containing these postures were collected for experimental evaluation. The results demonstrate the superior performances of our approach to the state-of-the-art techniques. The framework achieves noninvasive, cost-effective thermal comfort evaluation, and has potential value in improving energy efficiency of HVAC systems.

A deep learning algorithm-based visual strategy intervention study for children with autism spectrum disorders - extraction and detection of children's behavioral features.

Autism spectrum disorder is a group of neurodevelopmental disorders. The disease's etiology is unclear, and there is no specific drug treatment for the core symptoms of autism spectrum disease. The study aims to explore effective intervention methods for children with autism spectrum disorders.This paper proposes a visual strategy intervention method for children with autism spectrum disorders. This method combines feature extraction and abnormal behavior detection and can use a visual cue strategy to integrate children into social groups. Firstly, the spatial-temporal feature fusion structure is added to extract behavioral features from children, and the spatial information contained in MotionNet is fused with temporal features. Optical Flow Feature (OFF) subnetwork is added to the optical flow extraction feature network. Each layer feature is input to the OFF subnet to extract the time feature further. Then, a behavior detection method based on the sequential pool is proposed. This method combines attention mechanism and clustering pool to effectively describe human behavior dynamics in the long, redundant video under complex background. Finally, feature extraction and behavior detection experiments are carried out on SDUFall, Weizmann, and HMDB51 data sets.The model's accuracy is still slightly higher than others in that only the video Red-Green-Blue (RGB) frame is used as input. Compared with OFF, SDUFall can reach 88.64%, and HMDB51 can only reach 63.81%. In contrast, the proposed model can reach 72.09%, higher than others. The descriptor obtained the best result of 92.57%, which is 3.64%, 2.58%, and 1.73% higher than the other three comparison descriptors. The data show that the method presented here is effective and has advantages in detecting children's abnormal behavior.This method and visual intervention for children with autism spectrum disorders can help them to overcome social barriers.

SIGN LANGUAGE DETECTION USING ACTION RECOGNITION WITH PYTHON

Sign language detection plays a crucial role in bridging the communication gap between individuals with hearing impairments and the rest of the population. This study presents a novel approach for sign language detection using action recognition and LSTM deep learning models implemented in Python. The proposed system aims to accurately recognize and interpret sign language gestures in real-time. The research focuses on leveraging the power of deep learning techniques, specifically the Long Short-Term Memory (LSTM) architecture, to effectively capture temporal dependencies and classify sign language gestures. A comprehensive dataset comprising a wide range of sign language gestures is collected and preprocessed to train and evaluate the LSTM model. The pipeline for sign language detection consists of several stages, including video acquisition, preprocessing, feature extraction, and model training. In the pre-processing stage, the acquired video data is segmented into individual frames, and various image processing techniques are applied to enhance the quality and remove noise. Next, robust features are extracted from the preprocessed frames using techniques like optical flow or deep learning-based feature extraction methods. The LSTM model is then trained on the extracted features to learn the temporal dynamics of sign language gestures. Transfer learning is also explored to leverage pre-trained models on large-scale action recognition datasets. The trained model is evaluated using a comprehensive set of metrics, including accuracy, precision, recall, and F1-score, to assess its performance. Experimental results demonstrate the effectiveness of the proposed approach in accurately recognizing sign language gestures. The LSTM-based deep learning model achieves a high accuracy rate, showcasing its capability to handle the temporal nature of sign language. The system exhibits real-time performance, enabling it to be deployed in various applications, such as real-time interpretation tools or assistive devices for individuals with hearing impairments

Knowledge Distillation for Optical Flow-Based Video Super-resolution

Recently, deep learning-based super-resolution (SR) models have been used to improve SR performance by equipping preprocessing networks with baseline SR networks. In particular, in video SR, which creates a high-resolution (HR) image with multiple frames, optical flow extraction is accompanied by a preprocessing process. These preprocessing networks work effectively in terms of quality, but at the cost of increased network parameters, which increase the computational complexity and memory consumption for SR tasks with restricted resources. One well-known approach is the knowledge distillation (KD) method, which can transfer the original model’s knowledge to a lightweight model with less performance degradation. Moreover, KD may improve SR quality with reduced model parameters. In this study, we propose an effective KD method that can effectively reduce the original SR model parameters and even improve network performance. The experimental results demonstrated that our method achieved a better PSNR than the original state-of-the-art SR network despite having fewer parameters.

Hyperspectral Meets Optical Flow: Spectral Flow Extraction for Hyperspectral Image Classification.

Hyperspectral image (HSI) classification has always been recognised as a difficult task. It is therefore a research hotspot in remote sensing image processing and analysis, and a number of studies have been conducted to better extract spectral and spatial features. This study aimed to track the variation of the spectrum in hyperspectral images from a sequential data perspective to obtain more distinguishable features. Based on the characteristics of optical flow, this study introduces an optical flow technique for the extraction of spectral flow that denotes the spectral variation and implements a dense optical flow extraction method based on deep matching. Lastly, the extracted spectral flow are combined with the original spectral features and input into a commonly used support vector machine (SVM) classifier to complete the classification. Extensive classification experiments on three benchmark HSI test sets show that the classification accuracy obtained by the spectral flow extracted in this study (SpectralFlow) is higher than traditional spatial feature extraction methods, texture feature extraction methods, and the latest deep-learning-based methods. Furthermore, the proposed method can produce finer classification thematic maps, thereby demonstrating strong practical application potential.

TMF: Temporal Motion and Fusion for action recognition

Temporal motion information plays an important role in video understanding, human action recognition and other fields. Optical flow, which contains rich temporal motion information, has been widely used in many visual tasks and has achieved superior performance. However, the extraction of optical flow is time-consuming and laborious. In this paper, we propose a Temporal Motion and Fusion (TMF) module, including a motion extraction (ME) module and a temporal crossing fusion (TCF) module. The ME module can replace the traditional optical flow, establish the matching relationship between adjacent frames on the convoluted feature maps. And then extract simple and effective short-term motion information. TCF module crosses adjacent frames and fuse the information of nonadjacent video frames to realize long-term motion information modeling. Finally, the extracted motion information is fused with the appearance information captured by 2D convolution for final recognition. The experiment proved that with only a few additional parameters and calculation costs increased, our proposed lightweight model achieves state-of-the-art results on Something-Something-V1&V2 and Diving-48, and obtains competitive results on HMDB-51 and UCF-101 among the single models.

Integrated Multiscale Appearance Features and Motion Information Prediction Network for Anomaly Detection.

The rise of video-prediction algorithms has largely promoted the development of anomaly detection in video surveillance for smart cities and public security. However, most current methods relied on single-scale information to extract appearance (spatial) features and lacked motion (temporal) continuity between video frames. This can cause a loss of partial spatiotemporal information that has great potential to predict future frames, affecting the accuracy of abnormality detection. Thus, we propose a novel prediction network to improve the performance of anomaly detection. Due to the objects of various scales in each video, we use different receptive fields to extract detailed appearance features by the hybrid dilated convolution (HDC) module. Meanwhile, the deeper bidirectional convolutional long short-term memory (DB-ConvLSTM) module can remember the motion information between consecutive frames. Furthermore, we use RGB difference loss to replace optical flow loss as temporal constraint, which greatly reduces the time for optical flow extraction. Compared with the state-of-the-art methods in the anomaly-detection task, experiments prove that our method can more accurately detect abnormalities in various video surveillance scenes.

Fast Optical Flow Extraction from Compressed Video.

We propose the fast optical flow extractor, a filtering method that recovers artifact-free optical flow fields from HEVCcompressed video. To extract accurate optical flow fields, we form a regularized optimization problem that considers the smoothness of the solution and the pixelwise confidence weights of an artifactridden HEVC motion field. Solving such an optimization problem is slow, so we first convert the problem into a confidence-weighted filtering task. By leveraging the already-available HEVC motion parameters, we achieve a 100-fold speed-up in the running times compared to similar methods, while producing subpixel-accurate flow estimates. Je fast optical flow extractor is useful when video frames are already available in coded formats. Our method is not specific to a coder, and works with motion fields from video coders such as H.264/AVC and HEVC.

Real-time monitoring of driver drowsiness on mobile platforms using 3D neural networks

Driver drowsiness increases crash risk, leading to substantial road trauma each year. Drowsiness detection methods have received considerable attention, but few studies have investigated the implementation of a detection approach on a mobile phone. Phone applications reduce the need for specialised hardware and hence, enable a cost-effective roll-out of the technology across the driving population. While it has been shown that three-dimensional (3D) operations are more suitable for spatiotemporal feature learning, current methods for drowsiness detection commonly use frame-based, multi-step approaches. However, computationally expensive techniques that achieve superior results on action recognition benchmarks (e.g. 3D convolutions, optical flow extraction) create bottlenecks for real-time, safety-critical applications on mobile devices. Here, we show how depthwise separable 3D convolutions, combined with an early fusion of spatial and temporal information, can achieve a balance between high prediction accuracy and real-time inference requirements. In particular, increased accuracy is achieved when assessment requires motion information, for example, when sunglasses conceal the eyes. Further, a custom TensorFlow-based smartphone application shows the true impact of various approaches on inference times and demonstrates the effectiveness of real-time monitoring based on out-of-sample data to alert a drowsy driver. Our model is pre-trained on ImageNet and Kinetics and fine-tuned on a publicly available Driver Drowsiness Detection dataset. Fine-tuning on large naturalistic driving datasets could further improve accuracy to obtain robust in-vehicle performance. Overall, our research is a step towards practical deep learning applications, potentially preventing micro-sleeps and reducing road trauma.

Video Object Detection with Locally-Weighted Deformable Neighbors

Deep convolutional neural networks have achieved great success on various image recognition tasks. However, it is nontrivial to transfer the existing networks to video due to the fact that most of them are developed for static image. Frame-byframe processing is suboptimal because temporal information that is vital for video understanding is totally abandoned. Furthermore, frame-by-frame processing is slow and inefficient, which can hinder the practical usage. In this paper, we propose LWDN (Locally-Weighted Deformable Neighbors) for video object detection without utilizing time-consuming optical flow extraction networks. LWDN can latently align the high-level features between keyframes and keyframes or nonkeyframes. Inspired by (Zhu et al. 2017a) and (Hetang et al. 2017) who propose to aggregate features between keyframes and keyframes, we adopt brain-inspired memory mechanism to propagate and update the memory feature from keyframes to keyframes. We call this process Memory-Guided Propagation. With such a memory mechanism, the discriminative ability of features in keyframes and non-keyframes are both enhanced, which helps to improve the detection accuracy. Extensive experiments on VID dataset demonstrate that our method achieves superior performance in a speed and accuracy trade-off, i.e., 76.3% on the challenging VID dataset while maintaining 20fps in speed on Titan X GPU.

High-level Video Analytics PC Subsystem Using SoC With Heterogeneous Multicore Architecture

The high-level video analytics PC subsystem as PCIe add-in card is developed for surveillance, retail marketing tool, and various applications which require acceleration of computer vision analytics for real-time video. The PC subsystem has multicore architecture SoC, which comprises proposed computer vision engine (CVE) based on the filter, optical flow extraction, object detection accelerators, and memory sharing using the L3 cache. The optical flow accelerator has parallel and pipelined datapath with tight-coupled DMA control scheme to improve performance and less latency. The object detection accelerator has the capability of online learning from images so that it can give a robustness of tracking targets. L3 cache contributes to shortening memory access latency. Heterogeneous structure with the CPU and CVE accelerator achieves performance at 3500 GOPS, and power efficiency at 1871 GOPS/W, it is approximately 2–15 $\times$ higher performance and higher efficiency than other SoCs.

Event-Based Computation of Motion Flow on a Neuromorphic Analog Neural Platform.

Estimating the speed and direction of moving objects is a crucial component of agents behaving in a dynamic world. Biological organisms perform this task by means of the neural connections originating from their retinal ganglion cells. In artificial systems the optic flow is usually extracted by comparing activity of two or more frames captured with a vision sensor. Designing artificial motion flow detectors which are as fast, robust, and efficient as the ones found in biological systems is however a challenging task. Inspired by the architecture proposed by Barlow and Levick in 1965 to explain the spiking activity of the direction-selective ganglion cells in the rabbit's retina, we introduce an architecture for robust optical flow extraction with an analog neuromorphic multi-chip system. The task is performed by a feed-forward network of analog integrate-and-fire neurons whose inputs are provided by contrast-sensitive photoreceptors. Computation is supported by the precise time of spike emission, and the extraction of the optical flow is based on time lag in the activation of nearby retinal neurons. Mimicking ganglion cells our neuromorphic detectors encode the amplitude and the direction of the apparent visual motion in their output spiking pattern. Hereby we describe the architectural aspects, discuss its latency, scalability, and robustness properties and demonstrate that a network of mismatched delicate analog elements can reliably extract the optical flow from a simple visual scene. This work shows how precise time of spike emission used as a computational basis, biological inspiration, and neuromorphic systems can be used together for solving specific tasks.

Video Aesthetic Quality Assessment by Temporal Integration of Photo- and Motion-Based Features

This paper presents a new method for accessing the aesthetic quality of videos. It consists of two processes: aesthetic features construction and temporal integration. First, our method combines both photo-based and motion-based visual clues to extract the aesthetic features for each frame in a video. We introduce new motion-based features built from optical flow and salient region extraction, and show their effectiveness to enhance the estimation of aesthetic values. Then, a temporal-order-aware framework that integrates the frame-based features is presented to further improve the evaluation accuracy by taking the time-varying properties into consideration. The experimental results demonstrate that our approach can accomplish remarkable improvement for aesthetic quality assessment of videos.

Human V6: Functional Characterisation and Localisation

Human visual area V6, in the parieto-occipital sulcus, is thought to have an important role in the extraction of optic flow for the monitoring and guidance of self-motion (egomotion) because it responds differentially to egomotion-compatible optic flow when compared to: (a) coherent but egomotion-incompatible flow (Cardin & Smith, 2010), and (b) incoherent motion (Pitzalis et al., 2010). It is not clear, however, whether V6 responds more strongly to egomotion-incompatible global motion than to incoherent motion. This is relevant not only for determining the functional properties of V6, but also in order to choose optimal stimuli for localising V6 accurately with fMRI. Localisation with retinotopic mapping is difficult and there is a need for a simple, reliable method. We conducted an event-related 3T fMRI experiment in which participants viewed a display of dots which either: a) followed a time-varying optic flow trajectory in a single, egomotion-compatible (EC) display; b) formed an egomotion-incompatible (EI) 3×3 array of optic flow patches; or c) moved randomly (RM). Results from V6 show an ordering of response magnitudes: EC > EI > RM. Neighbouring areas V3A and V7 responded more strongly to EC than to RM, but about equally to EC and EI. Our results suggest that although V6 may have a general role in the extraction of global motion, in clear contrast to neighbouring motion areas it is especially concerned with encoding EC stimuli. They suggest two strategies for localising V6: (1) contrasting EC and EI; or (2) contrasting EC and RM, which is more sensitive but carries a risk of including voxels from neighbouring regions that also show a EC > RM preference.

Mobile Object Detection from an Embedded camera: how to minimize the latency time?

AbstractThis paper presents a global approach devoted to the detection of Mobile Objects from images acquired by a single camera embedded on a vehicle or a robot. The general method is presented involving at first optical flow extraction from tracked interest points, then clustering between static points belonging to the background and mobile ones extracted on different mobile objects, and then the tracking procedure performed for every detected mobile object. First results obtained from a sequential implementation, have shown that the latency time was too important. Here two improvements are proposed: first, a probabilistic map is built in order to detect points in more probable regions where a mobile object could be found. Then sequential and parallel architectures proposed to integrate point detection, point tracking and point clustering, are analyzed with respect to the latency time. Finally experimental results are presented and discussed.