Histogram Of Optical Flow Research Articles

Multi-scale affined-HOF and dimension selection for view-unconstrained action recognition

In this paper an action recognition method that can adaptively handle the problems of variations in camera viewpoint is introduced. Our contribution is three-fold. First, a space-sampling algorithm based on affine transform in multiple scales is proposed to yield a series of different viewpoints from a single one. A histogram of dense optical flow is then extracted over each fixed-size patch for a given generated viewpoint as a local feature descriptor. Second, a dimension selection procedure is also proposed to retain only the dimensions that have distinctive information and discard the unnecessary ones in the feature vector space. Third, to adapt to a situation in which video data in multiple viewpoints are used for training; an extended method with a voting algorithm is also introduced to increase the recognition accuracy. By conducting experiments using both simulated and realistic datasets (http://www.aislab.org/index.php/en/mvar-datasets), the proposed method is validated. The method is found to be accurate and capable of maintaining its accuracy under a wide range of viewpoint changes. In addition, the method is less sensitive to variations in subject scale, subject position, action speed, partial occlusion, and background. The method is also validated by comparing with state-of-the-art view-invariant action recognition methods using well-known i3DPost and MuHAVi public datasets.

Fusing pyramid histogram of gradients and optical flow for hand gesture recognition

Human computer interaction systems based on hand gestures catch the eye of the research community for implementing natural communication between man and machines. However, different persons perform the same gestures differently in terms of velocity and motion scale. This poses a challenging issue in minimising the variations between different persons and maximises the coherence of the same gestures. In this paper, the original pyramid histogram of gradients in three orthogonal planes combining with the dense optical flow to create dynamic descriptor is explored in to discriminate features for recognition of hand gestures. The shape and motion features of images in a video sequence are captured to obtain the geometric and illumination invariant spatio-temporal feature descriptor for classification. A multiclass support vector machine classifier is used to recognise the hand gestures. The proposed method gives an excellent recognition rate and excels the existing approaches.

Fusing pyramid histogram of gradients and optical flow for hand gesture recognition

Fusing pyramid histogram of gradients and optical flow for hand gesture recognition

RETRACTED ARTICLE: A modified fuzzy histogram of optical flow for emotion classification

Human beings tend to express various emotions based on the activities. The criticality of facial expression has been recognized widely in the social interaction along with the social intellect. Human perception is subjective in nature and this makes the classification of emotion an extremely challenging problem. The mood, personality, age, and environment have a major influence on the perception of emotion. Facial expressions are a key to emotion; various studies devote on emotion classification based on facial expression. For the identification of these emotions, there is a mixture of models that make use of the feature representation that are gradient based, a mixture of various dynamic textures along with contextual information. In this work, histogram of optical flow (HOF) was used for the extraction of features and a neural network for bringing about an improvement to the accuracy of classification. With the availability of big data analytics, there has been a major increase in the power of computation in terms of analysing live video data, huge number of images and faster processing which is critical for emotion classification. The work has investigated efficacy of the flow of HOF and proposed a modified fuzzy histogram of optical flow. For choosing optimal rules in fuzzy system, heuristic method namely, charged system search was used. The results have proved that there has been a significant improvement to the methodology proposed.

Histogram of Fuzzy Local Spatio-Temporal Descriptors for Video Action Recognition

Feature extraction plays a vital role in visual action recognition. Many existing gradient-based feature extractors, including histogram of oriented gradients, histogram of optical flow, motion boundary histograms, and histogram of motion gradients, build histograms for representing different actions over the spatio-temporal domain in a video. However, these methods require to set the number of bins for information aggregation in advance. Varying numbers of bins usually lead to inherent uncertainty within the process of pixel voting with regard to the bins in the histogram. This article proposes a novel method to handle such uncertainty by fuzzifying these feature extractors. The proposed approach has two advantages: it better represents the ambiguous boundaries between the bins and, thus, the fuzziness of the spatio-temporal visual information entailed in videos; and the contribution of each pixel is flexibly controlled by a fuzziness parameter for various scenarios. The proposed family of fuzzy descriptors and a combination of them are evaluated on two publicly available datasets, demonstrating that the proposed approach outperforms the original counterparts and other state-of-the-art methods.

Human action recognition based on action relevance weighted encoding

Human action recognition in videos is one of the most important and active topics in computer vision, and building more discriminative video representation is crucial for action classification. In this paper, we propose a new video representation which not only captures temporal dynamics in a video but also takes account of the importance of different trajectories and video frames in action recognition. More specifically, first, we propose a new trajectory descriptor based on traditionally used local descriptors including histograms of oriented gradients, histograms of optical flow and motion boundary histogram, and it takes the correlation between trajectories and target action into consideration. Secondly, a new method to compute saliency map based on optical flow is introduced to highlight regions of foreground motion. Thirdly, to identify discriminating trajectories and frames in a video, trajectory action relevance and frame action relevance are defined and used as weights during encoding. Finally, two video representations which describe a video at different levels of abstraction are proposed, they are complementary to each other and combined to generate more expressive and discriminative representation. Experimental results on KTH, JHMDB and HMDB51 demonstrate the effectiveness and excellent performance of the proposed approach.

Action and movements recognition methods

The article describes the formulation of the problem of recognition of the movements of objects in a video sequence, the stages of its solution, the analysis of the basic methods of each of the stages. A wide range of applications and growing requirements on the quality of recognition determines the relevance of the study. The process of action recognition and detection begins with extracting useful features, from the input video sequence. Features are then processed through a classier to identify the action class (for example, running, walking, jumping, various gestures). The article describes the main feature descriptors, in the filter-based category: histogram of oriented gradients, cuboid descriptor, scale-invariant feature transform, gradient location-orientation histogram, local trinary patterns, and spatiotemporal patches, optical flow-based descriptors: histograms of optical flow, the motion boundary histogram, dense trajectory, convolutional neural network-based descriptors. Some algorithms require the extraction of primitive features and further refinement of the auxiliary features before they can be passed to the classifier. Examples of the use of specialized primitive features are methods based on silhouettes / contours and methods based on object tracking. There are methods for classifying extracted features, including the following: support vector machines, adaptive boost, artificial neural networks, convolutional neural networks. The key difficulties arising in solving the problem are considered. There are ways to compare various methods. One of the ways to draw comparisons is to quantitatively evaluate each approach on the same database with the same protocol. From simple KTH datasets and Weizmannnd to Carnegie Mellon University Crowded Videos dataset and Microsoft Research Action Group dataset to more complex video conditions and large-scale UCF101 and ActivityNet datasets. Existing approaches to recognition of motion in video sequences are analyzed. The article reveals characteristics, strengths and weaknesses of the various methods of detecting features and their classification. Leading methods that show the best results widely use convolutional neural networks. One of such methods is a spatio-temporal graph convolutional neural network for action recognition based on the object's skeleton. A method for further research and improvement was chosen.

Multi-Dimensional Traffic Congestion Detection Based on Fusion of Visual Features and Convolutional Neural Network

In intelligent transportation systems, there are many tasks that rely on the detection of road congestion, such as traffic signal scheduling and traffic accident detection. As traditional methods for traffic congestion detection are difficult to use, expensive, and may cause damage to the road surface, this paper presents a method for road congestion detection that is based on multidimensional visual features and a convolutional neural network (CNN). This method first detects the density of foreground objects by using a gray-level co-occurrence matrix; second, the speed of moving objects is detected by using the Lucas-Kanade optical flow with pyramid implementation. Third, a Gaussian mixture model is used to model the background, and the CNN is then used to accurately detect the final foreground from the candidate foregrounds. Finally, the proposed method performs road congestion detection in terms of a multidimensional feature space, including traffic density, traffic velocity, road occupancy, and traffic flow. Furthermore, we propose an information entropy method using a histogram of optical flow to enhance the accuracy and reliability of road congestion detection. Simulation results via quantitative and qualitative assessment indicate that the proposed method is able to significantly outperform the state-of-the-art road-traffic congestion detection methods due to the fusion of multidimensional features using the CNN.

Research on motion recognition algorithm based on bag-of-words model

In this study, we proposed a motion recognition algorithm based on optimized temporal-spatial features and integrating BOW model. Firstly, detect the key points of the input video and then set up multiple 3D patches in the domain space, enhancing the spatiotemporal characteristic of the key points. With that, randomly sample the key point sets and carry out the description by histogram of oriented gradient (HOG) and optical flow histogram (OFH), thereby obtaining the composite descriptor for features. The clustering algorithm will be then applied to set up a visual dictionary and represent the input video samples. In the final step, a classifier is trained and generated by utilizing multi-core SVM; thus, the motion recognition has been completed. The experimental results suggested that the recognition rate and robustness of the proposed algorithm can outperform most existing motion recognition algorithms.

3D Features for human action recognition with semi‐supervised learning

Human action recognition (HAR) is a very challenging task because of intra-class variations and complex backgrounds. Here, a motion history image (MHI)-based interest point refinement is proposed to remove the noisy interest points. Histogram of oriented gradient (HOG) and histogram of optical flow (HOF) techniques are extended from spatial to spatio-temporal domain to preserve the temporal information. These local features are used to build the trees for the random forest technique. During tree building, a semi-supervised learning is proposed for better splitting of data points at each node. For recognition of an action, mutual information is estimated for all the extracted interest points to each of the trained class by passing them through the random forest. The proposed method is evaluated on KTH, Weizmann, and UCF Sports standard datasets. The experimental results indicate that the proposed technique provides better performance compared to earlier reported techniques.

An Improved Micro-Expression Recognition Method Based on Necessary Morphological Patches

Micro-expression is a spontaneous emotional representation that is not controlled by logic. A micro-expression is both transitory (short duration) and subtle (small intensity), so it is difficult to detect in people. Micro-expression detection is widely used in the fields of psychological analysis, criminal justice and human-computer interaction. Additionally, like traditional facial expressions, micro-expressions also have local muscle movement. Psychologists have shown micro-expressions have necessary morphological patches (NMPs), which are triggered by emotion. Furthermore, the objective of this paper is to sort and filter these NMPs and extract features from NMPs to train classifiers to recognize micro-expressions. Firstly, we use the optical flow method to compare the on-set frame and the apex frame of the micro-expression sequences. By doing this, we could find facial active patches. Secondly, to find the NMPs of micro-expressions, this study calculates the local binary pattern from three orthogonal planes (LBP-TOP) operators and cascades them with optical flow histograms to form the fusion features of the active patches. Finally, a random forest feature selection (RFFS) algorithm is used to identify the NMPs and to characterize them via support vector machine (SVM) classifier. We evaluated the proposed method on two popular publicly available databases: CASME II and SMIC. Results show that NMPs are statistically determined and contribute to significant discriminant ability instead of holistic utilization of all facial regions.

A new spatial-temporal histograms of gradients descriptor and HOD-VLAD encoding for human action recognition

Automatic human action recognition is a core functionality of systems for video surveillance and human object interaction. In the whole recognition system, feature description and encoding represent two crucial key steps. In order to construct a powerful action recognition framework, it is important that the two steps must provide reliable performance. In this paper, we proposed a new human action feature descriptor which is called spatio-temporal histograms of gradients (SPHOG). SPHOG is based on the spatial and temporal derivation signal, which extracts the gradient changes between consecutive frames. Compared to the traditional descriptors histograms of optical flow, our proposed SPHOG costs less computation resource. In order to incorporate the distribution information of local descriptors into Vector of Locally Aggregated Descriptors (VLAD), which is a popular encoding approach for Bag-of-Feature representation, a Gaussian kernel is implanted to compute the weighted distance histograms of local descriptors. By doing this, the encoding schema for bag-of-feature (BOF) representation is more effective. We validated our proposed algorithm for human action recognition on three public available datasets KTH, UCF Sports and HMDB51. The evaluation experiment results indicate that the proposed descriptor and encoding method can improve the efficiency of human action recognition and the recognition accuracy.

Characterizing Human Behaviours Using Statistical Motion Descriptor

Identifying human behaviors is a challenging research problem due to the complexity and variation of appearances and postures, the variation of camera settings, and view angles. In this paper, we try to address the problem of human behavior identification by introducing a novel motion descriptor based on statistical features. The method first divide the video into N number of temporal segments. Then for each segment, we compute dense optical flow, which provides instantaneous velocity information for all the pixels. We then compute Histogram of Optical Flow (HOOF) weighted by the norm and quantized into 32 bins. We then compute statistical features from the obtained HOOF forming a descriptor vector of 192- dimensions. We then train a non-linear multi-class SVM that classify different human behaviors with the accuracy of 72.1%. We evaluate our method by using publicly available human action data set. Experimental results shows that our proposed method out performs state of the art methods.

Arabic sign language recognition using vision and hand tracking features with HMM

Sign language employs signs made by hands and facial expressions to convey meaning. Sign language recognition facilitates the communication between community and hearing-impaired people. This work proposes a recognition system for Arabic sign language using four types of features, namely modified Fourier transform, local binary pattern, histogram of oriented gradients, and a combination of histogram of oriented gradients and histogram of optical flow. These features are evaluated using hidden Markov model on two databases. The best performance is achieved with modified Fourier transform and histogram of oriented gradients features with 99.11% and 99.33% accuracies, respectively. In addition, two algorithms are proposed, one for segmenting sign video streams captured by Microsoft Kinect V2 into signs and the second for hand detection in video streams. The obtained results show that our algorithms are efficient in segmenting sign video streams and detecting hands in video streams.

Yapısal Benzerlik ve Optik Akış Histogramı ile Anomali Tespiti

In this paper, we present an anomaly detection and localization system for surveillance systems. A new feature descriptor is proposed. The spatio-temporal identifiers are obtained by using optical flow histogram and the structural similarity index from the videos that contain normal conditions. An artificial neural network, Selforganizing maps are used in modeling. The proposed system has been tested on the UCSD dataset.

Violence detection in surveillance video using low-level features.

It is very important to automatically detect violent behaviors in video surveillance scenarios, for instance, railway stations, gymnasiums and psychiatric centers. However, the previous detection methods usually extract descriptors around the spatiotemporal interesting points or extract statistic features in the motion regions, leading to limited abilities to effectively detect video-based violence activities. To address this issue, we propose a novel method to detect violence sequences. Firstly, the motion regions are segmented according to the distribution of optical flow fields. Secondly, in the motion regions, we propose to extract two kinds of low-level features to represent the appearance and dynamics for violent behaviors. The proposed low-level features are the Local Histogram of Oriented Gradient (LHOG) descriptor extracted from RGB images and the Local Histogram of Optical Flow (LHOF) descriptor extracted from optical flow images. Thirdly, the extracted features are coded using Bag of Words (BoW) model to eliminate redundant information and a specific-length vector is obtained for each video clip. At last, the video-level vectors are classified by Support Vector Machine (SVM). Experimental results on three challenging benchmark datasets demonstrate that the proposed detection approach is superior to the previous methods.

On an algorithm for human action recognition

Human action recognition which needs video processing in real time, requires large memory size and execution time. This work proposes a local maxima of difference image (LMDI) based interest point detection technique, random projection tree with overlapping split and modified voting score for human action recognition. In LMDI based interest point detection method, difference images are obtained using consecutive frame differencing technique and next, 3D peak detection is applied on the bunch of calculated difference images. Histogram of oriented gradients and histogram of optical flow as local features are extracted by defining a block of size 16 × 16 around each of the interest point. These local features are then indexed by random projection trees. Overlapping split is used during tree structuring to reduce failure probability. Hough voting technique is applied on testing video to compute highest similarity matching score with individual training classes. In addition to Hough voting score, the number of matched interest points of a single query video with each training class, is considered for recognition. The proposed method is evaluated on segmented UT-interaction dataset, J-HMDB dataset and UCF101 dataset. The experimental results indicate that the proposed technique provides better performance compared to earlier reported techniques.

Abnormal event detection based on cosparse reconstruction

A novel video abnormal event detection method based on cosparse reconstruction with local self-similarity constraint is proposed. For a given spatio-temporal patch which is represented by a feature vector using concatenated multi-scale histogram of optical flow, abnormal event detection is implemented by cosparse reconstruction with respect to an analysis dictionary learned from normal event set. To adapt to the diversity of normal events, feature space is partitioned into meaningful subspaces by clustering and cosparse sub-dictionary is learned from each cluster. Experimental results show that the proposed approach achieves competitive performance with the state-of-the-art methods.

A Real Time Video Summarization for YouTube Videos and Evaluation of Computational Algorithms for their Time and Storage Reduction

Theaim of creating video summarization is for gathering huge video data and makes important points to be highlighted. Focus of this view is to avail the complete content of data for any particular video can be easy and clarity of indexing video. In recent days people use internet to surf and watch videos, images, play games, shows and many more activities. But it is highly impossible to go through each and every show or video because it can consume more time and data. Instead, providing highlights of any such shows or game videos then it will be helpful to go through and decide about that video. Also we can provide trailer part of any news/movie videos which can yield to make judgement of those incidents. We propose an interesting principle for highlighting videos mostly they can be online. These online videos can be shortened and summarized the huge video into smaller parts. In order to achieve this we use feature extracting algorithms called the gradient and optical flow histograms (HOG & HOF). In order to enhance the efficiency of the method several optimization techniques are also being implemented.

Human action recognition by means of subtensor projections and dense trajectories

In last years, most human action recognition works have used dense trajectories features, to achieve state-of-the-art results. Histograms of Oriented Gradients (HOG), Histogram of Optical Flow (HOF) and Motion Boundary Histograms (MBH) features are extracted from regions and being tracked across the frames.The goal of this paper is to improve the performance obtained by means of Improved Dense Trajectories (IDTs), adding new features based on temporal templates. We construct these templates considering a video sequence as a third-order tensor and computing three different projections. We use several functions for projecting the fibers from the video sequences, and combined them by means of sum pooling.As a first contribution of our work, we present in detail the method based on tensor projections. First, we have assessed the results obtained using only template based action recognition. Next, in order to achieve state-of-art recognition rates, we have fused our features with those of IDTs.This is the second contribution of the article.Experiments on four different public datasets have shown that this technique improves IDTs performance and that the results outperform the ones obtained by most of the state-of-the-art techniques for action recognition.