Weizmann Dataset Research Articles

A Discriminative Framework for Action Recognition Using f-HOL Features

Inspired by the overwhelming success of Histogram of Oriented Gradients (HOG) features in many vision tasks, in this paper, we present an innovative compact feature descriptor called fuzzy Histogram of Oriented Lines (f-HOL) for action recognition, which is a distinct variant of the HOG feature descriptor. The intuitive idea of these features is based on the observation that the slide area of the human body skeleton can be viewed as a spatiotemporal 3D surface, when observing a certain action being performed in a video. The f-HOL descriptor possesses an immense competitive advantage, not only of being quite robust to small geometric transformations where the small translation and rotations make no large fluctuations in histogram values, but also of not being very sensitive under varying illumination conditions. The extracted features are then fed into a discriminative conditional model based on Latent-Dynamic Conditional random fields (LDCRFs) to learn to recognize actions from video frames. When tested on the benchmark Weizmann dataset, the proposed framework substantially supersedes most existing state-of-the-art approaches, achieving an overall recognition rate of 98.2%. Furthermore, due to its low computational demands, the framework is properly amenable for integration into real-time applications.

A Novel Approach for Human Action Recognition from Silhouette Images

ABSTRACTIn this work, we propose a novel human action recognition (HAR) technique for human silhouette sequence based on spatio-temporal body parts movement (STBPM) and action-code classification (ACC). STBPM feature is designed to accumulate the signature of the activity of several body parts to accomplish any action. ACC is a code-based classifier for HAR, which needs no training and the codes of any action is created by analyzing the STBPM features. The proposed approach is view independent except the top view and scale invariant. The experimental results on publicly available Weizmann, MuHVAi, and IXMAS datasets clearly show that our proposed technique outperforms the related research works in terms of accuracy in the human action detection.

Human action classification using adaptive key frame interval for feature extraction

Human action classification based on the adaptive key frame interval (AKFI) feature extraction is presented. Since human movement periods are different, the action intervals that contain the intensive and compact motion information are considered in this work. We specify AKFI by analyzing an amount of motion through time. The key frame is defined to be the local minimum interframe motion, which is computed by using frame differencing between consecutive frames. Once key frames are detected, the features within a segmented period are encoded by adaptive motion history image and key pose history image. The action representation consists of the local orientation histogram of the features during AKFI. The experimental results on Weizmann dataset, KTH dataset, and UT Interaction dataset demonstrate that the features can effectively classify action and can classify irregular cases of walking compared to other well-known algorithms.

Human Action Recognition Using Temporal Partitioning of activities and Maximum Average Correlation Height Filter

We proposed a method for Human action Recognition. It is based on the construction of a set of templates for each activity. Each template is constructed based on the Accumulated Motion Image of the Video. Each template contains where motion has occurred in the video. FFT Transform is applied to each template. A 3D Spatiotemporal Volume is generated for each class. A Single action Maximum average Correlation height Filter is generated for each class. The filter is applied to the test video and using the threshold the actions are classified. The experiments are conducted on Weizmann dataset.

Self-Organization of Spatio-Temporal Hierarchy via Learning of Dynamic Visual Image Patterns on Action Sequences.

It is well known that the visual cortex efficiently processes high-dimensional spatial information by using a hierarchical structure. Recently, computational models that were inspired by the spatial hierarchy of the visual cortex have shown remarkable performance in image recognition. Up to now, however, most biological and computational modeling studies have mainly focused on the spatial domain and do not discuss temporal domain processing of the visual cortex. Several studies on the visual cortex and other brain areas associated with motor control support that the brain also uses its hierarchical structure as a processing mechanism for temporal information. Based on the success of previous computational models using spatial hierarchy and temporal hierarchy observed in the brain, the current report introduces a novel neural network model for the recognition of dynamic visual image patterns based solely on the learning of exemplars. This model is characterized by the application of both spatial and temporal constraints on local neural activities, resulting in the self-organization of a spatio-temporal hierarchy necessary for the recognition of complex dynamic visual image patterns. The evaluation with the Weizmann dataset in recognition of a set of prototypical human movement patterns showed that the proposed model is significantly robust in recognizing dynamically occluded visual patterns compared to other baseline models. Furthermore, an evaluation test for the recognition of concatenated sequences of those prototypical movement patterns indicated that the model is endowed with a remarkable capability for the contextual recognition of long-range dynamic visual image patterns.

Using Discrete Cosine Transform Based Features for Human Action Recognition

Recognizing human action in complex video sequences has always been challenging for researchers due to articulated movements, occlusion, background clutter, and illumination variation. Human action recognition has wide range of applications in surveillance, human computer interaction, video indexing and video annotation. In this paper, a discrete cosine transform based features have been exploited for action recognition. First, motion history image is computed for a sequence of images and then blocked-based truncated discrete cosine transform is computed for motion history image. Finally, K-Nearest Neighbor (K-NN) classifier is used for classification. This technique exhibits promising results for KTH and Weizmann dataset. Moreover, the proposed model appears to be computationally efficient and immune to illumination variations; however, this model is prone to viewpoint variations

Human Activity Recognition using Binary Motion Image and Deep Learning

Abstract View based recognition methods use visual templates for recognition and hence do not extract complex features from the image. Instead they retain the entire raw image as a single feature in high dimensional space. These example images or templates are learnt under different poses and illumination conditions for recognition. With this in mind, we build on the idea of 2-D representation of action video sequence by combining the image sequences into a single image called Binary Motion Image (BMI) to perform human activity recognition. For classification, we employ Convolutional Neural Networks which inherently provide slight invariance to translational and rotational shifts, partial occlusions as well as background noise. We test our method on Weizmann dataset focusing on actions that look similar like run, walk, jump, side and skip actions. We also extended our method to 3-D depth maps using MSR Action3D dataset by extracting three BMI projections namely the front view, the side view and the top view. From the results obtained, we believe that BMI is sufficient for activity recognition and has shown to be invariant to speed of the action performed in addition to the aforementioned variations.

A human action recognition approach with a novel reduced feature set based on the natural domain knowledge of the human figure

Current video surveillance systems are not designed to raise an automatic alert in case of situations that put people lives at risk such as accidents, assaults and terrorism among others. This is due to the fact that these systems are not able to analyze huge amounts of video signals at higher processing speed where these signals come from cameras installed in the worldwide network. Faced with this situation, scientific communities are combining efforts to design algorithms and hardware to accelerate the processing of video signals. However, most of the methods proposed to date are too complex to be implemented in hardware at the place where the video camera is installed. In this paper, we report a significantly reduced novel feature set to design an analysis algorithm of significant less complexity which recognizes human actions from video sequences. The proposed method is based on the natural domain knowledge of the human figure such as proportions of the human body and foot positions. The analysis is characterized by working on sub-sequences of the entire video signals, processing a small fragment of the whole image, estimating the location of the region of interest, using simple operations (sum, subtraction, multiplications, divisions), extracting a reduced number of features per frame (6 features), and using a combination of four linear classifiers (one perceptron and three support vector machines) with a hierarchical structure. The method is evaluated on two of the datasets cited in the human action recognition literature, the Weizmann and the UIUC datasets. Results show that for the case of the Weizmann dataset, the correct classification rate (CCR) is 99.95% when the LOOCV Protocol is used and 98.38% for the case of Protocol 60–40, which is comparable or even higher than that of current state-of-the-art methods. Confusion matrices were also obtained for the UIUC dataset, where the obtained CCR is 100% for the case of the LOOCV Protocol and 99.35% when Protocol 60–40 is used. The experimental results are promising with much fewer features (between 85 and 113 times less features), compared with other methods, and the possibility of processing more than 200fps.

Optical flow-motion history image (OF-MHI) for action recognition

The motion history image (MHI) is a global spatiotemporal representation for video sequences. It is computationally very simple and efficient. It has been widely used for many real-time action recognition tasks. However, the conventional MHI assigns a fixed motion strength to each detected foreground point and then updates it with a small constant for the background point. Local body parts with different movement speeds and durations will then have the same intensity in the MHI. Similar actions may generate indistinguishable MHI patterns. In this paper, we propose a new motion history representation that incorporates both optical flow and a revised MHI. The motion strength of each pixel point is adaptively accumulated by the optical flow length at that location. It is then exponentially updated over time. It can better describe local movements of body parts in the global temporal template. The motion duration is implicitly given by the update rate for better description of various actions in the scene. For action classification, a set of training action samples are first collected and form the basis templates. An action sequence is then constructed as the linear combination of the basis templates. The coefficients of the combination give the feature vector. The Euclidean distance is finally used to evaluate the similarity between the feature vectors. Experimental results on the widely used KTH and Weizmann datasets have shown that the proposed scheme yields 100 % recognition rates on both test datasets with a fast processing rate of 47 fps on $$200\times 150$$ images.

A fuzzy logic-based system for the automation of human behavior recognition using machine vision in intelligent environments

The recent years have witnessed significant progress in the automation of human behavior recognition using machine vision in order to realize intelligent environments which are capable of detecting users' actions and gestures so that the needed services can be provided automatically and instantly for maximizing the user comfort and safety as well as minimizing energy. However, the majority of traditional human behavior machine vision-based recognition approaches rely on assumptions (such as known spatial locations and temporal segmentations) or computationally expensive approaches (such as sliding window search through a spatio-temporal volume). Hence, it is difficult for such methods to scale up and handle the high uncertainty levels and complexities available in real-world applications. This paper proposes a novel fuzzy machine vision-based framework for efficient humans' behavior recognition. A model-based feature set is utilized to extract visual feature cues including silhouette slices and movement speed from the human silhouette in video sequences which are analyzed as inputs by the proposed fuzzy system. We have employed fuzzy c-means clustering to learn the membership functions of the proposed fuzzy system. The behavior recognition was implemented via selecting the best candidate's behavior category with the highest output degree as the recognized behavior. We have successfully tested our system on the publicly available Weizmann human action dataset where our fuzzy-based system produced an average recognition accuracy of 94.03 %, which outperformed the traditional non-fuzzy systems based on hidden Markov models and other state-of-the-art approaches which were applied on the Weizmann dataset.

Projection transform on spatio-temporal context for action recognition

This paper discusses the task of human action recognition. This task is important to applications like video surveillance and video retrieval. Most of the existing local interest points based works on human action analysis, lost the information about spatio-temporal distribution of features and neglected the relationship between features and each defined actions. In this paper, through the analysis of feature distribution and their interactions over spatio-temporal domain, we propose a novel projection transform to take the two factors into account. A video sequence of human action in our perspective can be modeled by three types of features of spatio-temporal interest points: the global projection transform feature, the relative position distribution feature and the bag of visual words based feature. Then a new context-K-nearest-neighbor classifier is utilized to fuse them to form discriminative feature sets for action matching. In most of the case, our experiments have indicated that the novel method outperforms other previous published results on the Weizmann and KTH datasets.

Human Action Recognition Based on Normalized Interest Points and Super-Interest Points

Visual-based human action recognition is currently one of the most active research topics in computer vision. The feature representation directly has a crucial impact on the performance of the recognition. Feature representation based on bag-of-words is popular in current research, but the spatial and temporal relationship among these features is usually discarded. In order to solve this issue, a novel feature representation based on normalized interest points is proposed and utilized to recognize the human actions. The novel representation is called super-interest point. The novelty of the proposed feature is that the spatial-temporal correlation between the interest points and human body can be directly added to the representation without considering scale and location variance of the points by introducing normalized points clustering. The novelty concerns three tasks. First, to solve the diversity of human location and scale, interest points are normalized based on the normalization of the human region. Second, to obtain the spatial-temporal correlation among the interest points, the normalized points with similar spatial and temporal distance are constructed to a super-interest point by using three-dimensional clustering algorithm. Finally, by describing the appearance characteristic of the super-interest points and location relationship among the super-interest points, a new feature representation is gained. The proposed representation formation sets up the relationship among local features and human figure. Experiments on Weizmann, KTH, and UCF sports dataset demonstrate that the proposed feature is effective for human action recognition.

The Complex Action Recognition via the Correlated Topic Model

Human complex action recognition is an important research area of the action recognition. Among various obstacles to human complex action recognition, one of the most challenging is to deal with self-occlusion, where one body part occludes another one. This paper presents a new method of human complex action recognition, which is based on optical flow and correlated topic model (CTM). Firstly, the Markov random field was used to represent the occlusion relationship between human body parts in terms of an occlusion state variable. Secondly, the structure from motion (SFM) is used for reconstructing the missing data of point trajectories. Then, we can extract the key frame based on motion feature from optical flow and the ratios of the width and height are extracted by the human silhouette. Finally, we use the topic model of correlated topic model (CTM) to classify action. Experiments were performed on the KTH, Weizmann, and UIUC action dataset to test and evaluate the proposed method. The compared experiment results showed that the proposed method was more effective than compared methods.

A pyramidal deep learning architecture for human action recognition

This paper proposes a pyramidal deep learning architecture for human action recognition based on depth images from a 3D vision sensor. This method consists of three steps: 1) pre-processing depth image; 2) building a hidden deep neural network; 3) pattern recognition. A novel pyramidal stacked de-noising auto-encoder (pSDAE) is proposed to build a deep neural network so that its weights can be learnt layer by layer. A feed-forward neural network based on the deep learned weights is trained to classify each action pattern. Based on the experimental results from the Kinect dataset of human actions sampled in experiments, it is clear that the proposed approach outperforms the existing classical classify method. The robust experiment results on the Weizmann dataset show the good expansibility of the proposed method.

An enhanced method for human action recognition

This paper presents a fast and simple method for human action recognition. The proposed technique relies on detecting interest points using SIFT (scale invariant feature transform) from each frame of the video. A fine-tuning step is used here to limit the number of interesting points according to the amount of details. Then the popular approach Bag of Video Words is applied with a new normalization technique. This normalization technique remarkably improves the results. Finally a multi class linear Support Vector Machine (SVM) is utilized for classification. Experiments were conducted on the KTH and Weizmann datasets. The results demonstrate that our approach outperforms most existing methods, achieving accuracy of 97.89% for KTH and 96.66% for Weizmann.

Learning Discriminative Key Poses for Action Recognition

In this paper, we present a new approach for human action recognition based on key-pose selection and representation. Poses in video frames are described by the proposed extensive pyramidal features (EPFs), which include the Gabor, Gaussian, and wavelet pyramids. These features are able to encode the orientation, intensity, and contour information and therefore provide an informative representation of human poses. Due to the fact that not all poses in a sequence are discriminative and representative, we further utilize the AdaBoost algorithm to learn a subset of discriminative poses. Given the boosted poses for each video sequence, a new classifier named weighted local naive Bayes nearest neighbor is proposed for the final action classification, which is demonstrated to be more accurate and robust than other classifiers, e.g., support vector machine (SVM) and naive Bayes nearest neighbor. The proposed method is systematically evaluated on the KTH data set, the Weizmann data set, the multiview IXMAS data set, and the challenging HMDB51 data set. Experimental results manifest that our method outperforms the state-of-the-art techniques in terms of recognition rate.

Position and locality constrained soft coding for human action recognition

Although the traditional bag-of-words model has shown promising results for human action recognition, in the feature coding phase, the ambiguous features from different body parts are still difficult to distinguish. Furthermore, it also suffers from serious representation error. We propose an innovative coding strategy called position and locality constrained soft coding (PLSC) to overcome these limitations. PLSC uses the feature position in a human oriented region of interest (ROI) to distinguish the ambiguous features. We first construct a subdictionary for each feature by selecting the bases from their spatial neighbor in human ROI. Then, a modified soft coding with locality constraint is adopted to alleviate the quantization error and preserve the manifold structure of features. This novel coding algorithm increases both the representation accuracy and discriminative power with low computational cost. The human action recognition experimental results on KTH, Weizmann, and UCF sports datasets show that PLSC can achieve a better performance than previous competing feature coding methods.

Local Spatiotemporal Coding and Sparse Representation Based Human Action Recognition

To handle with the limitation of bag-of-features (BoF) model which ignores spatial and temporal relationships of local features in human action recognition in video, a Local Spatiotemporal Coding (LSC) is proposed. Rather than the exiting methods only uses the feature appearance information for coding, LSC encodes feature appearance and spatiotemporal positions information simultaneously with vector quantization (VQ). It can directly models the spatiotemporal relationships of local features in space time volume (STV). In implement, the local features are projected into sub-space-time-volume (sub-STV), and encoded with LSC. In addition a multi-level LSC is also provided. Then a group of sub-STV descriptors obtained from videos with multi-level LSC and Avg-pooling are used for action video classification. A sparse representation based classification method is adopted to classify action videos upon these sub-STV descriptors. The experimental results on KTH, Weizmann, and UCF sports datasets show that our method achieves better performance than the previous local spatiotemporal features based human action recognition methods.

Learning action patterns in difference images for efficient action recognition

A new framework is presented for single-person oriented action recognition. This framework does not require detection/location of bounding boxes of human body nor motion estimation in each frame. The novel descriptor/pattern for action representation is learned with local temporal self-similarities (LTSSs) derived directly from difference images. The bag-of-words framework is then employed for action classification taking advantages of these descriptors. We investigated the effectiveness of the framework on two public human action datasets: the Weizmann dataset and the KTH dataset. In the Weizmann dataset, the proposed framework achieves a performance of 95.6% in the recognition rate and that of 91.1% in the KTH dataset, both of which are competitive with those of state-of-the-art approaches, but it has a high potential to achieve a faster execution performance.

A new evaluation measure for color image segmentation based on genetic programming approach

One of the greatest challenges while working on image segmentation algorithms is a comprehensive measure to evaluate their accuracy. Although there are some measures for doing this task, but they can consider only one aspect of segmentation in evaluation process. The performance of evaluation measures can be improved using a combination of single measures. However, combination of single measures does not always lead to an appropriate criterion. Besides its effectiveness, the efficiency of the new measure should be considered. In this paper, a new and combined evaluation measure based on genetic programming (GP) has been sought. Because of the nature of evolutionary approaches, the proposed approach allows nonlinear and linear combinations of other single evaluation measures and can search within many and different combinations of basic operators to find a good enough one. We have also proposed a new fitness function to make GP enable to search within search space effectively and efficiently. To test the method, Berkeley and Weizmann datasets besides several different experiments have been used. Experimental results demonstrate that the GP based approach is suitable for effective combination of single evaluation measures.