IXMAS Dataset Research Articles

Enhanced human motion detection with hybrid RDA-WOA-based RNN and multiple hypothesis tracking for occlusion handling

Human motion detection in complex scenarios poses challenges due to occlusions. This paper presents an integrated approach for accurate human motion detections by combining Adapted Canny Edge detection as a preprocessing step, backbone-modified Mask R-CNN for precise segmentation, Hybrid RDA-WOA-based RNN as a classification, and a Multiple-hypothesis model for effective occlusion handling. Adapted Canny Edge detection is utilized as an initial preprocessing step to highlight significant edges in the input image. The resulting edge map enhances object boundaries and highlights structural features, simplifying subsequent processing steps. The improved image is then passed through backbone-modified Mask R-CNN for the pixel-level segmentation of humans. Backbone-modified Mask R-CNN along with IoU, Euclidean Distance, and Z-Score recognizes moving objects in complex scenes exactly. After recognizing moving objects, the optimized Hybrid RDA-WOA-based RNN classifies humans. To handle the self-occlusion, Multiple Hypothesis Tracking (MHT) is used. Real-world situations frequently include occlusions where humans can be partially or completely hidden by objects. The proposed approach integrates a Multiple-hypothesis model into the detection pipeline to address this challenge. Moreover, the proposed human motion detection approach includes an optimized Hybrid RDA-WOA-based RNN trained with 2D representations of 3D skeletal motion. The proposed work was evaluated using the IXMAS, KTH, Weizmann, NTU RGB + D, and UCF101 Datasets. It achieved an accuracy of 98% on the IXMAS, KTH, Weizmann, and UCF101 Datasets and 97.1% on the NTU RGB + D Dataset. The simulation results unveil the superiority of the proposed methodology over the existing approaches.

Micro-network Based Convolutional Neural Network with Integration of Multilayer Feature Fusion Strategy for Human Activity Recognition

Convolutional neural networks (CNN) have shown remarkable performance in enormous computer vision applications over the years, and many works have been done for human activity recognition (HAR) using CNN. However, most of deep learning-based architectures require large training data and plenty of computational resources. Therefore, we proposed a simple and efficient deep learning model for human activity recognition, which works on complex visual data and require less computational resources. In the proposed work, we designed a novel CNN architecture by stacking the repeated components called micro-networks to incorporate multiscale processing in the network layers. We have used a feature fusion strategy to pass the previous layer’s abstract complementary information to the next adjacent layer, believing that each layer encapsulates specific feature maps. Therefore, the hidden complementary information can potentially enhance the feature discrimination capacity of the network and help in learning the network. The proposed architecture is fully trained from scratch with stochastic gradient descent (SGD) optimizer at 0.05 initial learning rate and a softmax classifier is used for activity recognition. The merit of the proposed method over standard deep learning models is its computational efficiency in terms of learnable parameters and computational resources. The proposed model gives good performance on small as well as large size datasets. For authentication of the proposed method several extensive experiments are conducted on publically available datasets, namely UCF-101, HMDB-51, YouTube, and IXMAS datasets. The results have shown the outperformance of the proposed method over the existing state-of-the-art methods.

View-Independent Discriminant Analysis with Gradient Self-Similarities for Action Recognition under Multiple Views

In Multi-View Human Action Recognition, the actions are not of single view and hence to achieve an effective recognition performance under multi-view actions, there is a need of multi-view subclass discrimination analysis. Based on this inspiration, this paper proposed a novel action recognition framework based on the Subclass Discriminant Analysis (SDA), an extended version of Linear Discriminant Analysis (LDA). Further, a new key frames selection method is proposed based on Self-Similarity Matrix (SSM), called as Gradient SSM (GSSM). Once the key frames are selected, a composite feature set is extracted through three different set filters such as Gaussian Filter, Gabor filter and Wavelet Filter. Next, the SDA obtain an optimal feature subspace for every action under multiple Views. Finally the SVM algorithm classifies the action. The proposed framework is systematically evaluated on IXMAS dataset and NIXMAS dataset. Experimental results enumerate that our method outperforms the conventional techniques in terms of recognition accuracy.

Three-Stream Network With Bidirectional Self-Attention for Action Recognition in Extreme Low Resolution Videos

This letter presents a novel three-stream network for action recognition in extreme low resolution (LR) videos. In contrast to the existing networks, the new network uses the trajectory-spatial network, which is robust against visual distortion, instead of the pose information to complement the two-stream network. Also, the new three-stream network is combined with the inflated 3D ConvNet (I3D) model pre-trained on kinetics to produce more discriminative spatio-temporal features in blurred LR videos. Moreover, a bidirectional self-attention network is aggregated with the three-stream network to further manifest various temporal dependence among the spatio-temporal features. A new fusion strategy is devised as well to integrate the information from the three different modalities. Simulations show that the new architecture outperforms the main state-of-the-art extreme LR action recognition methods on the HMDB-51 and IXMAS datasets.

Multi-views Action Recognition on Deep Learning and K-SVD

In order to solve the problem of low action accuracy due to changes of view angles, this paper investigates this issue based on deep learning and K-SVD sparse algorithm. Firstly, the paper extracts the feature maps from the different views by convolutional neural networks (CNN) and long short term memory (LSTM), and the extracted feature maps are the multi-views high-level features with semantic information. Secondly, the paper uses the K-SVD sparse algorithm to get the dictionaries corresponding to each views, the dictionaries have very good sparse representations for the features of the action. Then the softmax classifier is used for classification and recognition. The results show that the accuracy are 89.22% and 91.4% on IXMAS datasets and WVU datasets respectively.

Investigating image stitching for action recognition

Action recognition is usually a central problem for many practical applications, such as video annotations, video surveillance and human computer interaction. Most action recognition approaches are based on localized spatio-temporal features that can vary significantly when the viewpoint changes. However, their performance rapidly drops when the viewpoints of the training and testing data are different. In this paper, we propose a transfer learning framework for view-invariant action recognition by the way of sharing image stitching feature among different views. Experimental results on multi-view action recognition IXMAS dataset demonstrate that our method produces remarkably good results and outperforms baseline methods.

Multi-view representation learning for multi-view action recognition

Although multiple methods have been proposed for human action recognition, the existing multi-view approaches cannot well discover meaningful relationship among multiple action categories from different views. To handle this problem, this paper proposes an multi-view learning approach for multi-view action recognition. First, the proposed method leverages the popular visual representation method, bag-of-visual-words (BoVW)/fisher vector (FV), to represent individual videos in each view. Second, the sparse coding algorithm is utilized to transfer the low-level features of various views into the discriminative and high-level semantics space. Third, we employ the multi-task learning (MTL) approach for joint action modeling and discovery of latent relationship among different action categories. The extensive experimental results on M2I and IXMAS datasets have demonstrated the effectiveness of our proposed approach. Moreover, the experiments further demonstrate that the discovered latent relationship can benefit multi-view model learning to augment the performance of action recognition.

A Novel Approach for Human Action Recognition from Silhouette Images

ABSTRACTIn this work, we propose a novel human action recognition (HAR) technique for human silhouette sequence based on spatio-temporal body parts movement (STBPM) and action-code classification (ACC). STBPM feature is designed to accumulate the signature of the activity of several body parts to accomplish any action. ACC is a code-based classifier for HAR, which needs no training and the codes of any action is created by analyzing the STBPM features. The proposed approach is view independent except the top view and scale invariant. The experimental results on publicly available Weizmann, MuHVAi, and IXMAS datasets clearly show that our proposed technique outperforms the related research works in terms of accuracy in the human action detection.

A coupled discriminative dictionary and transformation learning approach with applications to cross domain matching

Cross domain and cross-modal matching has many applications in the field of computer vision and pattern recognition. A few examples are heterogeneous face recognition, cross view action recognition, etc. This is a very challenging task since the data in two domains can differ significantly. In this work, we propose a coupled dictionary and transformation learning approach that models the relationship between the data in both domains. The approach learns a pair of transformation matrices that map the data in the two domains in such a manner that they share common sparse representations with respect to their own dictionaries in the transformed space. The dictionaries for the two domains are learnt in a coupled manner with an additional discriminative term to ensure improved recognition performance. The dictionaries and the transformation matrices are jointly updated in an iterative manner. The applicability of the proposed approach is illustrated by evaluating its performance on different challenging tasks: face recognition across pose, illumination and resolution, heterogeneous face recognition and cross view action recognition. Extensive experiments on five datasets namely, CMU-PIE, Multi-PIE, ChokePoint, HFB and IXMAS datasets and comparisons with several state-of-the-art approaches show the effectiveness of the proposed approach.

Cross-view action matching using a novel projective invariant on non-coplanar space-time points

Existing action matching methods from the geometric respect typically assume the collinearity or coplanarity for view invariance. These assumptions curb the application to uncontrolled action patterns. In this paper, a new projective invariant named characteristic number (CN) is used, which can be used to describe 3D non-coplanar points. For motion trajectories of actions, we propose the temporal CN (TCN) for individual joint point of a human body in temporal series. This view-invariant feature can characterize an action well with limited number of joints(a single one in our experiments). In addition to TCN, we are also able to define the spatial characteristic number (SCN) on several (five in our paper) joint points in the spatial domain for one frame. SCN works complementary to temporal features, when limited snapshots of an action are available. We validate both SCN and TCN on the widely used CMU Motion Capture Database (Mocap) database, KTH Multiview Football Dataset II and IXMAS dataset. The promising recognition results indicate the invariance to varying viewpoints compared with the state-of-the-art. The results on CMU and KTH database corrupted by noise show the robustness to noise.

Multi-View Human Action Recognition Using Wavelet Data Reduction and Multi-Class Classification

Human action recognition from video has several potential to apply in different real-life applications, but the most cases in this field suffer from the variation in viewpoint. Most of published methods in this area are considered the performance of each single camera, therefore the change in the viewpoints significantly decrease the recognition rate. In this paper, multiple views are considered together and a method has proposed to recognize human action depicted in multi-view image sequences. In the first step, the border of the human body's silhouette is extracted and distance signal is calculated. In the next step, the wavelet transform is applied to extract coefficients of single-view features, and then the extracted features are combined to compose multi-view features. Finally a hierarchical classifier using support vector machine and Naive Bayes classifiers is implemented to classify the actions. The average of overall action recognition accuracy for 12 actions using 5 different angles of views on the IXMAS dataset is 88.22. The results of experiments on the popular multi-view dataset have shown the proposed method achieves high and state-of-the-art success rates. In other word, combination of single-view extracted features from the wavelet approximation coefficients and composing the multi-view features can be used as the multi-view features. Further, the hierarchical classifier can be applied to recognize actions in multi-view human action recognition area.

Learning Discriminative Transferable Sparse Coding for Cross-View Action Recognition in Wireless Sensor Networks

Human action recognition in wireless sensor networks (WSN) is an attractive direction due to its wide applications. However, human actions captured from different sensor nodes in WSN show different views, and the performance of classifier tends to degrade sharply. In this paper, we focus on the issue of cross-view action recognition in WSN and propose a novel algorithm named discriminative transferable sparse coding (DTSC) to overcome the drawback. We learn the sparse representation with an explicit discriminative goal, making the proposed method suitable for recognition. Furthermore, we simultaneously learn the dictionaries from different sensor nodes such that the same actions from different sensor nodes have similar sparse representations. Our method is verified on the IXMAS datasets, and the experimental results demonstrate that our method achieves better results than that of previous methods on cross-view action recognition in WSN.

Cross-View Action Recognition Using Contextual Maximum Margin Clustering

Recently, maximum margin clustering (MMC) has been proposed for a cross-view action recognition. However, such a method neglects the temporal relationship between contiguous frames in the same action video. In this paper we propose a novel method called contextual maximum margin clustering (CMMC) to tackle cross-view action recognition. In CMMC, we add temporal regularization to give a high penalty when the contiguous frames are dissimilar. Thus, the CMMC not only achieves the goal of finding maximum margin hyperplanes, but also explicitly considers the temporal information among contiguous frames. Our method is verified on the IXMAS dataset and the experimental results demonstrate that our method can achieve better performance than the state-of-the-art methods.

View invariant action recognition using projective depth

In this paper, we investigate the concept of projective depth, demonstrate its application and significance in view-invariant action recognition. We show that projective depths are invariant to camera internal parameters and orientation, and hence can be used to identify similar motion of body-points from varying viewpoints. By representing the human body as a set of points, we decompose a body posture into a set of projective depths. The similarity between two actions is, therefore, measured by the motion of projective depths. We exhaustively investigate the different ways of extracting planes, which can be used to estimate the projective depths for use in action recognition including (i) ground plane, (ii) body-point triplets, (iii) planes in time, and (iv) planes extracted from mirror symmetry. We analyze these different techniques and analyze their efficacy in view-invariant action recognition. Experiments are performed on three categories of data including the CMU MoCap dataset, Kinect dataset, and IXMAS dataset. Results evaluated over semi-synthetic video data and real data confirm that our method can recognize actions, even when they have dynamic timeline maps, and the viewpoints and camera parameters are unknown and totally different.

SOFT-ASSIGNMENT RANDOM-FOREST WITH AN APPLICATION TO DISCRIMINATIVE REPRESENTATION OF HUMAN ACTIONS IN VIDEOS

The bag-of-features model is a distinctive and robust approach to detect human actions in videos. The discriminative power of this model relies heavily on the quantization of the video features into visual words. The quantization determines how well the visual words describe the human action. Random forests have proven to efficiently transform the features into distinctive visual words. A major disadvantage of the random forest is that it makes binary decisions on the feature values, and thus not taking into account uncertainties of the values. We propose a soft-assignment random forest, which is a generalization of the random forest, by substitution of the binary decisions inside the tree nodes by a sigmoid function. The slope of the sigmoid models the degree of uncertainty about a feature's value. The results demonstrate that the soft-assignment random forest improves significantly the action detection accuracy compared to the original random forest. The human actions that are hard to detect — because they involve interactions with or manipulations of some (typically small) item — are structurally improved. Most prominent improvements are reported for a person handing, throwing, dropping, hauling, taking, closing or opening some item. Improvements are achieved for the state-of-the-art on the IXMAS and UT-Interaction datasets by using the soft-assignment random forest.

Efficient descriptor tree growing for fast action recognition

Video and image classification based on Instance-to-Class (I2C) distance attracted many recent studies, due to the good generalization capabilities it provides for non-parametric classifiers. In this work we propose a method for action recognition. Our approach needs no intensive learning stage, and its classification performance is comparable to the state-of-the-art. A smart organization of training data allows the classifier to achieve reasonable computation times when working with large training databases. An efficient method for organizing training data in such a way is proposed. We perform thorough experiments on two popular action recognition datasets: the KTH dataset and the IXMAS dataset, and we study the influence of one of the key parameters of the method on classification performance.

A new invariant descriptor for action recognition based on spherical harmonics

The aim of this paper is to introduce a new descriptor for the spatio-temporal volume (STV). Human motion is completely represented by STV (action volume) which is constructed over successive frames by stacking human silhouettes in consecutive frames. Action volume comprehensively contains spatial and temporal information about an action. The main contribution of this paper is to propose a new affine invariant action volume descriptor based on a function of spherical harmonic coefficients. This means, it is invariant under rotation, non-uniform scaling and translation. In the 3D shape analysis literature, there have been a few attempts to use coefficients of spherical harmonics to describe a 3D shape. However, those descriptors are not affine invariant and they are only rotation invariant. In addition, the proposed approach employs a parametric form of spherical harmonics that handles genus zero surfaces regardless of whether they are stellar or not. Another contribution of this paper is the way that action volume is constructed. We applied the proposed descriptor to the KTH, Weizmann, IXMAS and Robust datasets and compared the performance of our algorithm to competing methods available in the literature. The results of our experiments show that our method has a comparable performance to the most successful and recent existing algorithms.

Action recognition from arbitrary views using 3D-key-pose set

Recovering three-dimensional (3D) human pose sequence from arbitrary view is very difficult, due to loss of depth information and self-occlusion. In this paper, view-independent 3D-key-pose set is selected from 3D action samples, for the purpose of representing and recognizing those same actions from a single or few cameras without any restriction of the relative orientations between cameras and subjects. First, 3D-key-pose set is selected from the 3D human joint sequences of 3D training action samples that are built from multiple viewpoints. Second, 3D key pose sequence, which matches best with the observation sequence, is selected from the 3D-key-pose set to represent the observation sequence of arbitrary view. 3D key pose sequence contains many discriminative view-independent key poses but cannot accurately describe pose of every frame in the observation sequence. Considering the above reasons, pose and dynamic of action are modeled respectively in this paper. Exemplar-based embedding and probability of unique key pose are applied to model pose property. Complementary dynamic feature is extracted to model these actions that share the same poses but have different dynamic features. Finally, these action models are fused to recognize observation sequence from a single or few cameras. Effectiveness of the proposed approach is demonstrated with experiments on IXMAS dataset.

Three-dimensional action recognition using volume integrals

This work proposes the volume integral (VI) as a new descriptor for three-dimensional action recognition. The descriptor transforms the actor’s volumetric information into a two-dimensional representation by projecting the voxel data to a set of planes that maximize the discrimination of actions. Our descriptor significantly reduces the amount of data of the three-dimensional representations yet preserves the most important information. As a consequence, the action recognition process is greatly speeded up while achieving very high success rates. The method proposed is therefore especially appropriate for applications in which limitations of computing power and space are significant aspects to consider, such as real-time applications or mobile devices. Additionally, the descriptor is sensitive to reflected actions, i.e., same actions performed with different limbs can be differentiated. This paper tests the VI using several Dimensionality Reduction techniques (namely PCA, 2D-PCA, LDA) and different Machine Learning approaches (namely Clustering, SVM and HMM) so as to determine the best combination of these for the action recognition task. Experiments conducted on the public IXMAS dataset show that the VI compares favorably with state-of-the-art descriptors both in terms of classification rates and computing times.

A probabilistic, discriminative and distributed system for the recognition of human actions from multiple views

This paper presents a distributed system for the recognition of human actions using views of the scene grabbed by different cameras. 2D frame descriptors are extracted for each available view to capture the variability in human motion. These descriptors are projected into a lower dimensional space and fed into a probabilistic classifier to output a posterior distribution of the action performed according to the descriptor computed at each camera. Classifier fusion algorithms are then used to merge the posterior distributions into a single distribution. The generated single posterior distribution is fed into a sequence classifier to make the final decision on the performed activity. The system can instantiate different algorithms for the different tasks, as the interfaces between modules are clearly defined. Results on the classification of the actions in the IXMAS dataset are reported. The accuracy of the proposed system is similar to state-of-the-art 3D methods, even though it uses only well-known 2D pattern recognition techniques and does not need to project the data into a 3D space or require camera calibration parameters.