Action Descriptors Research Articles

Real‐time action feature extraction via fast PCA‐Flow

SummaryAction recognition is a research hotspot in the field of Internet of Things (IoT). Currently, local pixel‐domain spatiotemporal feature extraction methods have reached the state‐of‐the‐art action recognition performance on many challenging datasets. However, the poor computational complexity of these approaches prevents them from scaling up to real‐time applications. For solving this problem, we present a novel real‐time video feature extraction technique by exploiting the fast PCA‐Flow algorithm. Firstly, we down‐sample video images in form of grid. Based on the down‐sampling images, PCA‐Flow algorithm is used to calculate optical flow among adjacent images. The PCA‐Flow matrices are then expanded to the original video image size by using efficient gCLSR super‐resolution method to keep the inherent geometric structure of the optical flow. Finally, we compute action descriptors based on original pixel frames and the enlarged PCA‐Flow images. The proposed approach is validated on three challenging datasets: UCF50, Hollywood2, and HMDB51. Experimental results indicate that the proposed method is more efficient in computation and can achieve competitive quality than the state‐of‐the‐art methods.

A visual cognizance based multi-resolution descriptor for human action recognition using key pose

Human activity recognition using videos sequences is a well-known phenomenon which has many real-life applications such as daily assistive living, security and surveillance, patient monitoring, robotics, and sports analysis. Recently, single or still images based action recognition is becoming very popular due to spatial cues present in an image and required less computation. Hence, a robust framework is constructed by computation of textural and spatial cues of still images at multi-resolution. A fuzzy inference model is used to select the single key image from action video sequences using maximum histogram distance between stacks of frames. To represent, these key pose images the textural traits at various orientations and scales are extracted using Gabor wavelet while shape traits are computed through a multilevel approach called Spatial Edge Distribution of Gradients (SEDGs). Finally, a hybrid model of action descriptor is developed using shape and textural evidence, which is known as Extended Multi-Resolution Features (EMRFs) model. The highest classification accuracy is achieved through SVM classifier on various human action datasets: Weizmann Action (100%), KTH (95.35%), Ballet (92.75%), and UCF YouTube (96.36%). The highest accuracy achieved on these datasets are compared with similar state-of-the-art approaches and EMRFs shows superior performance.

Improving multimodal action representation with joint motion history context

Automatic recognition of actions can be addressed by employing data from multiple sensors, such as RGB cameras, depth sensors or inertial measurement units. Recent studies show that multimodal representations of actions are effective in providing rich information about motion patterns. In this work, we propose a novel action descriptor, called Joint Motion History Context, which is based on depth and skeleton data. It improves action representation when used with previously introduced descriptors that are based on depth, skeleton and inertial data. A feature selection method is proposed as well, which ranks features on the basis of their inter-class discriminative power, while minimizing redundancy in the selected feature subset. Decision-level fusion, based on Support Vector Machines and Multilayer Perceptron is employed to effectively combine motion pattern information from multiple feature sets. Experimental results on two publicly available datasets, FFD and UTD-MHAD, demonstrated that the proposed methods outperform state-of-the-art algorithms.

Adolescents’ Implicit Theories of a Creative Person: A Longitudinal Investigation in Three Countries

Our research examined whether adolescents in the United States, China, and Japan differed in their conceptions of a creative person. Participants were American (n = 321), Chinese (n = 235), and Japanese (n = 393) adolescents in 7th and 8th grades who completed surveys at 3 time points. Using an open-ended questionnaire, adolescents were asked to list up to 10 attributes of a creative person. The responses were coded into 4 categories: action, emotion, characteristics, and self (vs. others). Results indicated that adolescents in the United States used more action and emotion descriptors (e.g., draws, happy) when conceptualizing a creative person, compared to Chinese and Japanese adolescents. In contrast, Chinese (vs. American and Japanese) adolescents were more likely to use descriptors about the characteristics of a person (e.g., hardworking, smart) in their conceptualizations of a creative person. Changes over time in adolescents’ conceptions of a creative person were evident, with the rates of such changes being uneven across countries. Findings provide evidence in support of the idea that adolescents’ implicit theories of creativity may be rooted in their cultural experiences.

Hierarchical Gaussian descriptor based on local pooling for action recognition

In this paper, we propose a new approach based on Gaussian descriptors for action recognition. We first develop a feature representation technique that encodes high-order statistics of local features in two levels, where single Gaussians are used to capture the distributions involved. To deal with the possible loss of information about the distribution of features caused by heterogeneous feature vectors when summarizing them, we use K-means clustering and Sparse Coding to construct some sets of feature vectors over which the summarization is performed. We then present two methods based on depth images and pose data for action recognition. In both methods, the proposed feature representation technique is applied to effectively obtain discriminative action descriptors. Experimental evaluation on the seven benchmark datasets, i.e., MSRAction3D, MSRGesture3D, DHA, SKIG, Florence, UTKinect, and HDM05, shows that our methods achieve very promising results on all the datasets.

HandSense: smart multimodal hand gesture recognition based on deep neural networks

Hand gesture recognition (HGR) is a promising enabler for human–computer interaction (HCI). Hand gestures are normally classified into multi-modal actions, including static gestures, fine-grained dynamic gestures, and coarse-grained dynamic gestures. Among them, the fine-grained action detection is limited under the small-scale image region condition. To solve this problem, we propose the HandSense, a new system for the multi-modal HGR based on a combined RGB and depth cameras to improve the fine-grained action descriptors as well as preserve the ability to perform general action recognition. First of all, two interconnected 3D convolutional neural network (3D-CNN) are employed to extract the spatial–temporal features from the RGB and depth images. Second, these spatial–temporal features are integrated into a fusion feature. Finally, the Support Vector Machine (SVM) is used to recognize different gestures based on the fusion feature. To validate the effectiveness of the HandSense, the extensive experiments are conducted on the public gesture dataset, namely the SKIG hand gesture dataset. In addition, the feasibility of the proposed system is also demonstrated by using a challenging multi-modal RGB-Depth hand gesture dataset.

Second-order Temporal Pooling for Action Recognition

Deep learning models for video-based action recognition usually generate features for short clips (consisting of a few frames); such clip-level features are aggregated to video-level representations by computing statistics on these features. Typically zero-th (max) or the first-order (average) statistics are used. In this paper, we explore the benefits of using second-order statistics.Specifically, we propose a novel end-to-end learnable feature aggregation scheme, dubbed temporal correlation pooling that generates an action descriptor for a video sequence by capturing the similarities between the temporal evolution of clip-level CNN features computed across the video. Such a descriptor, while being computationally cheap, also naturally encodes the co-activations of multiple CNN features, thereby providing a richer characterization of actions than their first-order counterparts. We also propose higher-order extensions of this scheme by computing correlations after embedding the CNN features in a reproducing kernel Hilbert space. We provide experiments on benchmark datasets such as HMDB-51 and UCF-101, fine-grained datasets such as MPII Cooking activities and JHMDB, as well as the recent Kinetics-600. Our results demonstrate the advantages of higher-order pooling schemes that when combined with hand-crafted features (as is standard practice) achieves state-of-the-art accuracy.

Vector space based augmented structural kinematic feature descriptor for human activity recognition in videos

A vector space based augmented structural kinematic (VSASK) feature descriptor is proposed for human activity recognition. An action descriptor is...

Kinematic Spline Curves: A temporal invariant descriptor for fast action recognition

Over the last few decades, action recognition applications have attracted the growing interest of researchers, especially with the advent of RGB-D cameras. These applications increasingly require fast processing. Therefore, it becomes important to include the computational latency in the evaluation criteria.In this paper, we propose a novel human action descriptor based on skeleton data provided by RGB-D cameras for fast action recognition. The descriptor is built by interpolating the kinematics of skeleton joints (position, velocity and acceleration) using a cubic spline algorithm. A skeleton normalization is done to alleviate anthropometric variability. To ensure rate invariance which is one of the most challenging issues in action recognition, a novel temporal normalization algorithm called Time Variable Replacement (TVR) is proposed. It is a change of variable of time by a variable that we call Normalized Action Time (NAT) varying in a fixed range and making the descriptors less sensitive to execution rate variability. To map time with NAT, increasing functions (called Time Variable Replacement Function (TVRF)) are used. Two different Time Variable Replacement Functions (TVRF) are proposed in this paper: the Normalized Accumulated kinetic Energy (NAE) of the skeleton and the Normalized Pose Motion Signal Energy (NPMSE) of the skeleton. The action recognition is carried out using a linear Support Vector Machine (SVM). Experimental results on five challenging benchmarks show the effectiveness of our approach in terms of recognition accuracy and computational latency.

Single- and two-person action recognition based on silhouette shape and optical point descriptors

In this paper, we present action descriptors that are capable of performing single- and two-person simultaneous action recognition. In order to exploit the shape information of action silhouettes, we detect junction points and geometric patterns at the silhouette boundary. The motion information is exploited by using optical flow points. We compute centroid distance signatures to construct the junction points and optical flow-based action descriptors. By taking advantage of the distinct poses, we extract key frames and construct geometric pattern action descriptor, which is based on histograms of the geometric patterns classes obtained by a distance-based classification method. In order to exploit the shape and motion information simultaneously, we follow the information fusion strategy and construct a joint action descriptor by combining geometric patterns and optical flow descriptors. We evaluate the performance of these descriptors on the two widely used action datasets, i.e., Weizmann dataset (single-person actions) and SBU Kinect interaction dataset, clean and noisy versions (two-person actions). The experimental outcomes demonstrate the ability of the individual descriptors to give satisfactory performance on average. It is found that the joint action descriptor shows the best performance among the proposed descriptors due to its high discriminative power and also outperforms state-of-the-art approaches.

Tensor-based linear dynamical systems for action recognition from 3D skeletons

Recent years have seen a growth in interest in skeleton-based human behavior recognition. Skeleton sequences can be expressed naturally as high-order tensor time series, and in this paper we report on the modeling and analysis of such time series using a linear dynamical system (LDS). Owing to their relative simplicity and efficiency, LDSs are the most common tool used in various disciplines for encoding spatiotemporal time series data. However, conventional LDSs process the latent and observed states at each frame of a video as a column vector, a representation that fails to take into account valuable structural information associated with human action. To correct this, we propose a tensor-based linear dynamical system (tLDS) for modeling tensor observations in time series and employ Tucker decomposition to estimate the parameters of the LDS model as action descriptors. In this manner, an action can be expressed as a subspace corresponding to a point on a Grassmann manifold on which classification can be performed using dictionary learning and sparse coding. Experiments using the MSR Action3D, UCF Kinect, and Northwestern-UCLA Multiview Action3D datasets demonstrate the excellent performance of our proposed method.

Region-sequence based six-stream CNN features for general and fine-grained human action recognition in videos

This paper addresses the problems of both general and also fine-grained human action recognition in video sequences. Compared with general human actions, fine-grained action information is more difficult to detect and occupies relatively small-scale image regions. Our work seeks to improve fine-grained action discrimination, while also retaining the ability to perform general action recognition. Our method first estimates human pose and human parts positions in video sequences by extending our recent work on human pose tracking, and crops different scaled patches to obtain richer action information in a variety of different scales of appearance and motion cues. We then utilize a Convolutional Neural Network (CNN) to process each such image patch. Instead of using the output one dimension feature from the full-connection layer, we utilize the outputs of the pooling layer of CNN structure, which contains more spatial information. Then the high dimension of the pooling features is reduced by encoding, to generate the final human action descriptors for classification. Our method reduces feature dimension while also effectively combining appearance and motion information in a unified framework. We have carried out empirical experiments using two publicly available human action datasets, comparing the human action recognition result of our algorithm against six recent state-of-the-art methods from the literature. The results suggest comparatively strong performance of our method.

Human action recognition based on kinematic similarity in real time.

Human action recognition using 3D pose data has gained a growing interest in the field of computer robotic interfaces and pattern recognition since the availability of hardware to capture human pose. In this paper, we propose a fast, simple, and powerful method of human action recognition based on human kinematic similarity. The key to this method is that the action descriptor consists of joints position, angular velocity and angular acceleration, which can meet the different individual sizes and eliminate the complex normalization. The angular parameters of joints within a short sliding time window (approximately 5 frames) around the current frame are used to express each pose frame of human action sequence. Moreover, three modified KNN (k-nearest-neighbors algorithm) classifiers are employed in our method: one for achieving the confidence of every frame in the training step, one for estimating the frame label of each descriptor, and one for classifying actions. Additional estimating of the frame’s time label makes it possible to address single input frames. This approach can be used on difficult, unsegmented sequences. The proposed method is efficient and can be run in real time. The research shows that many public datasets are irregularly segmented, and a simple method is provided to regularize the datasets. The approach is tested on some challenging datasets such as MSR-Action3D, MSRDailyActivity3D, and UTD-MHAD. The results indicate our method achieves a higher accuracy.

Spatio‐temporal multi‐scale motion descriptor from a spatially‐constrained decomposition for online action recognition

This study presents a spatio‐temporal motion descriptor that is computed from a spatially‐constrained decomposition and applied to online classification and recognition of human activities. The method starts by computing a dense optical flow without explicit spatial regularisation. Potential human actions are detected at each frame as spatially consistent moving regions of interest (RoIs). Each of these RoIs is then sequentially partitioned to obtain a spatial representation of small overlapped subregions with different sizes. Each of these region parts is characterised by a set of flow orientation histograms. A particular RoI is then described along the time by a set of recursively calculated statistics that collect information from the temporal history of orientation histograms, to form the action descriptor. At any time, the whole descriptor can be extracted and labelled by a previously trained support vector machine. The method was evaluated using three different public datasets: (i) the ViSOR dataset was used for global classification obtaining an average accuracy of 95% and for recognition in long sequences, achieving an average per‐frame accuracy of 92.3%. (ii) The KTH dataset was used for global classification and (iii) the UT‐datasets were used for recognition task, obtaining an average accuracy of 80% (frame rate).

Evaluation of regularized multi-task leaning algorithms for single/multi-view human action recognition

Regularized multi-task learning (MTL) algorithms have been exploited in the field of pattern recognition and computer vision gradually, which can fully excavate the relationships of different related tasks. Therefore, many dramatically favorable approaches based on regularized MTL have been proposed. In the past decades, although the promising results about human action recognition have been achieved, most of existing action recognition algorithms focus on action descriptors, single/multi-view and multi-modality action recognition, and few works are related with MTL, especial of lacking the systematic evaluation of existing MTL algorithms for human action recognition. Thus, in the paper, seven popular regularized MTL algorithms in which different actions are considered as different tasks, are systematically exploited on two public multi-view action datasets. In detail, dense trajectory features are firstly extracted for each view, and then the shared codebook are constructed for all views by k-means, and then each video is coded by the shared codebook. Moreover, according to different regularized MTL algorithms, all actions or part of actions are considered as related, and then these actions are set to different tasks in MTL. Further, the effectiveness of different number of training samples from different action views is also evaluated for MTL. Large scale experimental results show that: 1) Regularized MTL is very useful for action recognition which can dig the latent relationship among different actions; 2) Not of all human actions are related, if irrelative actions are put together in MTL, its performance will fall; 3) With the increase of the training samples from different views, the relationships about different actions can be fully exploited, and it promotes the accuracy improvement of action recognition.

Development and application of CatFACS: Are human cat adopters influenced by cat facial expressions?

The domestic cat (Felis silvestris catus) is quickly becoming the most popular animal companion in the world. The evolutionary processes that occur during domestication are known to have wide effects on the morphology, behaviour, cognition and communicative abilities of a species. Since facial expression is central to human communication, it is possible that cat facial expression has been subjected to selection during domestication. Standardised measurement techniques to study cat facial expression are, however, currently lacking. Here, as a first step to enable cat facial expression to be studied in an anatomically based and objective way, CatFACS (Cat Facial Action Coding System) was developed. Fifteen individual facial movements (Action Units), six miscellaneous movements (Action Descriptors) and seven Ear Action Descriptors were identified in the domestic cat. CatFACS was then applied to investigate the impact of cat facial expression on human preferences in an adoption shelter setting. Rehoming speed from cat shelters was used as a proxy for human selective pressure. The behaviour of 106 cats ready for adoption in three different shelters was recorded during a standardised encounter with an experimenter. This experimental setup aimed to mimic the first encounter of a cat with a potential adopter, i.e. an unfamiliar human. Each video was coded for proximity to the experimenter, body movements, tail movements and face movements. Cat facial movements were not related to rehoming speed, suggesting that cat facial expression may not have undergone significant selection. In contrast, rubbing frequency was positively related to rehoming speed. The findings suggest that humans are more influenced by overt prosocial behaviours than subtle facial expression in domestic cats.

Learning semantic context feature-tree for action recognition via nearest neighbor fusion

The spatio-temporal context learnt by the traditional methods for action recognition lacks the semantic meanings and temporal relationships. In order to deal with the drawbacks, we propose a novel semantic context feature-tree model to model the video clip for efficient human action recognition. The proposed method enforces spatio-temporal interest points (STIPs) within an irregular spatio-temporal volume to construct a semantic trees-structured relationship by nearest neighbor fusion. Specifically, we firstly extract STIPs, and moreover, utilize super-pixels to segment the motion image obtained by STIP detection. The points, which fall into super-pixel, are viewed as spatial semantic co-occurring features to represent one body part. Secondly, by patch matching, the point sets which are temporal nearest neighbors are merged into a new node of the next layer to describe the context of one moving part. After matching and associating, the sets of the frame indexes are renewed to group for the next fusion process until the conditions of the recursive process do not satisfy. Using KTH, UCF-YouTube and HOHA action datasets for human action recognition, our representation based on the learnt tree-structured features enhances the discriminative power of action descriptor, and obtains promising results.

Multiple Continuous Virtual Paths Based Cross-View Action Recognition

In this paper, we propose a novel method for cross-view action recognition via multiple continuous virtual paths which connect the source view and the target view. Each point on one virtual path is a virtual view which is obtained by a linear transformation of an action descriptor. All the virtual views are concatenated into an infinite-dimensional feature to characterize continuous changes from the source to the target view. To utilize these infinite-dimensional features directly, we propose a virtual view kernel (VVK) to compute the similarity between two infinite-dimensional features, which can be readily used to construct any kernelized classifiers. In addition, a constraint term is introduced to fully utilize the information contained in the unlabeled samples which are easier to obtain from the target view. The rationality behind the constraint is that any action video belongs to only one class. To further explore complementary visual information, we utilize multiple continuous virtual paths. The original source and target views are projected to different auxiliary source and target views using the random projection technique. Then we fuse all the VVKs generated from all pairs of auxiliary views. Our method is verified on the IXMAS and MuHAVi datasets, and the experimental results demonstrate that our method achieves better performance than the state-of-the-art methods.

Probability‐based method for boosting human action recognition using scene context

In this study, the authors investigate the possibility of boosting action recognition performance by exploiting the associated scene context. Towards this end, the authors model a scene as a mid-level `middle layer' in order to bridge action descriptors and action categories. This is achieved via a scene topic model, in which hybrid visual descriptors, including spatial-temporal action features and scene descriptors, are first extracted from a video sequence. Then, the authors learn a joint probability distribution between scene and action using a naive Bayes nearest neighbour algorithm, which is adopted to jointly infer the action categories online by combining off-the-shelf action recognition algorithms. The authors demonstrate the advantages of their approach by comparing it with state-of-the-art approaches using several action recognition benchmarks.

Distribution of action movements (DAM): a descriptor for human action recognition

Human action recognition fromskeletal data is an important and active area of research in which the state of the art has not yet achieved near-perfect accuracy on many wellknown datasets. In this paper, we introduce the Distribution of Action Movements Descriptor, a novel action descriptor based on the distribution of the directions of the motions of the joints between frames, over the set of all possible motions in the dataset. The descriptor is computed as a normalized histogram over a set of representative directions of the joints, which are in turn obtained via clustering. While the descriptor is global in the sense that it represents the overall distribution of movement directions of an action, it is able to partially retain its temporal structure by applying a windowing scheme. The descriptor, together with a standard classifier, outperforms several state-of-the-art techniques on many wellknown datasets.