Action Recognition Datasets Research Articles

Learning from Temporal Spatial Cubism for Cross-Dataset Skeleton-based Action Recognition

Rapid progress and superior performance have been achieved for skeleton-based action recognition recently. In this article, we investigate this problem under a cross-dataset setting, which is a new, pragmatic, and challenging task in real-world scenarios. Following the unsupervised domain adaptation (UDA) paradigm, the action labels are only available on a source dataset, but unavailable on a target dataset in the training stage. Different from the conventional adversarial learning-based approaches for UDA, we utilize a self-supervision scheme to reduce the domain shift between two skeleton-based action datasets. Our inspiration is drawn from Cubism, an art genre from the early 20th century, which breaks and reassembles the objects to convey a greater context. By segmenting and permuting temporal segments or human body parts, we design two self-supervised learning classification tasks to explore the temporal and spatial dependency of a skeleton-based action and improve the generalization ability of the model. We conduct experiments on six datasets for skeleton-based action recognition, including three large-scale datasets (NTU RGB+D, PKU-MMD, and Kinetics) where new cross-dataset settings and benchmarks are established. Extensive results demonstrate that our method outperforms state-of-the-art approaches. The source codes of our model and all the compared methods are available at https://github.com/shanice-l/st-cubism.

Efficient Augmentation of Human Action Recognition Datasets with Warped Windows

In this paper, an approach to augment action recognition time series datasets, devoted to improving the accuracy of deep learning classifiers, is proposed. In the introduced method, two operators are sequentially introduced that perform linear and nonlinear modifications in the time scale of the input time series. The resulting data samples contribute to the variability within classes and allow a deep learning-based classifier to better capture their boundaries, leading to a significant improvement in the classification accuracy. The extensive experiments performed on eight publicly available action recognition datasets using the popular Bidirectional Long Short-Term Memory (BiLSTM) classifier reveal the superiority of the proposed algorithm over related approaches.

FLAAP: An Open Human Activity Recognition (HAR) Dataset for Learning and Finding the Associated Activity Patterns

A significant quantity of research work has been completed to recognize human activities. The majority of the proposed learning algorithms have treated the activity data as the role of fuel in vehicles. The capacity of the learning algorithms to recognize the activity patterns can be improved by the adequate availability of activity data. In this paper, we introduced the FLAAP (Finding and Learning the Associated Activity Patterns) activity dataset and data acquired by using the smartphone (accelerometer and gyroscope) sensors placed at the waist of the subjects while performing the activities. This dataset contains the record of ten activities performed by eight distinct subjects. Between February 1st and May 31st, 2022, millions of raw sensor activity data samples were captured constantly at 100Hz sampling rates. The Human Activity Recognition (HAR) datasets, which keep a record of such activities and report associations in activity patterns (mostly used for recognizing the Activities of Daily Living (ADL)), were lacking. This paucity is addressed by the FLAAP dataset which can be useful in finding the associated patterns in ADL. The obtained experimental findings demonstrate that the learning algorithm Random Forest (RF), which was used, has recognized the activities with around 77.22% accuracy. The applied RF learning algorithm on the FLAAP dataset provides the research gap for the researchers in developing more delicate learning models for enhancing recognition rates. Furthermore, the research community could be particularly interested in examining the learning performance of algorithms while using various data pre-processing techniques, transferring knowledge to target domains, and other techniques.

Modeling Physical Activity in Children by Combining Raw Hip-Worn Accelerometry, 2D Pose Estimation, and Direct Observation

This work presents an inductive search of physical activity intensity features for developing computer vision data sets used in automatic physical activity observation systems. An online survey was conducted to calibrate the ground truth definitions in a data set of video synchronized with accelerometers. Experts from the research field were asked to classify 24 short video samples of children&#x2019;s physical activity relative to three metabolic equivalence of task units &#x2013; the presumed threshold of moderate to vigorous physical activity. Leave-one-out disagreement analysis was applied until moderate agreement was achieved (12 respondents remain, Light&#x2019;s <inline-formula> <tex-math notation="LaTeX">$\kappa $ </tex-math></inline-formula> &#x003D; 0.62). The predictive power of features from hip-worn ActiGraph wGT3X-BT 30Hz raw accelerations and several freely available 2D pose estimation models are explored by cut-point analyses and logistic regression while using several approaches to account for uncertainty. Features from the acceleration- and video pose estimation domains are combined in correlation analyses in a twelve-hour data set representing relatively unstructured behavior from 24 children ages 8&#x2013;13 filmed in four different locations. Results indicate that changes in the hip angles of pose-estimated kinematic skeletons across 10 fps video frames can supplement or possibly substitute accelerometers for estimating physical activity intensity in uncrowded indoor scenes. When taking such an approach to labeling physical activity recognition data sets, good joint tracking capacity of the pose estimation method should increase the robustness of the hip angle features.

Meta-Learning Paradigm and CosAttn for Streamer Action Recognition in Live Video

As an emerging field of network content production, live video has been in the vacuum zone of cyberspace governance for a long time. Streamer action recognition is conducive to the supervision of live video content. In view of the diversity and imbalance of streamer actions, it is attractive to introduce few-shot learning to realize streamer action recognition. Therefore, a meta-learning paradigm and CosAttn for streamer action recognition method in live video is proposed, including: (1) the training set samples similar to the streamer action to be recognized are pretrained to improve the backbone network; (2) video-level features are extracted by R(2+1)D-18 backbone and global average pooling in the meta-learning paradigm; (3) the streamer action is recognized by calculating cosine similarity after sending the video-level features to CosAttn to generate a streamer action category prototype. Experimental results on several real-world action recognition datasets demonstrate the effectiveness of our method.

Generalized Pose Decoupled Network for Unsupervised 3D Skeleton Sequence-Based Action Representation Learning.

Human action representation is derived from the description of human shape and motion. The traditional unsupervised 3-dimensional (3D) human action representation learning method uses a recurrent neural network (RNN)-based autoencoder to reconstruct the input pose sequence and then takes the midlevel feature of the autoencoder as representation. Although RNN can implicitly learn a certain amount of motion information, the extracted representation mainly describes the human shape and is insufficient to describe motion information. Therefore, we first present a handcrafted motion feature called pose flow to guide the reconstruction of the autoencoder, whose midlevel feature is expected to describe motion information. The performance is limited as we observe that actions can be distinctive in either motion direction or motion norm. For example, we can distinguish "sitting down" and "standing up" from motion direction yet distinguish "running" and "jogging" from motion norm. In these cases, it is difficult to learn distinctive features from pose flow where direction and norm are mixed. To this end, we present an explicit pose decoupled flow network (PDF-E) to learn from direction and norm in a multi-task learning framework, where 1 encoder is used to generate representation and 2 decoders are used to generating direction and norm, respectively. Further, we use reconstructing the input pose sequence as an additional constraint and present a generalized PDF network (PDF-G) to learn both motion and shape information, which achieves state-of-the-art performances on large-scale and challenging 3D action recognition datasets including the NTU RGB+D 60 dataset and NTU RGB+D 120 dataset.

Lightweight Few-Shot Learning Based on Triple Information for Internet-of-Things Applications

Under the stringent requirements of latency, reliability and privacy in IoT scenarios, IoT intelligence has gradually sunk to endpoint devices. However, in this process of deploying the deep learning models on endpoint devices, the challenges of constrained data resources and computing resources are encountered. In order to address these challenges, we propose in this paper a lightweight few-shot learning algorithm by the use of limited data and small-scale network structure, called TI-FSL (triple information based few-shot learning), which not only has strong generalization ability but also can be deployed on the resource-constrained endpoint devices for IoT applications. Our algorithm proceeds in 3 steps. First, we construct triples for few-shot classification, in which the anchor is from the query set and the positive and the negative belong to the support set. Second, based on the above triples, the triple information is measured by the maximum triplet loss of the anchor, positive class and different negative classes. Third, the lightweight classifier can be learned by optimizing the triple information to update the network structure. We evaluate our algorithm on the most popular few-shot dataset <i>mini</i>ImageNet and an IoT activity recognition dataset <i>SDA</i>; and the results demonstrate the superiority of our algorithm against other state-of-the-art methods.

A knowledge-driven activity recognition framework for learning unknown activities

Human activity recognition has increasingly received attention in recent years to track regular activities of people. Existing activity recognition approaches considerably contributed to the analysis of human behavior. However, they still confront numerous issues related to the variability of activities performed by people within dynamic environments. Generally, this variability renders the used training or ontology models with predefined activities unsuitable. Therefore, creating an activity recognition approach that is able to leverage dynamically new and unknown activities at runtime becomes important. In this paper, we propose a novel knowledge-driven activity recognition framework using smartphone. This framework envisions taking a knowledge-driven approach to reinforce the recognition accuracy and people's quality of life in the context of dynamic environments at runtime. More specifically, we propose an ontology-based context evolution along with a dynamic decision-making, so that new and unknown performed activities can be accurately recognized. Furthermore, we use a public activity recognition dataset to demonstrate the effectiveness of the proposed framework and show its advantage over a data-driven baselines in terms of accuracy. Experimental results reveal that our framework not only reinforces the accuracy, but also enables an effective activity learning when facing unknown activities at runtime.

Visual Feature Learning on Video Object and Human Action Detection: A Systematic Review.

Video object and human action detection are applied in many fields, such as video surveillance, face recognition, etc. Video object detection includes object classification and object location within the frame. Human action recognition is the detection of human actions. Usually, video detection is more challenging than image detection, since video frames are often more blurry than images. Moreover, video detection often has other difficulties, such as video defocus, motion blur, part occlusion, etc. Nowadays, the video detection technology is able to implement real-time detection, or high-accurate detection of blurry video frames. In this paper, various video object and human action detection approaches are reviewed and discussed, many of them have performed state-of-the-art results. We mainly review and discuss the classic video detection methods with supervised learning. In addition, the frequently-used video object detection and human action recognition datasets are reviewed. Finally, a summarization of the video detection is represented, e.g., the video object and human action detection methods could be classified into frame-by-frame (frame-based) detection, extracting-key-frame detection and using-temporal-information detection; the methods of utilizing temporal information of adjacent video frames are mainly the optical flow method, Long Short-Term Memory and convolution among adjacent frames.

Disagreement-based class incremental random forest for sensor-based activity recognition

Activity recognition plays a key role in many fields, such as health monitoring and elderly care. Handling changes in user habits is a significant technical challenge in activity recognition. Ideally, a model should adapt to newly emerging classes and concept drift dynamically. This paper proposes a novel semi-supervised class incremental learning method, namely, disagreement-based class incremental random forest (Di-CIRF). The proposed model can detect newly emerging classes and update a previously established activity recognition model through streaming data. First, it is necessary to identify novel candidates by employing the disagreement-based confidence voting mechanism and minimum bounding box (MBB)-based separation detection to annotate newly emerging data accurately. Then, the coarse coding-based cohesion detection strategy is adopted to filter out the true novelty instances. This paper also proposes the iterative MBB-based splitting strategy and the pseudo-instance generation mechanism in Di-CIRF for updating the activity model without retaining the trained data. According to experimental results on four public activity recognition datasets, Di-CIRF outperforms the state-of-the-art methods.

A Scalable Solution to Detect Behavior Changes of Elderly People Living Alone

As the world population is ageing rapidly and old age comes with multiple health issues, the need for medical services is likely to increase in a couple of decades beyond the limits of the medical systems of almost any country. In response to this trend, a variety of technologies have been developed with the aim of helping older people live independently as long as possible and reduce the burden on caregivers. In this paper, we propose a solution to encode the information about the activity of the monitored person, captured by a set of binary sensors, in the form of activity maps that reflect not only the intensity, but also the spatial distribution of the activity between a set of behaviorally meaningful places. Then, we propose a method for automatic analysis of the activity maps in order to detect deviations from the previously recorded routine. We have tested the method on two public activity recognition datasets and found that the proposed solution is not only feasible, but also has several important advantages (it is low cost, scalable, adaptable, requires little expert knowledge for setup and protects the privacy of the monitored persons) that make it applicable on a large scale, including for people with low socio-economic status.

Novel Machine Learning for Human Actions Classification Using Histogram of Oriented Gradients and Sparse Representation

Recognition of human actions is a trending research topic as it can be used for crucial medical applications like life care and healthcare. In this research, we propose a novel machine learning algorithm for the classification of human actions based on sparse representation theory. In the proposed framework, the input videos are initially partitioned into several temporal segments of a predefined length. From these temporal segments, the key-cuboids are then obtained. These cuboids are obtained based on the locations having maximum variation in orientation. From these regions, key-cuboids are extracted. From the key-cuboids, Histogram of Oriented Gradient (HOG) features are extracted. This new descriptor has the capability to express the dynamic features in the action videos. Using these features, a single shared dictionary is created from the videos belonging to different classes using K-Singular Value Decomposition (K-SVD) algorithm. This dictionary has the combined features of all the action classes. This shared dictionary is generated during the training phase. During the testing phase, the features belonging to a test class is classified using a novel Sparse Representation Modeling based Action Recognition (SRMAR) Algorithm using Orthogonal Matching Pursuit (OMP) and the shared dictionary. The proposed framework was evaluated using popular benchmark action recognition datasets like KTH dataset, Olympic dataset and the Hollywood dataset. The results obtained using these datasets were represented in the form of a confusion matrix. Evaluation was performed using metrics like overall classification accuracy, specificity, precision, recall and F-score that were obtained from the confusion matrix. This system achieved a high specificity of about 99.52%, 99.16% and 96.15% for the KTH dataset, Olympic dataset and the Hollywood datasets, respectively. Similarly, the proposed framework attained very good precision of 97.64%, 90.46% and 73.39% for the KTH dataset, Olympic dataset and the Hollywood datasets, respectively. Also, the average value of recall achieved was 97.58%, 90.86% and 74.09% for the KTH dataset, Olympic dataset and the Hollywood datasets, respectively. It was also observed that the proposed machine learning algorithm achieved outstanding results compared to the existing state-of-the-art human action recognition frameworks in the literature.

Sensor-Based Human Activity Recognition Using Adaptive Class Hierarchy.

In sensor-based human activity recognition, many methods based on convolutional neural networks (CNNs) have been proposed. In the typical CNN-based activity recognition model, each class is treated independently of others. However, actual activity classes often have hierarchical relationships. It is important to consider an activity recognition model that uses the hierarchical relationship among classes to improve recognition performance. In image recognition, branch CNNs (B-CNNs) have been proposed for classification using class hierarchies. B-CNNs can easily perform classification using hand-crafted class hierarchies, but it is difficult to manually design an appropriate class hierarchy when the number of classes is large or there is little prior knowledge. Therefore, in our study, we propose a class hierarchy-adaptive B-CNN, which adds a method to the B-CNN for automatically constructing class hierarchies. Our method constructs the class hierarchy from training data automatically to effectively train the B-CNN without prior knowledge. We evaluated our method on several benchmark datasets for activity recognition. As a result, our method outperformed standard CNN models without considering the hierarchical relationship among classes. In addition, we confirmed that our method has performance comparable to a B-CNN model with a class hierarchy based on human prior knowledge.

Optimal multiple key‐based homomorphic encryption with deep neural networks to secure medical data transmission and diagnosis

AbstractMedical database classification problems can be considered as complex optimization problems to assure the diagnosis support precisely. In healthcare, several computer researchers have employed different deep learning (DL) approaches to enhance the classification performance. Besides, encryption is an effective way to offer secure transmission of medical data over public network. With this motivation, this paper presents new privacy‐preserving encryption with DL based medical data transmission and classification (PPEDL‐MDTC) model. The presented model derives multiple key‐based homomorphic encryption (MHE) technique with sailfish optimization (SFO), called MHE‐SFO algorithm‐based encryption process. In addition, the cross‐entropy based artificial butterfly optimization‐based feature selection technique and optimal deep neural network (ODNN) based classification is carried out. In ODNN model, the hyperparameter optimization of the DNN model is carried out utilizing the use of chemical reaction optimization (CRO) algorithm. The proposed method has been simulated utilizing Python 3.6.5 tool, which is tested using activity recognition and sleep stage dataset. A detailed comparative outcomes analysis makes sure the higher efficiency of the PPEDL‐MDTC on the state of art techniques with the detection accuracy of 0.9813 and 0.9650 on the applied activity recognition and University College Dublin Sleep Stage dataset.

Spatial‐temporal slowfast graph convolutional network for skeleton‐based action recognition

In skeleton-based action recognition, the graph convolutional network (GCN) has achieved great success. Modelling skeleton data in a suitable spatial-temporal way and designing the adjacency matrix are crucial aspects for GCN-based methods to capture joint relationships. In this study, we propose the spatial-temporal slowfast graph convolutional network (STSF-GCN) and design the adjacency matrices for the skeleton data graphs in STSF-GCN. STSF-GCN contains two pathways: (1) the fast pathway is in a high frame rate, and joints of adjacent frames are unified to build ‘small’ spatial-temporal graphs. A new spatial-temporal adjacency matrix is proposed for these ‘small’ spatial-temporal graphs. Ablation studies verify the effectiveness of the proposed adjacency matrix. (2) The slow pathway is in a low frame rate, and joints from all frames are unified to build one ‘big’ spatial-temporal graph. The adjacency matrix for the ‘big’ spatial-temporal graph is obtained by computing self-attention coefficients of each joint. Finally, outputs from two pathways are fused to predict the action category. STSF-GCN can efficiently capture both long-range and short-range spatial-temporal joint relationships. On three datasets for skeleton-based action recognition, STSF-GCN can achieve state-of-the-art performance with much less computational cost.

Multi‐stream adaptive spatial‐temporal attention graph convolutional network for skeleton‐based action recognition

Skeleton-based action recognition algorithms have been widely applied to human action recognition. Graph convolutional networks (GCNs) generalize convolutional neural networks (CNNs) to non-Euclidean graphs and achieve significant performance in skeleton-based action recognition. However, existing GCN-based models have several issues, such as the topology of the graph is defined based on the natural skeleton of the human body, which is fixed during training, and it may not be applied to different layers of the GCN model and diverse datasets. Besides, the higher-order information of the joint data, for example, skeleton and dynamic information is not fully utilised. This work proposes a novel multi-stream adaptive spatial-temporal attention GCN model that overcomes the aforementioned issues. The method designs a learnable topology graph to adaptively adjust the connection relationship and strength, which is updated with training along with other network parameters. Simultaneously, the adaptive connection parameters are utilised to optimise the connection of the natural skeleton graph and the adaptive topology graph. The spatial-temporal attention module is embedded in each graph convolution layer to ensure that the network focuses on the more critical joints and frames. A multi-stream framework is built to integrate multiple inputs, which further improves the performance of the network. The final network achieves state-of-the-art performance on both the NTU-RGBD and Kinetics-Skeleton action recognition datasets. The simulation results prove that the proposed method reveals better results than existing methods in all perspectives and that shows the superiority of the proposed method.

Weakly supervised action segmentation with effective use of attention and self-attention

This paper generates human action sequences using a novel hybrid sequence-to-sequence model that outputs a sequence of actions in the chronological order of the actions being performed in the longer activity of a given video. At test time, our models are able to generate action for each frame using weak supervision. We evaluate several sequence-to-sequence models to solve this task and demonstrate that they are able to solve action segment generation tasks on three challenging action recognition datasets. We present how to use self-attention and standard attention mechanisms with known sequence-to-sequence models for weakly supervised video action segmentation. Our new architecture is effective for weakly supervised action segmentation that uses a combination of recurrent and transformer-based sequence-to-sequence models. Our architecture consists of Transformers and GRU encoders to encode temporal information and we use self-attention and standard attention during the decoding process. We introduce an effective positional weight prior to further improve action segmentation performance. Using this architecture and two types of attention along with positional weight priors, we obtain state-of-the-art results on Breakfast and 50Salads datasets for weakly supervised action segmentation.

Porting Rulex Software to the Raspberry Pi for Machine Learning Applications on the Edge.

Edge Computing enables to perform measurement and cognitive decisions outside a central server by performing data storage, manipulation, and processing on the Internet of Things (IoT) node. Also, Artificial Intelligence (AI) and Machine Learning applications have become a rudimentary procedure in virtually every industrial or preliminary system. Consequently, the Raspberry Pi is adopted, which is a low-cost computing platform that is profitably applied in the field of IoT. As for the software part, among the plethora of Machine Learning (ML) paradigms reported in the literature, we identified Rulex, as a good ML platform, suitable to be implemented on the Raspberry Pi. In this paper, we present the porting of the Rulex ML platform on the board to perform ML forecasts in an IoT setup. Specifically, we explain the porting Rulex’s libraries on Windows 32 Bits, Ubuntu 64 Bits, and Raspbian 32 Bits. Therefore, with the aim of carrying out an in-depth verification of the application possibilities, we propose to perform forecasts on five unrelated datasets from five different applications, having varying sizes in terms of the number of records, skewness, and dimensionality. These include a small Urban Classification dataset, three larger datasets concerning Human Activity detection, a Biomedical dataset related to mental state, and a Vehicle Activity Recognition dataset. The overall accuracies for the forecasts performed are: 84.13%, 99.29% (for SVM), 95.47% (for SVM), and 95.27% (For KNN) respectively. Finally, an image-based gender classification dataset is employed to perform image classification on the Edge. Moreover, a novel image pre-processing Algorithm was developed that converts images into Time-series by relying on statistical contour-based detection techniques. Even though the dataset contains inconsistent and random images, in terms of subjects and settings, Rulex achieves an overall accuracy of 96.47% while competing with the literature which is dominated by forward-facing and mugshot images. Additionally, power consumption for the Raspberry Pi in a Client/Server setup was compared with an HP laptop, where the board takes more time, but consumes less energy for the same ML task.

Simulation for a Mems-Based CTRNN Ultra-Low Power Implementation of Human Activity Recognition.

This paper presents an energy-efficient classification framework that performs human activity recognition (HAR). Typically, HAR classification tasks require a computational platform that includes a processor and memory along with sensors and their interfaces, all of which consume significant power. The presented framework employs microelectromechanical systems (MEMS) based Continuous Time Recurrent Neural Network (CTRNN) to perform HAR tasks very efficiently. In a real physical implementation, we show that the MEMS-CTRNN nodes can perform computing while consuming power on a nano-watts scale compared to the micro-watts state-of-the-art hardware. We also confirm that this huge power reduction doesn't come at the expense of reduced performance by evaluating its accuracy to classify the highly cited human activity recognition dataset (HAPT). Our simulation results show that the HAR framework that consists of a training module, and a network of MEMS-based CTRNN nodes, provides HAR classification accuracy for the HAPT that is comparable to traditional CTRNN and other Recurrent Neural Network (RNN) implantations. For example, we show that the MEMS-based CTRNN model average accuracy for the worst-case scenario of not using pre-processing techniques, such as quantization, to classify 5 different activities is 77.94% compared to 78.48% using the traditional CTRNN.

Rule Ensemble Learning Using Hierarchical Kernels in Structured Output Spaces

The goal in Rule Ensemble Learning (REL) is simultaneous discovery of a small set of simple rules and their optimal weights that lead to good generalization. Rules are assumed to be conjunctions of basic propositions concerning the values taken by the input features. It has been shown that rule ensembles for classification can be learnt optimally and efficiently using hierarchical kernel learning approaches that explore the exponentially large space of conjunctions by exploiting its hierarchical structure. The regularizer employed penalizes large features and thereby selects a small set of short features. In this paper, we generalize the rule ensemble learning using hierarchical kernels (RELHKL) framework to multi class structured output spaces. We build on the StructSVM model for sequence prediction problems and employ a ρ-norm hierarchical regularizer for observation features and a conventional 2-norm regularizer for state transition features. The exponentially large feature space is searched using an active set algorithm and the exponentially large set of constraints are handled using a cutting plane algorithm. The approach can be easily extended to other structured output problems. We perform experiments on activity recognition datasets which are prone to noise, sparseness and skewness. We demonstrate that our approach outperforms other approaches.