Human Activity Recognition Dataset Research Articles

Binarized neural network for edge intelligence of sensor-based human activity recognition

A wide diversity of sensors has been applied in human activity recognition. These sensors generate enormous amounts of data during human activity monitoring. The long-distance data traveling between sensors and servers increases the costs of bandwidth and latency. However, human activity recognition has a high demand for real-time processing. Recently, edge computing is surging to solve this problem by moving computation and data storage closer to the sensor devices, rather than relying on a central server/cloud. Edge servers are usually designed for low power, low cost, and low computation. They do not support computation-intensive deep learning algorithms or will result in high latency. Fortunately, the development of binarized neural networks enables edge intelligence which supports AI running at the network edge for real-time applications. In this paper, we implement a binarized neural network (BinaryDilatedDenseNet) to enable low-latency and low-memory human activity recognition at the network edge. We applied the BinaryDilatedDenseNet on three sensor-based human activity recognition datasets and evaluated it with four metrics. In comparison, the BinaryDilatedDenseNet outperforms the related work and other three binarized neural networks in accuracy and saves 10 memory and 4.5--8 inference time compared to the FPDilatedDenseNet(the full-precision version of the BinaryDilatedDenseNet).

Learning Disentangled Representation for Mixed- Reality Human Activity Recognition With a Single IMU Sensor

Together with the rapid development of the sensors technology in recent years, sensor-based human activity recognition (HAR) has shown promising performance using well-known supervised deep learning methods. However, it remains challenging in a realistic scenario, i.e., limited number of labeled samples and sensors. This article proposes a novel deep learning method to achieve accurate and robust HAR with only a single inertial measurement unit (IMU) sensor. Our contributions are twofold. First, based on the skinned multiperson linear (SMPL) model, we build a large synthetic HAR dataset containing multimodal measurements: acceleration and angular velocity, which were generated according to the forward kinematics. Second, We propose a multiple-level domain adaptive learning model with information-theoretically stimulated constraints to simultaneously align the distributions of low- and high-level representations of virtual and real HAR data. The proposed mutual information constraints encourage the neural network to learn a disentangled representation for the multimodal sensing data. Comprehensive experimental results on three publicly available datasets demonstrate that the proposed method compares favorably with competing ones and has robust performance with variable labeled samples.

Deep Neural Networks for Sensor-Based Human Activity Recognition Using Selective Kernel Convolution

Recently, the state-of-the-art performance in various sensor-based human activity recognition (HAR) tasks has been acquired by deep learning, which can extract automatically features from raw data. In standard convolutional neural networks (CNNs), there is usually the same receptive field (RF) size of artificial neurons within each feature layer. It is well known that the RF size of neurons is able to change adaptively according to the stimulus, which has rarely been exploited in HAR. In this article, a new multibranch CNN is introduced, which utilizes a selective kernel mechanism for HAR. To the best of our knowledge, it is for the first time to adopt an attention idea to perform kernel selection among multiple branches with different RFs in the HAR scenario. We perform extensive experiments on several benchmark HAR datasets, namely, UCI-HAR, UNIMIB SHAR, WISDM, PAMAP2, and OPPORTUNITY, as well as weakly labeled datasets. Ablation experiments show that the selective kernel convolution can adaptively choose an appropriate RF size among multiple branches for classifying numerous human activities. As a result, it can achieve a higher recognition accuracy under a similar computing budget.

Shallow Convolutional Neural Networks for Human Activity Recognition Using Wearable Sensors

Due to rapid development of sensor technology, human activity recognition (HAR) using wearable inertial sensors has recently become a new research hotspot. Deep learning, especially convolutional neural network (CNN) that can automatically learn intricate activity features have gained a lot of attention in ubiquitous HAR task. Most existing CNNs process sensor input by extracting channel-wise features, and the information from each channel can be separately propagated in a hierarchical way from lower layers to higher layers. As a result, they typically overlook information exchange among channels within the same layer. In this article, we first propose a shallow CNN that considers cross-channel communication in HAR scenario, where all channels in the same layer have a comprehensive interaction to capture more discriminative features of sensor input. One channel can communicate with all other channels by graph neural network to remove redundant information accumulated among channels, which is more beneficial for deploying lightweight deep models. Extensive experiments are conducted on multiple benchmark HAR datasets, namely UCI-HAR, OPPORTUNITY, PAMAP2 and UniMib-SHAR, which indicates that the proposed method enables shallower CNNs to aggregate more useful information, and surpasses baseline deep networks and other competitive methods. The inference speed is evaluated via deploying the HAR systems on an embedded system.

ISPLInception: An Inception-ResNet Deep Learning Architecture for Human Activity Recognition

Advances in deep learning (DL) model design have pushed the boundaries of the areas in which it can be applied. The fields with an immense availability of complex big data have been big beneficiaries of these advances. One such field is human activity recognition (HAR). HAR is a popular area of research in a connected world because internet-of-things (IoT) devices and smartphones are becoming more prevalent. A major research goal of recent research work has been to improve predictive accuracy for devices with limited computational resources. In this paper, we propose iSPLInception, a DL model motivated by the Inception-ResNet architecture from Google, that not only achieves high predictive accuracy but also uses fewer device resources. We evaluate the proposed model's performance on four public HAR datasets from the University of California, Irvine (UCI) machine learning repository. The proposed model's performance is compared to that of existing DL architectures that have been proposed in the recent past to solve the HAR problem. The proposed model outperforms these approaches on several metrics of accuracy, cross-entropy loss, and F <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> score on all the four datasets. The performance of the proposed iSPLInception model is validated on the UCI HAR using smartphones dataset, Opportunity activity recognition dataset, Daphnet freezing of gait dataset, and PAMAP2 physical activity monitoring dataset. The experiments and result analysis indicate that the proposed iSPLInception model achieves remarkable performance for HAR applications.

Spiking neural network based on joint entropy of optical flow features for human action recognition

In the recent past, human action recognition is inviting increased attention in the automated video surveillance systems. An efficient human action classification technique in an unconstrained environment is proposed in this paper. A novel descriptor relying on joint entropy of difference in magnitude and orientation of the optical flow feature is developed in order to model human actions. Initially, flow feature is computed using Pyramid–Warping–Cost volume Network (PWCNet), considering every two consecutive frames. Then, the feature descriptor is formed based on the joint entropy of difference in flow magnitude and orientation collected from the regular grid of each frame in the action sequence. Finally, in order to incorporate long-term temporal dependency, a spiking neural network is embedded to aggregate the information across the frames. Different optimization techniques and different types of hidden nodes are utilized in the spiking neural network to analyze the performance of the proposed work. Extensive experiments on the benchmark datasets for human action recognition show the efficacy of the proposed method.

Deriving Effective Human Activity Recognition Systems through Objective Task Complexity Assessment

Research in sensor based human activity recognition (HAR) has been a core concern of the mobile and ubiquitous computing community. Sophisticated systems have been developed with the main view on applications of HAR methods in research settings. This work addresses a related yet practically different problem that mainly focuses on users of HAR technology. We acknowledge that practitioners from outside the core HAR research community are motivated to employ HAR methods for practical deployments. Even though standard processing approaches exist, arguably, often times substantial modifications are necessary to derive effective analysis systems. It is not always clear a-priori how challenging a HAR task actually is and what dimensions of an analysis pipeline are crucial for successful automated assessments. In practice this can lead to disappointing results or disproportionate efforts that have to be invested into the optimization of data analysis pipelines, that were supposed to work "out of the box". We present a framework for the objective complexity assessment of HAR tasks that directly supports practitioners' decision making of whether and how to employ HAR for their deployments. We map a HAR task onto a vectorial representation that allows us to analyse the inherent challenges of the task and to draw conclusions through similarity analysis with regards to existing tasks. We validate our complexity assessment framework on 23 HAR datasets and derive a data-driven categorization of human activity recognition. We demonstrate how our objective analysis can be used to inform the deployment of HAR systems in practical scenarios.

KNN Prototyping Schemes for Embedded Human Activity Recognition with Online Learning

The kNN machine learning method is widely used as a classifier in Human Activity Recognition (HAR) systems. Although the kNN algorithm works similarly both online and in offline mode, the use of all training instances is much more critical online than offline due to time and memory restrictions in the online mode. Some methods propose decreasing the high computational costs of kNN by focusing, e.g., on approximate kNN solutions such as the ones relying on Locality-Sensitive Hashing (LSH). However, embedded kNN implementations also need to address the target device’s memory constraints, especially as the use of online classification needs to cope with those constraints to be practical. This paper discusses online approaches to reduce the number of training instances stored in the kNN search space. To address practical implementations of HAR systems using kNN, this paper presents simple, energy/computationally efficient, and real-time feasible schemes to maintain at runtime a maximum number of training instances stored by kNN. The proposed schemes include policies for substituting the training instances, maintaining the search space to a maximum size. Experiments in the context of HAR datasets show the efficiency of our best schemes.

Video sketch: A middle-level representation for action recognition

Different modalities extracted from videos, such as RGB and optical flows, may provide complementary cues for improving video action recognition. In this paper, we introduce a new modality named video sketch, which implies the human shape information, as a complementary modality for video action representation. We show that video action recognition can be enhanced by using the proposed video sketch. More specifically, we first generate video sketch with class distinctive action areas and then employ a two-stream network to combine the shape information extracted from image-based sketch and point-based sketch, followed by fusing the classification scores of two streams to generate shape representation for videos. Finally, we use the shape representation as the complementary one for the traditional appearance (RGB) and motion (optical flow) representations for the final video classification. We conduct extensive experiments on four human action recognition datasets – KTH, HMDB51, UCF101, Something-Something and UTI. The experimental results show that the proposed method outperforms the existing state-of-the-art action recognition methods.

Hybrid domain adaptation with deep network architecture for end-to-end cross-domain human activity recognition

Machine learning-based human activity recognition (HAR) as the means of human–computer interaction is important to empower the existing systems in the industry such as manufacturing and logistics to be more autonomous. However, it is often difficult to build HAR models due to the limitation of annotated samples. Domain adaptation has emerged to address such limitation (or absence) of labeled samples in the domain of interest (i.e., target domain) by using abundant amount of labeled samples in the other domain (i.e., source domain). Domain adaptation aims to solve learning problems where a source domain and a target domain are different but still related. With the use of homogeneous feature space, the existing approaches on homogeneous domain adaption is prohibitive for the real-life scenario, in which two different domains could be possibly of heterogeneous feature space. Although heterogeneous domain adaptation approaches exist, it still requires additional information (i.e., instance correspondence) which is difficult to satisfy when we deal with sensor data. Hybrid domain adaptation, a special case of heterogeneous domain adaptation where common feature between domain exists, is more realistic as the feature commonality is easier to satisfy. However, the existing approach operates using common features in the original feature space, which in fact, may still have distribution difference. In addition, the existing one requires hand-crafted feature extractions for more informative descriptors to classify human activities. In this work, we propose a deep architecture for hybrid domain adaptation, to enable end-to-end learning for human activity classification. The architecture is tested on sensor-based human activity recognition dataset. The experimental result shows that our approach yields better result compared to the existing human activity recognition approaches, both the deep and shallow approaches.

Skeleton edge motion networks for human action recognition

Human skeleton is receiving increasing attention from the community of human action recognition due to its robustness to complex image backgrounds. Previous methods usually utilize body joint-based representation, i.e., joint locations, while leaving edge-based movement poorly investigated. In this paper, we propose a new human action recognition method, skeleton edge motion networks (SEMN), to further explore the motion information of human body parts. Specifically, we address the movement of skeleton edge by using the angle changes of skeleton edge and the movement of the corresponding body joints. We then devise the proposed skeleton edge motion networks by stacking multiple spatial-temporal blocks to learn a robust deep representation from skeleton sequences. Furthermore, we propose a new progressive ranking loss to help the proposed skeleton edge motion networks maintain temporal order information in a self-supervised manner. Experimental results on five popular human action recognition datasets, PennAction, UTD-MHAD, NTU RGB+D, NTU RGB+D 120, and CSL, demonstrate the effectiveness of the proposed method.

Human action recognition method based on conformal geometric algebra and recurrent neural network

Deep Learning (DL) plays an important role in machine learning and artificial intelligence. DL is widely applied in many fields with high dimensional data, including natural language processing, image recognition. High dimensional data can lead to problems in machine learning such as overfitting, degradation of accuracy. To address these issues, some methods, such as Principal Components Analysis (PCA), principal component regression (PCR), Multi-class Linear Discriminant Analysis (MLDA), were proposed to reduce dimensions of the data and computational complexity simultaneously. The drawback of these methods is that they only work well on data distributed on the plane. In the case of the data distributed on the hyper-sphere, such as objects moving in space, the processing results are not so good as expected. In this paper, we propose the use of Conformal Geometric Algebra (CGA) to extract features and simultaneously reduce the dimensionality of a dataset for human activity recognition using Recurrent Neural Network (RNN). First, human activity data in a 3-dimensional coordinate system is pre-processed and normalized by calculating deviations from the mean coordinate. Next, the data is transformed to vectors in CGA space and its dimensions are reduced to return the feature vectors. Finally, we use the RNN model to train feature vectors. Empirical results performed on the CMU eight actions dataset show that the CGA combined with RNN gave the best test results of 92.5%.

CNN-Based Multistage Gated Average Fusion (MGAF) for Human Action Recognition Using Depth and Inertial Sensors

Convolutional Neural Network (CNN) provides leverage to extract and fuse features from all layers of its architecture. However, extracting and fusing intermediate features from different layers of CNN structure is still uninvestigated for Human Action Recognition (HAR) using depth and inertial sensors. To get maximum benefit of accessing all the CNN's layers, in this paper, we propose novel Multistage Gated Average Fusion (MGAF) network which extracts and fuses features from all layers of CNN using our novel and computationally efficient Gated Average Fusion (GAF) network, a decisive integral element of MGAF. At the input of the proposed MGAF, we transform the depth and inertial sensor data into depth images called sequential front view images (SFI) and signal images (SI) respectively. These SFI are formed from the front view information generated by depth data. CNN is employed to extract feature maps from both input modalities. GAF network fuses the extracted features effectively while preserving the dimensionality of fused feature as well. The proposed MGAF network has structural extensibility and can be unfolded to more than two modalities. Experiments on three publicly available multimodal HAR datasets demonstrate that the proposed MGAF outperforms the previous state-of-the-art fusion methods for depth-inertial HAR in terms of recognition accuracy while being computationally much more efficient. We increase the accuracy by an average of 1.5% while reducing the computational cost by approximately 50% over the previous state-of-art.

A Multiviewpoint Outdoor Dataset for Human Action Recognition

Advancements in deep neural networks have contributed to near perfect results for many computer vision problems such as object recognition, face recognition and pose estimation. However, human action recognition is still far from human-level performance. Owing to the articulated nature of the human body, it is challenging to detect an action from multiple viewpoints, particularly from an aerial viewpoint. This is further compounded by a scarcity of datasets that cover multiple viewpoints of actions. To fill this gap and enable research in wider application areas, we present a multi-viewpoint outdoor action recognition dataset collected from YouTube and our own drone. The dataset consists of 20 dynamic human action classes, 2324 video clips and 503086 frames. All videos are cropped and resized to 720x720 without distorting the original aspect ratio of the human subjects in videos. This dataset should be useful to many research areas including action recognition, surveillance and situational awareness. We evaluated the dataset with a two-stream CNN architecture coupled with a recently proposed temporal pooling scheme called kernelized rank pooling that produces nonlinear feature subspace representations. The overall baseline action recognition accuracy is 74.0%.

Novel tent pooling based human activity recognition approach

Nowadays, mobile devices have been widely used worldwide and they have variable sensors. By using these sensors, many applications for instance navigation, healthcare, and human activity recognition (HAR) have been developed. The primary objective of this paper is to present a multi lightweight HAR method with high prediction performance. To implement this objective, novel pooling methodology is presented and it is called as tent pooling since pooling is processed using the variable size of blocks. 2 × 2, 3 × 3, 4 × 4, 5 × 5, 6 × 6, 7 × 7 and 8 × 8 sized non-overlapping blocks are used for pooling and maximum, minimum and average pooling methods. These pooling methods have been utilized as a feature generation model. The generated features are forwarded to the ReliefF feature selection method. ReliefF is a weight-based feature selector and uses Manhattan distance based fitness function. By using relief, the weights of the features are calculated and the negative weighted features are eliminated. This operation is named positive weighted feature selection. The selected positive weighted features are classified using a cubic support vector machine (SVM). To test the proposed method, a publicly and freely published HAR dataset is used. There are 6 activities in the used dataset. Also, the results of the proposed method were compared to other state of art methods. The best accuracy rate of the proposed tent pooling based HAR method was calculated as 99.81% using the average pooling. This results clearly prove that the success of the proposed tent pooling based method. The proposed method is also simple and effective. It has the highest success rates among the selected state-art-of HAR methods.

IMUTube

The lack of large-scale, labeled data sets impedes progress in developing robust and generalized predictive models for on-body sensor-based human activity recognition (HAR). Labeled data in human activity recognition is scarce and hard to come by, as sensor data collection is expensive, and the annotation is time-consuming and error-prone. To address this problem, we introduce IMUTube, an automated processing pipeline that integrates existing computer vision and signal processing techniques to convert videos of human activity into virtual streams of IMU data. These virtual IMU streams represent accelerometry at a wide variety of locations on the human body. We show how the virtually-generated IMU data improves the performance of a variety of models on known HAR datasets. Our initial results are very promising, but the greater promise of this work lies in a collective approach by the computer vision, signal processing, and activity recognition communities to extend this work in ways that we outline. This should lead to on-body, sensor-based HAR becoming yet another success story in large-dataset breakthroughs in recognition.

Layer-Wise Training Convolutional Neural Networks With Smaller Filters for Human Activity Recognition Using Wearable Sensors

Recently, convolutional neural networks (CNNs) have set latest state-of-the-art on various human activity recognition (HAR) datasets. However, deep CNNs often require more computing resources, which limits their applications in embedded HAR. Although many successful methods have been proposed to reduce memory and FLOPs of CNNs, they often involve special network architectures designed for visual tasks, which are not suitable for deep HAR tasks with time series sensor signals, due to remarkable discrepancy. Therefore, it is necessary to develop lightweight deep models to perform HAR. As filter is the basic unit in constructing CNNs, it deserves further research whether re-designing smaller filters is applicable for deep HAR. In the paper, inspired by the idea, we proposed a lightweight CNN using Lego filters for HAR. A set of lower-dimensional filters is used as Lego bricks to be stacked for conventional filters, which does not rely on any special network structure. The local loss function is used to train model. To our knowledge, this is the first paper that proposes lightweight CNN for HAR in ubiquitous and wearable computing arena. The experiment results on five public HAR datasets, UCI-HAR dataset, OPPORTUNITY dataset, UNIMIB-SHAR dataset, PAMAP2 dataset, and WISDM dataset collected from either smartphones or multiple sensor nodes, indicate that our novel Lego CNN with local loss can greatly reduce memory and computation cost over CNN, while achieving higher accuracy. That is to say, the proposed model is smaller, faster and more accurate. Finally, we evaluate the actual performance on an Android smartphone.

LARa: Creating a Dataset for Human Activity Recognition in Logistics Using Semantic Attributes.

Optimizations in logistics require recognition and analysis of human activities. The potential of sensor-based human activity recognition (HAR) in logistics is not yet well explored. Despite a significant increase in HAR datasets in the past twenty years, no available dataset depicts activities in logistics. This contribution presents the first freely accessible logistics-dataset. In the ’Innovationlab Hybrid Services in Logistics’ at TU Dortmund University, two picking and one packing scenarios were recreated. Fourteen subjects were recorded individually when performing warehousing activities using Optical marker-based Motion Capture (OMoCap), inertial measurement units (IMUs), and an RGB camera. A total of 758 min of recordings were labeled by 12 annotators in 474 person-h. All the given data have been labeled and categorized into 8 activity classes and 19 binary coarse-semantic descriptions, also called attributes. The dataset is deployed for solving HAR using deep networks.

A Novel Multichannel Dilated Convolution Neural Network for Human Activity Recognition

A novel multichannel dilated convolution neural network for improving the accuracy of human activity recognition is proposed. The proposed model utilizes the multichannel convolution structure with multiple kernels of various sizes to extract multiscale features of high-dimensional data of human activity during convolution operation and not to consider the use of the pooling layers that are used in the traditional convolution with dilated convolution. Its advantage is that the dilated convolution can first capture intrinsical sequence information by expanding the field of convolution kernel without increasing the parameter amount of the model. And then, the multichannel structure can be employed to extract multiscale gait features by forming multiple convolution paths. The open human activity recognition dataset is used to evaluate the effectiveness of our proposed model. The experimental results showed that our model achieves an accuracy of 95.49%, with the time to identify a single sample being approximately 0.34 ms on a low-end machine. These results demonstrate that our model is an efficient real-time HAR model, which can gain the representative features from sensor signals at low computation and is hopeful for the effective tool in practical applications.

Discovering and incorporating latent target-domains for domain adaptation

In this paper, we aim to address the unsupervised domain adaptation problem where the data in the target domain are much more diverse compared with the data in the source domain. In particular, this problem is formulated as discovering and incorporating latent domains underlying target data of interest for unsupervised domain adaptation. More specifically, the discovery of the latent target domains is based on three criteria, including the maximization of compactness and distinctiveness of the data in the individual latent target-domain, as well as the minimization of total divergence from the latent target-domains to the source domain. For each pair formed by a latent target domain and the source domain, we learn a feature space where the discrepancy between the source domain and the specific latent target domain is shrunk. Finally, we consider the projected source domain data on the learned latent feature spaces as different views of the source domain, and propose an extended multiple kernel learning algorithm to train a more robust and precise classifier for predicting the unlabeled target data. The effectiveness of our proposed method is demonstrated on various benchmark datasets for object recognition and human activity recognition. Moreover, we also show that our proposed method can be treated as an effective complement to the deep learning based unsupervised domain adaptation.