Performance Of Human Activity Recognition Research Articles

Modality aware contrastive learning for multimodal human activity recognition

SummaryHuman activity recognition is a well‐established research problem in ubiquitous computing. The increased dependency on various smart devices in our daily lives allows us to investigate the sensor data world produced by multimodal sensors embedded in smart devices. However, the raw sensor data are often unlabeled and annotating this vast amount of data are a costly exercise that can often lead to privacy breaches. Self‐supervised learning‐based approaches are at the forefront of learning semantic representation from unlabeled sensor data, including when applied to human activity recognition tasks. As inferring human activity depends on multimodal sensors, addressing the modality difference and inter‐modality dependencies in a model is an important process. This paper proposes a novel self‐supervised learning approach, modality aware contrastive learning (MACL), for representation learning using multimodal sensor data. The approach uses different sensing modalities to create different views of an input signal. Thus, the model is able to learn the representations by maximizing the similarity among different sensing modalities of the same input signal. Extensive experiments were performed on four publicly available human activity recognition data sets to verify the effectiveness of our proposed MACL method. The experimental evaluation results show that the MACL method attains a comparable performance for human activity recognition to the compared baseline models, directly exceeding the performance of models using standard augmentation transformation strategies.

Multiscale Human Activity Recognition and Anticipation Network.

Deep convolutional neural networks have been leveraged to achieve huge improvements in video understanding and human activity recognition performance in the past decade. However, most existing methods focus on activities that have similar time scales, leaving the task of action recognition on multiscale human behaviors less explored. In this study, a two-stream multiscale human activity recognition and anticipation (MS-HARA) network is proposed, which is jointly optimized using a multitask learning method. The MS-HARA network fuses the two streams of the network using an efficient temporal-channel attention (TCA)-based fusion approach to improve the model's representational ability for both temporal and spatial features. We investigate the multiscale human activities from two basic categories, namely, midterm activities and long-term activities. The network is designed to function as part of a real-time processing framework to support interaction and mutual understanding between humans and intelligent machines. It achieves state-of-the-art results on several datasets for different tasks and different application domains. The midterm and long-term action recognition and anticipation performance, as well as the network fusion, are extensively tested to show the efficiency of the proposed network. The results show that the MS-HARA network can easily be extended to different application domains.

Spatio-Temporal Information Fusion and Filtration for Human Action Recognition

Human action recognition (HAR) as the most representative human-centred computer vision task is critical in human resource management (HRM), especially in human resource recruitment, performance appraisal, and employee training. Currently, prevailing approaches to human action recognition primarily emphasize either temporal or spatial features while overlooking the intricate interplay between these two dimensions. This oversight leads to less precise and robust action classification within complex human resource recruitment environments. In this paper, we propose a novel human action recognition methodology for human resource recruitment environments, which aims at symmetrically harnessing temporal and spatial information to enhance the performance of human action recognition. Specifically, we compute Depth Motion Maps (DMM) and Depth Temporal Maps (DTM) from depth video sequences as space and time descriptors, respectively. Subsequently, a novel feature fusion technique named Center Boundary Collaborative Canonical Correlation Analysis (CBCCCA) is designed to enhance the fusion of space and time features by collaboratively learning the center and boundary information of feature class space. We then introduce a spatio-temporal information filtration module to remove redundant information introduced by spatio-temporal fusion and retain discriminative details. Finally, a Support Vector Machine (SVM) is employed for human action recognition. Extensive experiments demonstrate that the proposed method has the ability to significantly improve human action recognition performance.

Video-Based Human Activity Recognition Using Deep Learning Approaches.

Due to its capacity to gather vast, high-level data about human activity from wearable or stationary sensors, human activity recognition substantially impacts people's day-to-day lives. Multiple people and things may be seen acting in the video, dispersed throughout the frame in various places. Because of this, modeling the interactions between many entities in spatial dimensions is necessary for visual reasoning in the action recognition task. The main aim of this paper is to evaluate and map the current scenario of human actions in red, green, and blue videos, based on deep learning models. A residual network (ResNet) and a vision transformer architecture (ViT) with a semi-supervised learning approach are evaluated. The DINO (self-DIstillation with NO labels) is used to enhance the potential of the ResNet and ViT. The evaluated benchmark is the human motion database (HMDB51), which tries to better capture the richness and complexity of human actions. The obtained results for video classification with the proposed ViT are promising based on performance metrics and results from the recent literature. The results obtained using a bi-dimensional ViT with long short-term memory demonstrated great performance in human action recognition when applied to the HMDB51 dataset. The mentioned architecture presented 96.7 ± 0.35% and 41.0 ± 0.27% in terms of accuracy (mean ± standard deviation values) in the train and test phases of the HMDB51 dataset, respectively.

CeHAR: CSI-Based Channel-Exchanging Human Activity Recognition

Despite the intense effort from the research community, state-of-the-art WiFi-based human activity recognition performance remains unsatisfactory. Current approaches usually use individual characteristics of CSI, i.e., amplitude or phase measurements, to model the relationship between the changes of channel state information (CSI) and human activities, which lead to their failure to achieve satisfactory accuracy due to information loss. To deal with this issue, this paper proposes CeHAR, a CSI-based Human Activity Recognition using a channel-exchanging fusion network to deep fuse the CSI amplitude and phase features to obtain the informative features for human activity recognition. The proposed CeHAR is a parameter-free dual-characteristic fusion framework that dynamically exchanges channels between sub-networks of two kinds of characteristics to comprehensively learn informative features from both. Specifically, the proposed approach employs two sub-network using convolutional neural networks to learn features from each characteristic of CSI. The magnitude of Batch-Normalization (BN) scaling factor is used to determine the channel importance of each characteristic, and then guides the exchange process. The proposed CeHAR also shares convolutional filters, but keeps private BNs layers in different characteristics, which, as an added benefit, allows our characteristic fusion network to be nearly as compact as a single-characteristic network. Extensive real-world experiments have been conducted to evaluate the performance of our proposed CeHAR, and the experimental results illustrate that our proposed approach outperforms baselines.

Human Action Recognition

human-to-human interaction and interpersonal relations Human activity recognition plays a significant role. Because to identity of a person, their personality, and psychological state it provides information, it is difficult to extract. another person's activities is one of the main subjects of study of the scientific areas of computer vision The human ability to recognize and soft computing learning. robotics for human behavior characterization, require a multiple activity recognition system, many applications, including video surveillance systems, human computer interaction, and human activity recognition is an important research direction In video analysis, a large number of papers have been published on human activity recognition in video and image sequences In the past, including methods, systems, and quantitative evaluation of the performance of human activity recognition In this paper, we provide a comprehensive survey of the recent development of the techniques, for the video improve classification, interpretation, and performance. The experimental results show that our method can significantly.

A Wearable Flexible Tactile Sensor with Textile Microstructure for Wirelessly Recognizing Human Activity

AbstractFlexible tactile sensors have attracted special attention in the detection of human posture and activity. However, achieving multiple stimulus responses is particularly challenging in terms of structural design and has not been fully addressed. Herein, a novel structural design inspired by combining the characteristics of a strain sensor and e‐textile into a hybrid interface to obtain stretchable multiple stimulus‐responsive tactile sensors (stretching stimulus, pressing stimulus, and bending stimulus) is proposed. The sensitive layer of the sensor is designed as a textile microstructure. The sensor, which has a stable performance with excellent ohm property and durability (>2 years), possesses an extraordinary gauge factor (≈200) and a fast response time (≈2.48 ms) under tensile strain, a high sensitivity (≈2.35 kPa−1) in a low‐pressure range (≈9 kPa) under pressure. It has excellent performance for Human Activity Recognition by being attached to human skin. In addition, a Bluetooth device that tracks muscle activity in real‐time and transmits the output signals to a smartphone application is utilized. These results imply the utility of this sensor for a diverse application of robotic e‐skins or e‐muscles.

Multi-level channel attention excitation network for human action recognition in videos

Channel attention mechanism has continuously attracted strong interests and shown great potential in enhancing the performance of deep CNNs. However, when applied to video-based human action recognition task, most existing methods generally learn channel attention at frame level, which ignores the temporal dependencies and may limit the recognition performance. In this paper, we propose a novel multi-level channel attention excitation (MCAE) module to model the temporal-related channel attention at both frame and video levels. Specifically, based on video convolutional feature maps, frame-level channel attention (FCA) is generated by exploring time-channel correlations, and video-level channel attention (VCA) is generated by aggregating global motion variations. MCAE firstly recalibrates video feature responses with frame-wise FCA, and then activates the motion-sensitive channel features with motion-aware VCA. MCAE module learns the channel discriminability from multiple levels and can act as a guidance to facilitate efficient spatiotemporal feature modeling in activated motion-sensitive channels. It can be flexibly embedded into 2D networks with very limited extra computation cost to construct MCAE-Net, which effectively enhances the spatiotemporal representation of 2D models for video action recognition task Extensive experiments on five human action datasets show that our method achieves superior or very competitive performance compared with the state-of-the-arts, which demonstrates the effectiveness of the proposed method for improving the performance of human action recognition.

Sample Intercorrelation-Based Multidomain Fusion Network for Aquatic Human Activity Recognition Using Millimeter-Wave Radar

To address the issue that existing multi-domain fusion methods do not consider data correlation within mini-batch data, the first attempt is made in this letter to propose a method based on sample inter-correlation learning multi-domain fusion network (SIMFNet), which aims to accommodate multivariate domain data and further enhance the aquatic human activity recognition performance. To fully utilize the radar multidimensional information, the three-branch convolution neural network (CNN) feature extractor is first employed to extract domain-specific features from the time-range map (TRM), time-Doppler map (TDM) and cadence velocity diagram (CVD). Then, the multi-domain features are fused and fed into the graph construction layer (GCL) to generate instance graphs. Next, a graph aggregation layer (GAL) is applied to aggregate node information from various-hop neighborhood domains. Finally, node-level classification is used to achieve aquatic human activity recognition. The experimental results evaluated on the built aquatic human activity recognition dataset demonstrate that the proposed SIMFNet has better generalization performance than the state-of-the-art multi-domain fusion methods.

TASKED: Transformer-based Adversarial learning for human activity recognition using wearable sensors via Self-KnowledgE Distillation

Wearable sensor-based human activity recognition (HAR) has emerged as a principal research area and is utilized in a variety of applications. Recently, deep learning-based methods have achieved significant improvement in the HAR field with the development of human–computer interaction applications. However, most of them are limited to operating in a local neighborhood in the process of a standard convolution neural network, and correlations between different sensors on body positions are ignored. In addition, even though several recent existing works considered the correlations between different sensor positions, they still face significant challenging problems with performance degradation due to large gaps in the distribution of training and test data, and behavioral differences between subjects. In this work, we propose a novel Transformer-based Adversarial learning framework for human activity recognition using wearable sensors via Self-KnowledgE Distillation (TASKED), that accounts for individual sensor orientations and spatial and temporal features. The proposed method is capable of learning cross-domain embedding feature representations from multiple subjects datasets using adversarial learning and the maximum mean discrepancy (MMD) regularization to align the data distribution over multiple domains. In the proposed method, we adopt the teacher-free self-knowledge distillation to improve the stability of the training procedure and the performance of human activity recognition. Experimental results show that TASKED not only outperforms state-of-the-art methods on the four real-world public HAR datasets (alone or combined) but also improves the subject generalization effectively.

Generic semi-supervised adversarial subject translation for sensor-based activity recognition

Performance of Human Activity Recognition (HAR) models, particularly deep neural networks, is highly contingent upon the availability of the massive amount of annotated training data. Though, data collection and manual labeling in the HAR domain are prohibitively expensive due to human resource dependence in both steps. Hence, domain adaptation techniques are proposed to adapt the knowledge from the existing source of data. More recently, adversarial transfer learning methods have shown promising results for visual classification, yet limited for HAR problems, which are still prone to the unfavorable effects of the imbalanced distribution of samples. This paper presents a novel generic semi-supervised approach that takes advantage of the adversarial framework to tackle these shortcomings by leveraging knowledge from annotated samples exclusively from the source subject and unlabeled ones of the target subject. An extensive subject translation experiments is conducted on three large, middle, and small-size datasets with different levels of imbalance to assess the robustness of the proposed model to the scale as well as the imbalance in the data. The results demonstrate the effectiveness of our proposed algorithms over state-of-the-art methods, which led to up to 13%, 4%, and 13% improvement of our high-level activities recognition metrics for Opportunity, LISSI, and PAMAP2 datasets, respectively.

Tuning the Hyperparameters of the 1D CNN Model to Improve the Performance of Human Activity Recognition

The human activity recognition (HAR) field has recently become one of the trendiest research topics due to ready-made sensors such as accelerometers and gyroscopes equipped with smartphones and smartwatches as an embedded devices, decreasing the cost and power consumption. As a result, human activity is considered a time series classification problem. Now a day, deep learning approaches such as Convolutional Neural Network (CNN) have been successful when implemented with HAR to learn automatically higher-order features and, at the same time, work as a classifier. Recently, a one-dimensional Convolutional Neural Network (1D CNN) has been suggested and carried out at the best performance levels in numerous applications, such as the classification of personalized biomedical data and time series classification. This paper studies how to leverage a 1D single CNN model to produce an excellent performance on the human activity raw data. This is done by empirically tuning the values of hyperparameters, such as kernel size, filter maps, number of epochs, batch size, and promoting an advanced multi-headed 1D CNN by employing each convolutional layer with a different kernel size to gain an ensemble–like results. The selected hyper parameter's impact is evaluated on a publicly available dataset named UCI HAR collected from smartphone sensors to perform six activities. A significant determinant of better results depends on the hyperparameter that has been chosen. The results demonstrated that tuning the hyperparameter of 1D CNN increased activity recognition accuracy.

Comparison of feed forward and cascade forward neural networks for human action recognition

Humans can perform an enormous number of actions like running, walking, pushing, and punching, and can perform them in multiple ways. Hence recognizing a human action from a video is a challenging task. In a supervised learning environment, actions are first represented using robust features and then a classifier is trained for classification. The selection of a classifier does affect the performance of human action recognition. This work focuses on the comparison of two structures of the neural network, namely, feed forward neural network and cascade forward neural network, for human action recognition. Histogram of oriented gradients (HOG) and histogram of optical flow (HOF) are used as features for representing the actions. HOG represents the spatial features of the video while HOF gives motion features of the video. The performance of two neural network architectures is compared based on recognition accuracy. Well-known publically available datasets for action and interaction detection are used for testing. It is seen that, for human action recognition applications, feed forward neural network gives better results in terms of higher recognition accuracy than Cascade forward neural network.

Performance evaluation of entropy-based cuboid selection technique for human interaction recognition

Local feature-based approaches are proven to be most successful for the application of human action recognition. This work aims to study the effect of the number of cuboids chosen on the performance of human action recognition. A 3D region, called cuboid, is extracted around every spatio-temporal interest point and an entropy-based cuboid selection method is implemented for choosing the cuboids having maximum information. The proposed method is evaluated using UT interaction and SBU kinect interaction datasets. Results show that maximum classification accuracy and F1 Score are obtained when the top 40 cuboids are selected for the feature extraction. The classification accuracy and F1 Score increase when the number of selected cuboids is increased from 10 to 40 and remain constant when the number is increased beyond 40. The selected 40 cuboids are then used for feature extraction. To carry out this experiment, wavelet coefficients extracted from selected cuboids are used as features.

Complex Network-based features extraction in RGB-D human action recognition

Analysis of human behavior through visual information has been one of the active research areas in computer vision community during the last decade. Vision-based human action recognition (HAR) is a crucial part of human behavior analysis, which is also of great demand in a wide range of applications. HAR was initially performed via images from a conventional camera; however, depth sensors have recently embedded as an additional informative resource to cameras. In this paper, we have proposed a novel approach to largely improve the performance of human action recognition using Complex Network-based feature extraction from RGB-D information. Accordingly, the constructed complex network is employed for single-person action recognition from skeletal data consisting of 3D positions of body joints. The indirect features help the model cope with the majority of challenges in action recognition. In this paper, the meta-path concept in the complex network has been presented to lessen the unusual actions structure challenges. Further, it boosts recognition performance. The extensive experimental results on two widely adopted benchmark datasets, the MSR-Action Pairs, and MSR Daily Activity3D indicate the efficiency and validity of the method.

Scene image and human skeleton-based dual-stream human action recognition

The dual stream-based human action recognition model offers the advantage of high recognition accuracy, but the algorithm is less robust in case of lighting changes. The human skeleton has a strong ability to express human behavior and actions; however, the scene information is ignored. Drawing on the idea of the dual-stream model, this paper proposes a human skeleton and scene image-based dual-stream model for human action recognition. The motion features are extracted through the spatio-temporal graph convolution of the human skeleton, and a scene recognition model is proposed based on the sparse frame sampling of video and video-level consensus strategy to process the scene video and gather the visual scene information. The proposed model exploits the advantages of skeleton information in motion expression and the superiority of the image in scene presentation. The scene information and spatio-temporal graph convolution-based human skeleton limbs are fused complementarily to achieve human action recognition. Compared to the conventional optical flow-based dual-stream action recognition method, this model is verified by experimenting under unstable light conditions, and the performance of human action recognition is robust and promising.

Neural Style Transfer Enhanced Training Support For Human Activity Recognition

This work presents an application of Integrated sensing and communication (ISAC) system for monitoring human activities directly related to healthcare. Real-time monitoring of humans can assist professionals in providing healthy living enabling technologies to ensure the health, safety, and well-being of people of all age groups. To enhance the human activity recognition performance of the ISAC system, we propose to use synthetic data generated through our human micro-Doppler simulator, SimHumalator to augment our limited measurement data. We generate a more realistic micro-Doppler signature dataset using a style-transfer neural network. The proposed network extracts environmental effects such as noise, multipath, and occlusions effects directly from the measurement data and transfers these features to our clean simulated signatures. This results in more realistic-looking signatures qualitatively and quantitatively. We use these enhanced signatures to augment our measurement data and observe an improvement in the classification performance by 5% compared to no augmentation case. Further, we benchmark the data augmentation performance of the style transferred signatures with three other synthetic datasets -- clean simulated spectrograms (no environmental effects), simulated data with added AWGN noise, and simulated data with GAN generated noise. The results indicate that style transferred simulated signatures well captures environmental factors more than any other synthetic dataset.

Comparative Assessment of Forest Optimization with Deep Ensembling Technique for Human Activity Recognition based on Data Collected from Wearable Devices

Aim: Improving the performance of Human Activity Recognition based on information sensed by wearable devices. Materials and methods: In this study we have considered two groups namely forest optimization with sample size of 1408 and deep ensemble techniques with sample size 1408 (Kane, Phar, and BCPS n.d.). Accuracy was computed for the dataset size of 9673 to recognise six various human activities (walking, jogging, standing, upstairs, sitting, downstairs). Result: It was observed that the forest optimization algorithm obtains accuracy as 96% and loss as 14%. Forest optimization technique appears to have better significance than deep ensemble technique with value of p=0.000. Conclusion: The result proves that forest optimization approaches with varying seed value have significant improvement in human activity recognition.

Arbitrary-view human action recognition via novel-view action generation

Arbitrary-view human action recognition is still a big challenge due to the view changes. A possible solution is to enlarge the view range of action samples in the training set. Therefore, we propose a Two-Branch Novel-View action Generation approach based on auxiliary conditional GAN, which generates a novel-view action sample for arbitrary-view human action recognition. The generated sample enlarge the view range of action samples for training. Furthermore, to narrow the representation of actions in different views, we propose a view-domain generalization model that improves the recognition performance of arbitrary-view human action recognition. Our approach is evaluated on three large-scale RGB+D skeleton datasets including UESTC varying-view RGB+D dataset, NTU RGB+D 60, and NTU RGB+D 120 datasets, with two types of view-invariant evaluations, i.e., the cross-view, and arbitrary-view recognition. The proposed approach achieves outstanding performance in human action recognition.

Synthetic Humans for Action Recognition from Unseen Viewpoints

Although synthetic training data has been shown to be beneficial for tasks such as human pose estimation, its use for RGB human action recognition is relatively unexplored. Our goal in this work is to answer the question whether synthetic humans can improve the performance of human action recognition, with a particular focus on generalization to unseen viewpoints. We make use of the recent advances in monocular 3D human body reconstruction from real action sequences to automatically render synthetic training videos for the action labels. We make the following contributions: (1) we investigate the extent of variations and augmentations that are beneficial to improving performance at new viewpoints. We consider changes in body shape and clothing for individuals, as well as more action relevant augmentations such as non-uniform frame sampling, and interpolating between the motion of individuals performing the same action; (2) We introduce a new data generation methodology, SURREACT, that allows training of spatio-temporal CNNs for action classification; (3) We substantially improve the state-of-the-art action recognition performance on the NTU RGB+D and UESTC standard human action multi-view benchmarks; Finally, (4) we extend the augmentation approach to in-the-wild videos from a subset of the Kinetics dataset to investigate the case when only one-shot training data is available, and demonstrate improvements in this case as well.