Multimodal Human Activity Recognition Research Articles

CLEAR: Multimodal Human Activity Recognition via Contrastive Learning Based Feature Extraction Refinement

Human activity recognition (HAR) has become a crucial research area for many applications, such as Healthcare, surveillance, etc. With the development of artificial intelligence (AI) and Internet of Things (IoT), sensor-based HAR has gained increasing attention and presents great advantages to existing work. Relying solely on existing labeled data may not adequately address the challenge of ensuring the model’s generalization ability to new data. The ’CLEAR’ method is designed to improve the accuracy of multimodal human activity recognition. This approach employs data augmentation, multimodal feature fusion, and contrastive learning techniques. These strategies are utilized to refine and extract highly discriminative features from various data sources, thereby significantly enhancing the model’s capacity to identify and classify diverse human activities accurately. CLEAR achieves high generalization performance on unknown datasets using only training data. Furthermore, CLEAR can be directly applied to various target domains without retraining or fine-tuning. Specifically, CLEAR consists of two parts. First, it employs data augmentation techniques in both the time and frequency domains to enrich the training data. Second, it optimizes feature extraction using attention-based multimodal fusion techniques and employs supervised contrastive learning to improve feature discriminability. We achieved accuracy rates of 81.09%, 90.45%, and 82.75% on three public datasets USC-HAD, DSADS, and PAMAP2, respectively. Additionally, when the training data are reduced from 100% to 20%, the model’s accuracy on the three datasets decreases by only about 5%, demonstrating that our model possesses strong generalization capabilities. Additionally, when the training data are reduced from 100% to 20%, the model’s accuracy on the three datasets decreases by only about 5%, demonstrating that our model possesses strong generalization capabilities.

BodyFlow: An Open-Source Library for Multimodal Human Activity Recognition

Human activity recognition is a critical task for various applications across healthcare, sports, security, gaming, and other fields. This paper presents BodyFlow, a comprehensive library that seamlessly integrates human pose estimation and multiple-person estimation and tracking, along with activity recognition modules. BodyFlow enables users to effortlessly identify common activities and 2D/3D body joints from input sources such as videos, image sets, or webcams. Additionally, the library can simultaneously process inertial sensor data, offering users the flexibility to choose their preferred input, thus facilitating multimodal human activity recognition. BodyFlow incorporates state-of-the-art algorithms for 2D and 3D pose estimation and three distinct models for human activity recognition.

Multiscale knowledge distillation with attention based fusion for robust human activity recognition

Knowledge distillation is an effective approach for training robust multi-modal machine learning models when synchronous multimodal data are unavailable. However, traditional knowledge distillation techniques have limitations in comprehensively transferring knowledge across modalities and models. This paper proposes a multiscale knowledge distillation framework to address these limitations. Specifically, we introduce a multiscale semantic graph mapping (SGM) loss function to enable more comprehensive knowledge transfer between teacher and student networks at multiple feature scales. We also design a fusion and tuning (FT) module to fully utilize correlations within and between different data types of the same modality when training teacher networks. Furthermore, we adopt transformer-based backbones to improve feature learning compared to traditional convolutional neural networks. We apply the proposed techniques to multimodal human activity recognition and compared with the baseline method, it improved by 2.31% and 0.29% on the MMAct and UTD-MHAD datasets. Ablation studies validate the necessity of each component.

Modality aware contrastive learning for multimodal human activity recognition

SummaryHuman activity recognition is a well‐established research problem in ubiquitous computing. The increased dependency on various smart devices in our daily lives allows us to investigate the sensor data world produced by multimodal sensors embedded in smart devices. However, the raw sensor data are often unlabeled and annotating this vast amount of data are a costly exercise that can often lead to privacy breaches. Self‐supervised learning‐based approaches are at the forefront of learning semantic representation from unlabeled sensor data, including when applied to human activity recognition tasks. As inferring human activity depends on multimodal sensors, addressing the modality difference and inter‐modality dependencies in a model is an important process. This paper proposes a novel self‐supervised learning approach, modality aware contrastive learning (MACL), for representation learning using multimodal sensor data. The approach uses different sensing modalities to create different views of an input signal. Thus, the model is able to learn the representations by maximizing the similarity among different sensing modalities of the same input signal. Extensive experiments were performed on four publicly available human activity recognition data sets to verify the effectiveness of our proposed MACL method. The experimental evaluation results show that the MACL method attains a comparable performance for human activity recognition to the compared baseline models, directly exceeding the performance of models using standard augmentation transformation strategies.

Deep Wavelet Convolutional Neural Networks for Multimodal Human Activity Recognition Using Wearable Inertial Sensors.

Recent advances in wearable systems have made inertial sensors, such as accelerometers and gyroscopes, compact, lightweight, multimodal, low-cost, and highly accurate. Wearable inertial sensor-based multimodal human activity recognition (HAR) methods utilize the rich sensing data from embedded multimodal sensors to infer human activities. However, existing HAR approaches either rely on domain knowledge or fail to address the time-frequency dependencies of multimodal sensor signals. In this paper, we propose a novel method called deep wavelet convolutional neural networks (DWCNN) designed to learn features from the time-frequency domain and improve accuracy for multimodal HAR. DWCNN introduces a framework that combines continuous wavelet transforms (CWT) with enhanced deep convolutional neural networks (DCNN) to capture the dependencies of sensing signals in the time-frequency domain, thereby enhancing the feature representation ability for multiple wearable inertial sensor-based HAR tasks. Within the CWT, we further propose an algorithm to estimate the wavelet scale parameter. This helps enhance the performance of CWT when computing the time-frequency representation of the input signals. The output of the CWT then serves as input for the proposed DCNN, which consists of residual blocks for extracting features from different modalities and attention blocks for fusing these features of multimodal signals. We conducted extensive experiments on five benchmark HAR datasets: WISDM, UCI-HAR, Heterogeneous, PAMAP2, and UniMiB SHAR. The experimental results demonstrate the superior performance of the proposed model over existing competitors.

MAG-Res2Net: a novel deep learning network for human activity recognition

Objective. Human activity recognition (HAR) has become increasingly important in healthcare, sports, and fitness domains due to its wide range of applications. However, existing deep learning based HAR methods often overlook the challenges posed by the diversity of human activities and data quality, which can make feature extraction difficult. To address these issues, we propose a new neural network model called MAG-Res2Net, which incorporates the Borderline-SMOTE data upsampling algorithm, a loss function combination algorithm based on metric learning, and the Lion optimization algorithm. Approach. We evaluated the proposed method on two commonly utilized public datasets, UCI-HAR and WISDM, and leveraged the CSL-SHARE multimodal human activity recognition dataset for comparison with state-of-the-art models. Main results. On the UCI-HAR dataset, our model achieved accuracy, F1-macro, and F1-weighted scores of 94.44%, 94.38%, and 94.26%, respectively. On the WISDM dataset, the corresponding scores were 98.32%, 97.26%, and 98.42%, respectively. Significance. The proposed MAG-Res2Net model demonstrates robust multimodal performance, with each module successfully enhancing model capabilities. Additionally, our model surpasses current human activity recognition neural networks on both evaluation metrics and training efficiency. Source code of this work is available at: https://github.com/LHY1007/MAG-Res2Net.

ATFA: Adversarial Time–Frequency Attention network for sensor-based multimodal human activity recognition

Human activity recognition (HAR) is a key component of mobile and ubiquitous computing. With the increasing number of sensors embedded in devices such as smartphones and smartwatches, models are enhanced to recognize more complex human behaviors. However, this also leads to increased data heterogeneity, caused by diverse user backgrounds, health conditions, and sensing environments, which makes it difficult to align the multimodal data distributions across individuals. This paper proposes a novel unsupervised domain adaptation framework via Adversarial Time-Frequency Attention (ATFA) to efficiently adapt models to new users. In particular, the proposed attention-based modality fusion module captures and fuses important modalities based on their context to reduce redundant information. Additionally, the network explores frequency domain features to improve performance in recognizing human activities. Extensive experiments are conducted on three publicly available HAR datasets to demonstrate the superiority of our proposed method compared to state-of-the-art baselines.

Multi-level feature fusion for multimodal human activity recognition in Internet of Healthcare Things

Human Activity Recognition (HAR) has become a crucial element for smart healthcare applications due to the fast adoption of wearable sensors and mobile technologies. Most of the existing human activity recognition frameworks deal with a single modality of data that degrades the reliability and recognition accuracy of the system for heterogeneous data sources. In this article, we propose a multi-level feature fusion technique for multimodal human activity recognition using multi-head Convolutional Neural Network (CNN) with Convolution Block Attention Module (CBAM) to process the visual data and Convolutional Long Short Term Memory (ConvLSTM) for dealing with the time-sensitive multi-source sensor information. The architecture is developed to be able to analyze and retrieve channel and spatial dimension features through the use of three branches of CNN along with CBAM for visual information. The ConvLSTM network is designed to capture temporal features from the multiple sensors’ time-series data for efficient activity recognition. An open-access multimodal HAR dataset named UP-Fall detection dataset is utilized in experiments and evaluations to measure the performance of the developed fusion architecture. Finally, we deployed an Internet of Things (IoT) system to test the proposed fusion network in real-world smart healthcare application scenarios. The findings from the experimental results reveal that the developed multimodal HAR framework surpasses the existing state-of-the-art methods in terms of multiple performance metrics.

DanHAR: Dual Attention Network for multimodal human activity recognition using wearable sensors

In the paper, we present a new dual attention method called DanHAR, which blends channel and temporal attention on residual networks to improve feature representation ability for sensor-based HAR task. Specially, the channel attention plays a key role in deciding what to focus, i.e., sensor modalities, while the temporal attention can focus on the target activity from a long sensor sequence to tell where to focus. Extensive experiments are conducted on four public HAR datasets, as well as weakly labeled HAR dataset. The results show that dual attention mechanism is of central importance for many activity recognition tasks. We obtain 2.02%, 4.20%, 1.95%, 5.22% and 5.00% relative improvement over regular ConvNets respectively on WISDM dataset, UNIMIB SHAR dataset, PAMAP2 dataset, OPPORTUNITY dataset, as well as weakly labeled HAR dataset. The DanHAR is able to surpass other state-of-the-art algorithms at negligible computational overhead. Visualizing analysis is conducted to show that the proposed attention can capture the spatial–temporal dependencies of multimodal sensing data, which amplifies the more important sensor modalities and timesteps during classification. The results are in good agreement with normal human intuition.

A review of multimodal human activity recognition with special emphasis on classification, applications, challenges and future directions

Human activity recognition (HAR) is one of the most important and challenging problems in the computer vision. It has critical application in wide variety of tasks including gaming, human–robot interaction, rehabilitation, sports, health monitoring, video surveillance, and robotics. HAR is challenging due to the complex posture made by the human and multiple people interaction. Various artifacts that commonly appears in the scene such as illuminations variations, clutter, occlusions, background diversity further adds the complexity to HAR. Sensors for multiple modalities could be used to overcome some of these inherent challenges. Such sensors could include an RGB-D camera, infrared sensors, thermal cameras, inertial sensors, etc. This article introduces a comprehensive review of different multimodal human activity recognition methods where different types of sensors being used along with their analytical approaches and fusion methods. Further, this article presents classification and discussion of existing work within seven rational aspects: (a) what are the applications of HAR; (b) what are the single and multi-modality sensing for HAR; (c) what are different vision based approaches for HAR; (d) what and how wearable sensors based system contributes to the HAR; (e) what are different multimodal HAR methods; (f) how a combination of vision and wearable inertial sensors based system contributes to the HAR; and (g) challenges and future directions in HAR. With a more and comprehensive understanding of multimodal human activity recognition, more research in this direction can be motivated and refined.

MMHAR-EnsemNet: A Multi-Modal Human Activity Recognition Model

In this article, we propose a new deep learning model named as MMHAR-EnsemNet (Multi-Modal Human Activity Recognition Ensemble Network) which makes use of four different modalities to perform sensor-based Human Activity Recognition (HAR).Two separate Convolutional Neural Networks (CNNs) are made for skeleton data. While one CNN and one LSTM is trained for RGB images. For Accelerometer and Gyroscope data first it is converted to signal diagram then another CNN model is trained. Finally, all the outputs of the said models have been used to form an ensemble so that performance of the HAR model gets improved. The proposed model has been evaluated on two standard benchmark datasets namely UTD-MHAD and Berkeley-MHAD which contain four different modalities of input information. Experimental results confirm that the MMHAR-EnsemNet model has outperformed some recently proposed models considered here for comparison. Source code of this work can be found at: https://github.com/abhi1998das/MMHAREnsemNet.

Bayesian optimization on graph-structured search spaces: Optimizing deep multimodal fusion architectures

Abstract A popular testbed for deep learning has been multimodal recognition of human activity or gesture involving diverse inputs like video, audio, skeletal pose and depth images. Deep learning architectures have excelled on such problems due to their ability to combine modality representations at different levels of nonlinear feature extraction. However, designing an optimal architecture in which to fuse such learned representations has largely been a non-trivial human engineering effort. We treat fusion structure optimization as a hyperparameter search and cast it as a discrete optimization problem under the Bayesian optimization framework. We propose two methods to compute structural similarities in the search space of tree-structured multimodal architectures, and demonstrate their effectiveness on two challenging multimodal human activity recognition problems.