Human Activity Recognition Dataset Research Articles

A benchmark for domain adaptation and generalization in smartphone-based human activity recognition

Human activity recognition (HAR) using smartphone inertial sensors, like accelerometers and gyroscopes, enhances smartphones’ adaptability and user experience. Data distribution from these sensors is affected by several factors including sensor hardware, software, device placement, user demographics, terrain, and more. Most datasets focus on providing variability in user and (sometimes) device placement, limiting domain adaptation and generalization studies. Consequently, models trained on one dataset often perform poorly on others. Despite many publicly available HAR datasets, cross-dataset generalization remains challenging due to data format incompatibilities, such as differences in measurement units, sampling rates, and label encoding. Hence, we introduce the DAGHAR benchmark, a curated collection of datasets for domain adaptation and generalization studies in smartphone-based HAR. We standardized six datasets in terms of accelerometer units, sampling rate, gravity component, activity labels, user partitioning, and time window size, removing trivial biases while preserving intrinsic differences. This enables controlled evaluation of model generalization capabilities. Additionally, we provide baseline performance metrics from state-of-the-art machine learning models, crucial for comprehensive evaluations of generalization in HAR tasks.

3D skeleton aware driver behavior recognition framework for autonomous driving system

The recognition of the driver’s behaviors inside an autonomous vehicle can effectively address emergency handling in autonomous driving and is crucial for ensuring the driver’s safety. Driver behavior recognition is a challenging task due to factors such as variations, diversities, complexities, and strong interferences in behaviors. In this paper, to realize the application in the autonomous driving scenes, a novel 3D skeleton aware behavior recognition framework is proposed to recognize various driver behaviors in autonomous driving systems. First, a 3D human pose estimation network (Pose-GTFNet) with temporal Transformer and spatial graph convolutional network (GCN) is designed to infer 3D human poses from 2D pose sequences. Second, based on the obtained 3D human pose sequences, a behavior recognition network (Beh-MSFNet) with multi-skeleton feature fusion is designed to recognize driver behaviors. In the experiments, the Pose-GTFNet and Beh-MSFNet can get the best performance compared with most state-of-the-art (SOTA) methods on the Human3.6M human pose dataset, JHMDB and SHREC action recognition dataset, respectively. In addition, the proposed driver behavior recognition framework can achieve SOTA performance on the Drive&Act and Driver-Skeleton driver behavior datasets.

ASK-HAR: Attention-Based Multi-Core Selective Kernel Convolution Network for human activity recognition

Human Activity Recognition (HAR) is an increasingly popular field of study aimed at automatically identifying and categorizing human movements and activities using various tracking devices and measurements, such as sensors and cameras. Smartphones have emerged as a prevalent sensor modality for HAR, offering abundant data on an individual’s movements through GPS, accelerometers, and gyroscopes. In conventional convolutional neural networks (CNNs), the artificial neurons within each feature layer typically possess identical receptive fields (RFs). This paper introduces a novel model, ASK-HAR (Attention-based Multi-Core Selective Kernel Convolution Network for HAR), which enhances HAR by performing kernel selection between multiple branches with different RFs using attention mechanisms. The selective kernel mechanism is leveraged to optimize HAR performance. Additionally, the CBAM attention module is employed for time series feature extraction and activity recognition within the overall framework. Extensive experiments conducted on benchmark HAR datasets, including UCI-HAR, USC-HAD, WISDM, PAMAP2, and DSADS, demonstrate that the ASK-HAR model consistently achieves high accuracy across all datasets.

CAPTURE-24: A large dataset of wrist-worn activity tracker data collected in the wild for human activity recognition

Existing activity tracker datasets for human activity recognition are typically obtained by having participants perform predefined activities in an enclosed environment under supervision. This results in small datasets with a limited number of activities and heterogeneity, lacking the mixed and nuanced movements normally found in free-living scenarios. As such, models trained on laboratory-style datasets may not generalise out of sample. To address this problem, we introduce a new dataset involving wrist-worn accelerometers, wearable cameras, and sleep diaries, enabling data collection for over 24 hours in a free-living setting. The result is CAPTURE-24, a large activity tracker dataset collected in the wild from 151 participants, amounting to 3883 hours of accelerometer data, of which 2562 hours are annotated. CAPTURE-24 is two to three orders of magnitude larger than existing publicly available datasets, which is critical to developing accurate human activity recognition models.

Feature-Level Fusion Multi-Sensor Aggregation Temporal Network for Smartphone-Based Human Activity Recognition

Smartphone-based Human Activity Recognition (HAR) identifies human movements using inertial signals gathered from multiple smartphone sensors. Generally, these signals are stacked as one (data-level fusion) and fed into deep learning algorithms for feature extractions. This research studies feature-level fusion, individually processing inertial signals from each sensor, and proposes a lightweight deep temporal learning model, Feature-Level Fusion Multi-Sensor Aggregation Temporal Network (FLF-MSATN), that performs feature extraction on inertial signals from each sensor separately. The raw signals, segmented into equally sized time windows, are passed into individual Dilated-Pooled Convolutional Heads (DPC Heads) for temporal feature analysis. Each DPC Head has a spatiotemporal block containing dilated causal convolutions and average pooling, to extract underlying patterns. The DPC Heads’ outputs are concatenated and passed into a Global Average Pooling layer to generate a condensed confidence map before activity classification. FLF-MSATN is assessed using a subject-independent protocol on a publicly available HAR dataset, UCI HAR, and a self-collected HAR dataset, achieving 96.67% and 82.70% accuracies, respectively. A Data-Level Fusion MSATN is built to compare and verify the model performance attained by the proposed FLF-MSATN. The empirical results show that implementing FLF-MSATN enhances the accuracy by ~3.4% for UCI HAR and ~9.68% for self-collected datasets.

Improved RNN Model for Real-Time Human Activity Recognition

This study focuses on the automatic detection of human actions in video streams. The requirement to detect what human activities happen in videos is recognition of human action due to significant differences in people's visual and motion appearance and actions, camera perspective shifts, moving background, occlusions, noise, and a massive amount of video data. The human activity recognition challenge involves identifying physical activities carried out by individuals or groups based on traces of movements, including gestures, actions, interactions, and group activities. The detection of concepts usually requires additional annotations for the training dataset. In this paper, useful methods for categorizing human action recognition are discussed. The current models are an accurate deep learning method that is based on models that have been changed to be more useful. The large disparities that result from the backdrop and the size of the objects have prevented the identification of activities in videos from being fully and effectively addressed. The main objective is to achieve better accuracy for the Long Short-Term Memory (LSTM) method, which was used to improve the Recurrent Neural Networks (RNN) model. In this paper, LSTM is used to come up with models for different action recognition tasks. The model was made better by making the LSTM have four layers and putting 128 units, 64 units, 32 units, and 16 units in each layer, respectively. In addition, the performance evaluation of deep learning-based approaches has been compared to other related works. Therefore, an improved approach to RNN is proposed to recognize human actions. To classify the videos, a multilayer RNN with a specific type of LSTM is used to extract features from video sequences. The UCF-101 and UCF Sports human action recognition datasets are utilized in this study for both training and assessment. Test findings demonstrate that the suggested strategy achieved increased accuracy. Finally, the enhanced RNN model's total model accuracy in the UCF-101 dataset is 93.78% and 95.70% for the UCF Sport dataset.

Recognizing human activities using light-weight and effective machine learning methodologies

Background Human activity recognition poses a complex challenge in predicting individuals’ movements from raw sensor data using machine learning models. This paper explores the application of six prominent machine learning techniques – decision tree, random forest, linear regression, Naïve Bayes, k-nearest neighbor, and neural networks – to enhance the accuracy of human activity detection for e-health systems. Despite previous research efforts employing data mining and machine learning, there remains room for improvement in performance. The study focuses on predicting activities such as walking, standing, laying, sitting, walking upstairs, and walking downstairs. Methods The research employs six machine learning algorithms to recognize human activities, including decision tree, random forest, linear regression, Naïve Bayes, k-nearest neighbor, and neural networks. Results Evaluation of the human activity recognition dataset reveals that the random forest classifier, CNN, GRN and neural network yield promising results, achieving high accuracy. However, Naïve Bayes falls short of satisfying outcomes. Conclusions The study successfully classifies activities like SITTING, STANDING, LAYING, WALKING, WALKING_DOWNSTAIRS, and WALKING_UPSTAIRS with a remarkable accuracy of 98%. The contribution lies in the thorough exploration of machine learning techniques, with neural networks emerging as the most effective in enhancing human activity recognition. The findings showcase the potential for advanced applications in e-health systems and beyond.

Revolutionizing health monitoring: Integrating transformer models with multi-head attention for precise human activity recognition using wearable devices.

A daily activity routine is vital for overall health and well-being, supporting physical and mental fitness. Consistent physical activity is linked to a multitude of benefits for the body, mind, and emotions, playing a key role in raising a healthy lifestyle. The use of wearable devices has become essential in the realm of health and fitness, facilitating the monitoring of daily activities. While convolutional neural networks (CNN) have proven effective, challenges remain in quickly adapting to a variety of activities. This study aimed to develop a model for precise recognition of human activities to revolutionize health monitoring by integrating transformer models with multi-head attention for precise human activity recognition using wearable devices. The Human Activity Recognition (HAR) algorithm uses deep learning to classify human activities using spectrogram data. It uses a pretrained convolution neural network (CNN) with a MobileNetV2 model to extract features, a dense residual transformer network (DRTN), and a multi-head multi-level attention architecture (MH-MLA) to capture time-related patterns. The model then blends information from both layers through an adaptive attention mechanism and uses a SoftMax function to provide classification probabilities for various human activities. The integrated approach, combining pretrained CNN with transformer models to create a thorough and effective system for recognizing human activities from spectrogram data, outperformed these methods in various datasets - HARTH, KU-HAR, and HuGaDB produced accuracies of 92.81%, 97.98%, and 95.32%, respectively. This suggests that the integration of diverse methodologies yields good results in capturing nuanced human activities across different activities. The comparison analysis showed that the integrated system consistently performs better for dynamic human activity recognition datasets. In conclusion, maintaining a routine of daily activities is crucial for overall health and well-being. Regular physical activity contributes substantially to a healthy lifestyle, benefiting both the body and the mind. The integration of wearable devices has simplified the monitoring of daily routines. This research introduces an innovative approach to human activity recognition, combining the CNN model with a dense residual transformer network (DRTN) with multi-head multi-level attention (MH-MLA) within the transformer architecture to enhance its capability.

WISNet: A deep neural network based human activity recognition system

Nowadays, Human Activity Recognition (HAR) is a key research area with many ubiquitous innovative solutions, where both accelerometer and gyroscope data provide information about an observed person’s physical activity. HAR offers a diverse variety of important applications, including healthcare, burglary detection, workplace monitoring, and emergency identification. Traditional recognition approaches rely on extracting handmade features from the obtained data to identify the typeof human action. Additionally, the efficacy of these works is dependent upon the specific customized features that are chosen. One potential approach to tackle this issue is to utilize Convolutional Neural Networks (CNN) to automatically learn the relevant features. In this paper, we propose a deep learning model, WISNet, a custom 1D-CNN approach to recognize six complex human activities: Jogging, Walking Downstairs, Sitting, Standing, Walking and Climbing Upstairs. The model includes a Convolved Normalized Pooled (CNPM) Block to generate significant features from the initial layers. An Identity and Basic (IDBN) Block is incorporated to extract residual progressive features for capturing complex sequential data dependencies. Channel and Spatial attention (CASb) Block is integrated with the network to prioritize or minimize essential features based on relative weights. The proposed WISNet model achieved an enhanced accuracy and F1-score of 96.41 % and 0.95 for the HAR dataset by surpassing the existing transfer learning architectures such as Gated Recurrent Units (GRU), Long Short-Term Memory (LSTM), and Recurrent Neural Network (SimpleRNN). By strategically integrating the CNPM, IDBN, and CASb blocks, this study aims to tackle distinct challenges encountered in the classification process by enhancing the discernment of features essential for the precise identification of multi-class human activity recognition. The seamless integration of these blocks within the model plays a pivotal role in elevating the overall performance of the WISNet architecture. The work also validates WISNet with similar open-source datasets (UCI-HAR and KU-HAR) and dissimilar open-source datasets (Sleep state detection, Fall detection, and ECG Heartbeat).

Interpretability in Video-based Human Action Recognition: Saliency Maps and GradCAM in 3D Convolutional Neural Networks

Interpretability plays a vital role in understanding complex deep learning models by providing transparency and insights. It addresses the black-box nature of these models, aids in human-in-the-loop systems, enhances model development, and supports education. However, existing interpretability algorithms in computer vision primarily target images, leaving a gap for video applications. In this article, we emphasize the importance of interpretability in video-based Human Action Recognition (HAR). We extend existing 2D interpretability techniques to the 3D domain, specifically focusing on saliency maps and gradient-weighted class activation maps. The proposed interpretability system is then employed in the analysis of a well-known HAR dataset to better understand action recognition in videos.

Video Surveillance System-Based Human Activity Recognition Using Hierarchical Auto-Associative Polynomial Convolutional Neural Network with Garra Rufa Fish Optimization

Nowadays, video surveillance systems are gaining increasing attention in the computer vision fields due to user demands for security purposes. Observing human movement and predicting such movement perception are promising due to video surveillance systems. In this work, Hierarchical Auto-Associative Polynomial Convolutional Neural Networks (HA-PCNN) with Garran Rufa Fish Optimization (GRFO) is proposed for Human Activity Recognition (HAR) using a video surveillance system. Initially, video surveillance systems-based human activity images are obtained, which are collected by video camera by recording daily human activities. After that, the input images are fed to a pre-processing approach named as Switched Mode Fuzzy Median Filter (SMFM) method. In this, the noise presented in the input images is reduced by applying the SMFM model to normalize the dataset and improve the image quality. After that, the pre-processed images are given to the developed Fast Discrete Curvelet Transform with Wrapping (FDCT-WRP)-based feature extraction method to extract the relevant features. Moreover, the extracted features are given to the HA-PCNN model for HA classification. Generally, the HA-PCNN method does not express any adoption of optimization methods for scaling the ideal parameters and classification. Here, Garra Rufa Fish Optimization (GRFO) is proposed to enhance the parameters of HA-PCNN. Thus, the proposed HA-PCNN-GRFO methodology can classify human activities like standing, sitting, running, walking and sleeping. By then the performance of the proposed HA-PCNN-GRFO methodology is evaluated using Python Platform, and metrics like recall, accuracy, F-measure, precision, and specificity are analyzed. Thus, the proposed HA-PCNN-GRFO approach achieved 97.3% average accuracy rate, 96.5% average recall, 97.2% average precision, 95.9% average specificity, and 96.8% average F1-score for classifying human activities using UCI HAR dataset. Also, the proposed HA-PCNN-GRFO approach is outperformed in HAR than the conventional approaches.

Deep generative domain adaptation with temporal relation attention mechanism for cross-user activity recognition

In sensor-based Human Activity Recognition (HAR), a predominant assumption is that the data utilized for training and evaluation purposes are drawn from the same distribution. It is also assumed that all data samples are independent and identically distributed (i.i.d.). Contrarily, practical implementations often challenge this notion, manifesting data distribution discrepancies, especially in scenarios such as cross-user HAR. Domain adaptation is the promising approach to address these challenges inherent in cross-user HAR tasks. However, a clear gap in domain adaptation techniques is the neglect of the temporal dependency relation embedded within time series data during the phase of aligning data distributions. Addressing this oversight, our research presents the Deep Generative Domain Adaptation with Temporal Attention (DGDATA) method. This novel method uniquely recognizes and integrates temporal dependency relations during the domain adaptation process. By synergizing the capabilities of generative models with the Temporal Relation Attention mechanism, our method improves the classification performance in cross-user HAR. The evaluation has been conducted on three public sensor-based HAR datasets targeting daily living activity and sports fitness activity scenarios to demonstrate the efficacy of the proposed DGDATA method.

Fall Detection Method for Infrared Videos Based on Spatial-Temporal Graph Convolutional Network.

The timely detection of falls and alerting medical aid is critical for health monitoring in elderly individuals living alone. This paper mainly focuses on issues such as poor adaptability, privacy infringement, and low recognition accuracy associated with traditional visual sensor-based fall detection. We propose an infrared video-based fall detection method utilizing spatial-temporal graph convolutional networks (ST-GCNs) to address these challenges. Our method used fine-tuned AlphaPose to extract 2D human skeleton sequences from infrared videos. Subsequently, the skeleton data was represented in Cartesian and polar coordinates and processed through a two-stream ST-GCN to recognize fall behaviors promptly. To enhance the network's recognition capability for fall actions, we improved the adjacency matrix of graph convolutional units and introduced multi-scale temporal graph convolution units. To facilitate practical deployment, we optimized time window and network depth of the ST-GCN, striking a balance between model accuracy and speed. The experimental results on a proprietary infrared human action recognition dataset demonstrated that our proposed algorithm accurately identifies fall behaviors with the highest accuracy of 96%. Moreover, our algorithm performed robustly, identifying falls in both near-infrared and thermal-infrared videos.

Football teaching and training based on video surveillance using deep learning.

The objective performance evaluation of an athlete is essential to allow detailed research into elite sports. The automatic identification and classification of football teaching and training exercises overcome the shortcomings of manual analytical approaches. Video monitoring is vital in detecting human conduct acts and preventing or reducing inappropriate actions in time. The video's digital material is classified by relevance depending on those individual actions. The research goal is to systematically use the data from an inertial measurement unit (IMU) and data from computer vision analysis for the deep Learning of football teaching motion recognition (DL-FTMR). There has been a search for many libraries. The studies included have examined and analyzed training through profound model construction learning methods. Investigations show the ability to distinguish the efficiency of qualified and less qualified officers for sport-specific video-based decision-making assessments. Video-based research is an effective way of assessing decision-making due to the potential to present changing in-game decision-making scenarios more environmentally friendly than static picture printing. The data showed that the filtering accuracy of responses is improved without losing response time. This observation indicates that practicing with a video monitoring system offers a play view close to that seen in a game scenario. It can be an essential way to improve the perception of selection precision. This study discusses publicly accessible training datasets for Human Activity Recognition (HAR) and presents a dataset that combines various components. The study also used the UT-Interaction dataset to identify complex events. Thus, the experimental results of DL-FTMR give a performance ratio of 94.5%, behavior processing ratio of 92.4%, athletes energy level ratio of 92.5%, interaction ratio of 91.8%, prediction ratio of 92.5%, sensitivity ratio of 93.7%, and the precision ratio of 94.86% compared to the optimized convolutional neural network (OCNN), Gaussian Mixture Model (GMM), you only look once (YOLO), Human Activity Recognition- state-of-the-art methodologies (HAR-SAM). This finding proves that exercising a video monitoring system that provides a play view similar to that seen in a game scenario can be a valuable technique to increase selection accuracy perception.

HARWE: A multi-modal large-scale dataset for context-aware human activity recognition in smart working environments

In recent years, deep neural networks (DNNs) have provided high performances for various tasks, such as human activity recognition (HAR), in view of their end-to-end training process between the input data and output labels. However, the performances of the DNNs are highly dependent on the availability of large-scale data for their training processes. In this paper, we propose a novel dataset for the task of HAR, in which the labels are specified for the working environments (WE). Our proposed dataset, namely HARWE, considers multiple signal modalities, including visual signal, audio signal, inertial sensor signals, and biological signals, that are acquired using four different electronic devices. Furthermore, our HARWE dataset is acquired from a large number of participants while considering the realistic disturbances that can occur in the wild. Our HARWE data is context-driven, which means there exist a number of labels in it that even though they are correlated with each other, they have contextual differences. A deep conventional multi-modal neural network provides an accuracy of 99.06% and 68.60%, for the cases of the easy and difficult settings of our dataset, respectively, which indicates its applicability for the task of human activity recognition.

A Spatio-temporal Graph Transformer driven model for recognizing fine-grained data human activity

Human activity recognition(HAR) is an important research focus in ubiquitous computing. It has been widely applied in various domains, such as smart homes, healthcare assistance, and sports training. Accurate human activity recognition directly impacts the performance of downstream tasks. Existing methods for human activity recognition primarily rely on extracting temporal or spatial features from the data. These features are used for the task of human activity recognition. The existing methods mainly use CNN or RNN models to model the Euclidean correlations among spatially adjacent sensors or channels. However, non-Euclidean pairwise correlations among all sensors or channels are even critical for accurate classification, which has been ignored by the existing methods. In this paper, we incorporate fine-grained spatial structural data into the model to overcome these limitations. A novel deep learning model for human activity recognition is proposed, which is called the fine-grained data-oriented Spatial-Temporal Graph Transformer network (STGT). The introduction of the STGT model can eliminate the limitations of existing spatial feature extraction methods by leveraging a novel data organization approach proposed in this study. This model enhances the spatial features within the time feature extraction module for effective spatio-temporal feature extraction. We conducted experiments on four large-scale real-world HAR datasets to evaluate its performance. The experimental results demonstrate the superiority of our method over state-of-the-art approaches.

Hardware-algorithm co-design of Energy Efficient Federated Learning in Quantized Neural Network

The real-world implementation of federated learning (FL) for smart device-based applications necessitates energy-efficient convergent models due to resource-constrained devices. In the literature, deep learning (DL)–based models are frequently trained on FL to obtain high accuracy and quick convergence employing a 32-bit floating point precision level. However, such circumstances are not feasible for devices with limited resources. DL models demand a significant amount of processing power and energy, which has a noticeable negative impact on the environment. Hence, it is necessary to reduce the level of precision in DL models to enhance energy efficiency. In this paper, we propose a low-precision level green federated learning (GFL) model to maintain the balance of communication cycles, energy efficiency, and accuracy with an admissible convergence rate of the deep learning algorithms. Our proposed GFL framework achieves 98.04% server accuracy and 97.69% federated accuracy with a faster convergence rate and fewer communication rounds than state-of-the-art methods on the UCI human activity recognition dataset.

CrossHAR: Generalizing Cross-dataset Human Activity Recognition via Hierarchical Self-Supervised Pretraining

The increasing availability of low-cost wearable devices and smartphones has significantly advanced the field of sensor-based human activity recognition (HAR), attracting considerable research interest. One of the major challenges in HAR is the domain shift problem in cross-dataset activity recognition, which occurs due to variations in users, device types, and sensor placements between the source dataset and the target dataset. Although domain adaptation methods have shown promise, they typically require access to the target dataset during the training process, which might not be practical in some scenarios. To address these issues, we introduce CrossHAR, a new HAR model designed to improve model performance on unseen target datasets. CrossHAR involves three main steps: (i) CrossHAR explores the sensor data generation principle to diversify the data distribution and augment the raw sensor data. (ii) CrossHAR then employs a hierarchical self-supervised pretraining approach with the augmented data to develop a generalizable representation. (iii) Finally, CrossHAR fine-tunes the pretrained model with a small set of labeled data in the source dataset, enhancing its performance in cross-dataset HAR. Our extensive experiments across multiple real-world HAR datasets demonstrate that CrossHAR outperforms current state-of-the-art methods by 10.83% in accuracy, demonstrating its effectiveness in generalizing to unseen target datasets.

Deep Residual Network with a CBAM Mechanism for the Recognition of Symmetric and Asymmetric Human Activity Using Wearable Sensors

Wearable devices are paramount in health monitoring applications since they provide contextual information to identify and recognize human activities. Although sensor-based human activity recognition (HAR) has been thoroughly examined, prior studies have yet to definitively differentiate between symmetric and asymmetric motions. Determining these movement patterns might provide a more profound understanding of assessing physical activity. The main objective of this research is to investigate the use of wearable motion sensors and deep convolutional neural networks in the analysis of symmetric and asymmetric activities. This study provides a new approach for classifying symmetric and asymmetric motions using a deep residual network incorporating channel and spatial convolutional block attention modules (CBAMs). Two publicly accessible benchmark HAR datasets, which consist of inertial measurements obtained from wrist-worn sensors, are used to assess the model’s efficacy. The model we have presented is subjected to thorough examination and demonstrates exceptional accuracy on both datasets. The ablation experiment examination also demonstrates noteworthy contributions from the residual mappings and CBAMs. The significance of recognizing basic movement symmetries in increasing sensor-based activity identification utilizing wearable devices is shown by the enhanced accuracy and F1-score, especially in asymmetric activities. The technique under consideration can provide activity monitoring with enhanced accuracy and detail, offering prospective advantages in diverse domains like customized healthcare, fitness tracking, and rehabilitation progress evaluation.

Modality aware contrastive learning for multimodal human activity recognition

SummaryHuman activity recognition is a well‐established research problem in ubiquitous computing. The increased dependency on various smart devices in our daily lives allows us to investigate the sensor data world produced by multimodal sensors embedded in smart devices. However, the raw sensor data are often unlabeled and annotating this vast amount of data are a costly exercise that can often lead to privacy breaches. Self‐supervised learning‐based approaches are at the forefront of learning semantic representation from unlabeled sensor data, including when applied to human activity recognition tasks. As inferring human activity depends on multimodal sensors, addressing the modality difference and inter‐modality dependencies in a model is an important process. This paper proposes a novel self‐supervised learning approach, modality aware contrastive learning (MACL), for representation learning using multimodal sensor data. The approach uses different sensing modalities to create different views of an input signal. Thus, the model is able to learn the representations by maximizing the similarity among different sensing modalities of the same input signal. Extensive experiments were performed on four publicly available human activity recognition data sets to verify the effectiveness of our proposed MACL method. The experimental evaluation results show that the MACL method attains a comparable performance for human activity recognition to the compared baseline models, directly exceeding the performance of models using standard augmentation transformation strategies.