Human Activity Classification Research Articles

Suppressive Control of Incentive Salience in Real-World Human Vision.

Reward-related activity in the dopaminergic midbrain is thought to guide animal behaviour, in part by boosting the perceptual and attentional processing of reward-predictive environmental stimuli. In line with this incentive salience hypothesis, studies of human visual search have shown that simple synthetic stimuli - like lines, shapes, or Gabor patches - capture attention to their location when they are characterized by reward-associated visual features like colour. In the real world, however, we commonly search for members of a category of visually-heterogenous objects - like people, cars, or trees - where category examples do not share low-level features. Is attention captured to examples of a reward-associated real-world object category? Here, we have human participants search for targets in photographs of city- and landscapes that contain task-irrelevant examples of a reward-associated category. We use the temporal precision of EEG machine learning and ERPs to show that these distractors acquire incentive salience and draw attention, but do not capture it. Instead, we find evidence of rapid, stimulus-triggered attentional suppression, such that the neural encoding of these objects is degraded relative to neutral objects. Humans appear able to suppress the incentive salience of reward-associated objects when they know these objects will be irrelevant, supporting the rapid deployment of attention to other objects that might be more useful. Incentive salience is thought to underlie key behaviours in eating disorders and addiction, among other conditions, and the kind of suppression identified here likely plays a role in mediating the attentional biases that emerge in these circumstances.Significance StatementLike other animals, humans are prone to notice and interact with environmental objects that have proven rewarding in earlier experience. However, it is common that such objects have no immediate strategic use and are therefore distracting. Do these reward-associated real-world objects capture our attention, despite our strategic efforts otherwise? Or are we able to strategically control the impulse to notice them? Here we use machine learning classification of human electrical brain activity to show that we can establish strategic control over the salience of naturalistic reward-associated objects. These objects draw our attention, but do not necessarily capture it, and this kind of control may play an important role in mediating conditions like eating disorder and addiction.

Inside Front Cover

This cover image demonstates an artificial intelligence‐enabled piezo‐yarn device developed by Kim et al. for wearable intelligent sensing. It utilizes BaTiO3‐doped P(VDF‐TrFE) fibers, yielding remarkable shape adaptability, enhanced piezoelectric performance, and mechanical robustness. The AI yarn device enables real‐time monitoring and classification of human motion activities, achieving 99.6% accuracy using a convolutional neural network with multi‐kernel algorithms. image

Human Interaction Classification in Sliding Video Windows Using Skeleton Data Tracking and Feature Extraction.

A "long short-term memory" (LSTM)-based human activity classifier is presented for skeleton data estimated in video frames. A strong feature engineering step precedes the deep neural network processing. The video was analyzed in short-time chunks created by a sliding window. A fixed number of video frames was selected for every chunk and human skeletons were estimated using dedicated software, such as OpenPose or HRNet. The skeleton data for a given window were collected, analyzed, and eventually corrected. A knowledge-aware feature extraction from the corrected skeletons was performed. A deep network model was trained and applied for two-person interaction classification. Three network architectures were developed-single-, double- and triple-channel LSTM networks-and were experimentally evaluated on the interaction subset of the "NTU RGB+D" data set. The most efficient model achieved an interaction classification accuracy of 96%. This performance was compared with the best reported solutions for this set, based on "adaptive graph convolutional networks" (AGCN) and "3D convolutional networks" (e.g., OpenConv3D). The sliding-window strategy was cross-validated on the "UT-Interaction" data set, containing long video clips with many changing interactions. We concluded that a two-step approach to skeleton-based human activity classification (a skeleton feature engineering step followed by a deep neural network model) represents a practical tradeoff between accuracy and computational complexity, due to an early correction of imperfect skeleton data and a knowledge-aware extraction of relational features from the skeletons.

A large-scale fMRI dataset for human action recognition

Human action recognition is a critical capability for our survival, allowing us to interact easily with the environment and others in everyday life. Although the neural basis of action recognition has been widely studied using a few action categories from simple contexts as stimuli, how the human brain recognizes diverse human actions in real-world environments still needs to be explored. Here, we present the Human Action Dataset (HAD), a large-scale functional magnetic resonance imaging (fMRI) dataset for human action recognition. HAD contains fMRI responses to 21,600 video clips from 30 participants. The video clips encompass 180 human action categories and offer a comprehensive coverage of complex activities in daily life. We demonstrate that the data are reliable within and across participants and, notably, capture rich representation information of the observed human actions. This extensive dataset, with its vast number of action categories and exemplars, has the potential to deepen our understanding of human action recognition in natural environments.

Robust and Efficient Authentication and Group-Proof Scheme Using Physical Unclonable Functions for Wearable Computing.

Wearable computing has garnered a lot of attention due to its various advantages, including automatic recognition and categorization of human actions from sensor data. However, wearable computing environments can be fragile to cyber security attacks since adversaries attempt to block, delete, or intercept the exchanged information via insecure communication channels. In addition to cyber security attacks, wearable sensor devices cannot resist physical threats since they are batched in unattended circumstances. Furthermore, existing schemes are not suited for resource-constrained wearable sensor devices with regard to communication and computational costs and are inefficient regarding the verification of multiple sensor devices simultaneously. Thus, we designed an efficient and robust authentication and group-proof scheme using physical unclonable functions (PUFs) for wearable computing, denoted as AGPS-PUFs, to provide high-security and cost-effective efficiency compared to the previous schemes. We evaluated the security of the AGPS-PUF using a formal security analysis, including the ROR Oracle model and AVISPA. We carried out the testbed experiments using MIRACL on Raspberry PI4 and then presented a comparative analysis of the performance between the AGPS-PUF scheme and the previous schemes. Consequently, the AGPS-PUF offers superior security and efficiency than existing schemes and can be applied to practical wearable computing environments.

Predicting Human Activity – State of the Art

Predicting human actions is a very actual research field. Artificial intelligence methods are commonly used here. They enable early recognition and classification of human activities. Such knowledge is extremely needed in the work on robots and other interactive systems that communicate and cooperate with people. This ensures early reactions of such devices and proper planning of their future actions. However, due to the complexity of human actions, predicting them is a difficult task. In this article, we review state-of-the-art methods and summarize recent advances in predicting human activity. We focus in particular on four approaches using machine learning methods, namely methods using: artificial neural networks, support vector machines, probabilistic models and decision trees. We discuss the advantages and disadvantages of these approaches, as well as current challenges related to predicting human activity. In addition, we describe the types of sensors and data sets commonly used in research on predicting and recognizing human actions. We analyze the quality of the methods used, based on the prediction accuracy reported in scientific articles. We describe the importance of the data type and the parameters of machine learning models. Finally, we summarize the latest research trends. The article is intended to help in choosing the right method of predicting human activity, along with an indication of the tools and resources necessary to effectively achieve this goal.

Contactless monitoring of human behaviors in bed using RSSI signals.

In this paper, contactless monitoring and classification of human activities and sleeping postures in bed using radio signals is presented. The major contribution of this work is the development of a contactless monitoring and classification system with a proposed framework that uses received signal strength indicator (RSSI) signals collected from only one wireless link, where different human activities and sleep postures, including (a) no one in the bed, (b) a man sitting on the bed, (c) sleeping on his back, (d) seizuresleeping, and (e) sleeping on his side, are tested. With our proposed system, there is no need to attach any sensors or medical devices to the human body or the bed. That is the limitation of the sensor-based technology. Additionally, our system does not raise a privacy concern, which is the major limitation of vision-based technology. Experiments using low-cost, low-power 2.4GHz IEEE802.15.4 wireless networks have been conducted in laboratories. Results demonstrate that the proposed system can automatically monitor and classify human sleeping postures in real time. The average classification accuracy of activities and sleep postures obtained from different subjects, test environments, and hardware platforms is 99.92%, 98.87%, 98.01%, 87.57%, and 95.87% for cases (a) to (e), respectively. Here, the proposed system provides an average accuracy of 96.05%. Furthermore, the system can also monitor and separate the difference between the cases of the man falling from his bed and the man getting out of his bed. This autonomous system and sleep posture information can thus be used to support care people, physicians, and medical staffs in the evaluation and planning of treatment for the benefit of patients and related people. The proposed system for non-invasive monitoring and classification of human activities and sleeping postures in bed using RSSI signals.

Electroencephalography Signal Analysis for Human Activities Classification: A Solution Based on Machine Learning and Motor Imagery.

Electroencephalography (EEG) is a fundamental tool for understanding the brain's electrical activity related to human motor activities. Brain-Computer Interface (BCI) uses such electrical activity to develop assistive technologies, especially those directed at people with physical disabilities. However, extracting signal features and patterns is still complex, sometimes delegated to machine learning (ML) algorithms. Therefore, this work aims to develop a ML based on the Random Forest algorithm to classify EEG signals from subjects performing real and imagery motor activities. The interpretation and correct classification of EEG signals allow the development of tools controlled by cognitive processes. We evaluated our ML Random Forest algorithm using a consumer and a research-grade EEG system. Random Forest efficiently distinguishes imagery and real activities and defines the related body part, even with consumer-grade EEG. However, interpersonal variability of the EEG signals negatively affects the classification process.

The Human Activity Radar Challenge: benchmarking based on the 'Radar signatures of human activities' dataset from Glasgow University.

Radar is an extremely valuable sensing technology for detecting moving targets and measuring their range, velocity, and angular positions. When people are monitored at home, radar is more likely to be accepted by end-users, as they already use WiFi, is perceived as privacy-preserving compared to cameras, and does not require user compliance as wearable sensors do. Furthermore, it is not affected by lighting condi-tions nor requires artificial lights that could cause discomfort in the home environment. So, radar-based human activities classification in the context of assisted living can empower an aging society to live at home independently longer. However, challenges remain as to the formulation of the most effective algorithms for radar-based human activities classification and their validation. To promote the exploration and cross-evaluation of different algorithms, our dataset released in 2019 was used to benchmark various classification approaches. The challenge was open from February 2020 to December 2020. A total of 23 organizations worldwide, forming 12 teams from academia and industry, participated in the inaugural Radar Challenge, and submitted 188 valid entries to the challenge. This paper presents an overview and evaluation of the approaches used for all primary contributions in this inaugural challenge. The proposed algorithms are summarized, and the main parameters affecting their performances are analyzed.

Human Activity Classification Based on Dual Micro-Motion Signatures Using Interferometric Radar

Micro-Doppler signatures obtained from the Doppler radar are generally used for human activity classification. However, if the angle between the direction of motion and radar antenna broadside is greater than 60°, the micro-Doppler signatures generated by the radial motion of human body reduce significantly, thereby degrading the performance of the classification algorithm. For the accurate classification of different human activities irrespective of trajectory, we propose a new algorithm based on dual micro-motion signatures, namely, the micro-Doppler and interferometric micro-motion signatures, using an interferometric radar. First, the motion of different parts of the human body is simulated using motion capture (MOCAP) data, which is further utilized for radar echo signal generation. Second, time-varying Doppler and interferometric spectrograms obtained from time-frequency analysis of a single Doppler receiver and interferometric output data, respectively, are fed as input to the deep convolutional neural network (DCNN) for feature extraction and the training/testing process. The performance of the proposed algorithm is analyzed and compared with a micro-Doppler signatures-based classifier. Results show that a dual micro-motion-based DCNN classifier using an interferometric radar is capable of classifying different human activities with an accuracy level of 98%, where Doppler signatures diminish considerably, providing insufficient information for classification. Verification of the proposed classification algorithm based on dual micro-motion signatures is also performed using a real radar test dataset of different human walking patterns, and a classification accuracy level of approximately 90% is achieved.

Design of cuckoo search optimization with deep belief network for human activity recognition and classification

Design of cuckoo search optimization with deep belief network for human activity recognition and classification

Human activity classification using deep learning based on 3D motion feature

Human activity classification is needed to support various fields. The health sector, for example, requires the ability to monitor the activities of patients, the elderly, or people with special needs to provide services with fast response as needed. In the traditional classification model, the steps taken to start from the input of data and then proceed with feature extraction, representation, classifier and end with semantic labels. The classification stage uses Convolutional Neural Network (CNN) deep learning to data input, CNN, and semantic labels. This paper proposes a novel method of classifying nine activities based on the movement features of changes in joint distance using Euclidean on the order of frames in each activity segment as input to the CNN model. This study’s motion feature extraction technique was tested using various window sizes to obtain the best classification accuracy. The experimental results show that the selection of window size 16 on the motion feature setting will produce an optimal model accuracy of 94.08% in classifying human activities.

Activity recognition using a combination of high gain observer and deep learning computer vision algorithms

Inertial sensors have become increasingly popular in human activity classification due to their ease of use and affordability. This paper proposes a novel algorithm for human activity recognition that is a combination of a high-gain observer and deep learning computer vision classification algorithms. The nonlinear high-gain observer designed using Lyapunov analysis accurately estimates the attitude of the chest of a human subject using measurements from a single Inertial Measurement Unit (IMU). The signals processed by the observer are then converted into spectrograms to obtain “images” of the frequency response of the signals. The images for activities from a dataset of 7 human subjects are annotated and used for training/ fine-tuning of several well-known deep learning algorithms for image processing. The results from the best combination of our algorithms shows an exceptional accuracy of 98% for activity recognition. Using deep learning computer vision algorithms, this paper shows how to perform transfer learning from networks pre-trained on millions of images, thus showing how we can train a powerful deep learning network for activity recognition even with just small datasets. The algorithm that uses the high gain observer is shown to perform significantly better than an algorithm based on raw accelerometer and gyro signals.

Development of a Novel Lightweight CNN Model for Classification of Human Actions in UAV-Captured Videos

There has been increased attention paid to autonomous unmanned aerial vehicles (UAVs) recently because of their usage in several fields. Human action recognition (HAR) in UAV videos plays an important role in various real-life applications. Although HAR using UAV frames has not received much attention from researchers to date, it is still a significant area that needs further study because of its relevance for the development of efficient algorithms for autonomous drone surveillance. Current deep-learning models for HAR have limitations, such as large weight parameters and slow inference speeds, which make them unsuitable for practical applications that require fast and accurate detection of unusual human actions. In response to this problem, this paper presents a new deep-learning model based on depthwise separable convolutions that has been designed to be lightweight. Other parts of the HarNet model comprised convolutional, rectified linear unit, dropout, pooling, padding, and dense blocks. The effectiveness of the model has been tested using the publicly available UCF-ARG dataset. The proposed model, called HarNet, has enhanced the rate of successful classification. Each unit of frame data was pre-processed one by one by different computer vision methods before it was incorporated into the HarNet model. The proposed model, which has a compact architecture with just 2.2 million parameters, obtained a 96.15% success rate in classification, outperforming the MobileNet, Xception, DenseNet201, Inception-ResNetV2, VGG-16, and VGG-19 models on the same dataset. The proposed model had numerous key advantages, including low complexity, a small number of parameters, and high classification performance. The outcomes of this paper showed that the model’s performance was superior to that of other models that used the UCF-ARG dataset.

Artificial Intelligence Based Approach for Classification of Human Activities Using MEMS Sensors Data.

The integration of Micro Electronic Mechanical Systems (MEMS) sensor technology in smartphones has greatly improved the capability for Human Activity Recognition (HAR). By utilizing Machine Learning (ML) techniques and data from these sensors, various human motion activities can be classified. This study performed experiments and compiled a large dataset of nine daily activities, including Laying Down, Stationary, Walking, Brisk Walking, Running, Stairs-Up, Stairs-Down, Squatting, and Cycling. Several ML models, such as Decision Tree Classifier, Random Forest Classifier, K Neighbors Classifier, Multinomial Logistic Regression, Gaussian Naive Bayes, and Support Vector Machine, were trained on sensor data collected from accelerometer, gyroscope, and magnetometer embedded in smartphones and wearable devices. The highest test accuracy of 95% was achieved using the random forest algorithm. Additionally, a custom-built Bidirectional Long-Short-Term Memory (Bi-LSTM) model, a type of Recurrent Neural Network (RNN), was proposed and yielded an improved test accuracy of 98.1%. This approach differs from traditional algorithmic-based human activity detection used in current wearable technologies, resulting in improved accuracy.

Wrapper-based deep feature optimization for activity recognition in the wearable sensor networks of healthcare systems

The Human Activity Recognition (HAR) problem leverages pattern recognition to classify physical human activities as they are captured by several sensor modalities. Remote monitoring of an individual’s activities has gained importance due to the reduction in travel and physical activities during the pandemic. Research on HAR enables one person to either remotely monitor or recognize another person’s activity via the ubiquitous mobile device or by using sensor-based Internet of Things (IoT). Our proposed work focuses on the accurate classification of daily human activities from both accelerometer and gyroscope sensor data after converting into spectrogram images. The feature extraction process follows by leveraging the pre-trained weights of two popular and efficient transfer learning convolutional neural network models. Finally, a wrapper-based feature selection method has been employed for selecting the optimal feature subset that both reduces the training time and improves the final classification performance. The proposed HAR model has been tested on the three benchmark datasets namely, HARTH, KU-HAR and HuGaDB and has achieved 88.89%, 97.97% and 93.82% respectively on these datasets. It is to be noted that the proposed HAR model achieves an improvement of about 21%, 20% and 6% in the overall classification accuracies while utilizing only 52%, 45% and 60% of the original feature set for HuGaDB, KU-HAR and HARTH datasets respectively. This proves the effectiveness of our proposed wrapper-based feature selection HAR methodology.

Parallel LSTM-CNN Network With Radar Multispectrogram for Human Activity Recognition

Owing to the ability of automatic feature extraction, deep learning is extensively applied in human activity recognition (HAR) based on radar. In this article, a recognition method based on a multispectrogram and deep-learning model is proposed. The radar echo data are transformed into spectrograms with different feature expressions by performing three time–frequency analyses, including short-time Fourier transform (STFT), reduced interference distribution with Hanning kernel (RIDHK), and smoothed pseudo-Wigner–Ville distribution (SPWVD), and then the spectrograms are sent to deep-learning model to realize HAR. The model employs three long short-term memory (LSTM) networks to learn the temporal features of the spectrograms and uses a 3-D convolutional neural network (3-DCNN) to extract the independent spatial features of a single spectrogram and the spatial correlation features between different kinds of spectrograms, named parallel LSTM-CNN network (PLCN). The PLCN is capable of effectively improving the degree of feature utilization in the spectrogram and the accuracy of human activity classification by fusing both temporal and spatial features. The final experimental results show that the average recognition accuracy of the method based on the PLCN for eight human activities reaches 94.75%, which is significantly higher than that of the three parallel LSTMs and the single 3-DCNN.

Proposed spatio‐temporal features for human activity classification using ensemble classification model

SummaryClassifying human actions from still images or video sequences is a demanding task owing to issues, like lighting, backdrop clutter, variations in scale, partial occlusion, viewpoint, and appearance. A lot of appliances, together with video systems, human–computer interfaces, and surveillance necessitate a compound action recognition system. Here, the proposed system develops a novel scheme for HAR. Initially, filtering as well as background subtraction is done during preprocessing. Then, the features including local binary pattern (LBP), bag of the virtual word (BOW), and the proposed local spatio‐temporal features are extracted. Then, in the recognition phase, an ensemble classification model is introduced that includes Recurrent Neural networks (RNN 1 and RNN 2) and Multi‐Layer Perceptron (MLP 1 and MLP 2). The features are classified using RNN 1 and RNN 2, and the outputs from RNN 1 and RNN 2 are further classified using MLP 1 and MLP 2, respectively. Finally, the outputs attained from MLP 1 and MLP 2 are averaged and the final classified output is obtained. At last, the superiority of the developed approach is proved on varied measures.

Survey on Exact kNN Queries over High-Dimensional Data Space.

k nearest neighbours (kNN) queries are fundamental in many applications, ranging from data mining, recommendation system and Internet of Things, to Industry 4.0 framework applications. In mining, specifically, it can be used for the classification of human activities, iterative closest point registration and pattern recognition and has also been helpful for intrusion detection systems and fault detection. Due to the importance of kNN queries, many algorithms have been proposed in the literature, for both static and dynamic data. In this paper, we focus on exact kNN queries and present a comprehensive survey of exact kNN queries. In particular, we study two fundamental types of exact kNN queries: the kNN Search queries and the kNN Join queries. Our survey focuses on exact approaches over high-dimensional data space, which covers 20 kNN Search methods and 9 kNN Join methods. To the best of our knowledge, this is the first work of a comprehensive survey of exact kNN queries over high-dimensional datasets. We specifically categorise the algorithms based on indexing strategies, data and space partitioning strategies, clustering techniques and the computing paradigm. We provide useful insights for the evolution of approaches based on the various categorisation factors, as well as the possibility of further expansion. Lastly, we discuss some open challenges and future research directions.

GAN-Based Radar Spectrogram Augmentation via Diversity Injection Strategy

The classification of human activity using radar has gained considerable attention in recent years because of the radar sensor’s resistance to harsh settings. However, when using machine learning algorithms to train a radar-based classifier, a substantial amount of training data is always required, which means that the time-consuming and difficult radar measurements should be a prerequisite activity in most scenarios. The purpose of this article is to present a novel data augmentation methodology for generating sufficient and diverse training data for human activity classification, hence alleviating the reliance on complex radar measurements. It is worth noting that the proposed model only uses one spectrogram as a reference for spectrogram augmentation, which demonstrates its great potential in practical scenarios. Considering the contingent risk of a lack of diversity in augmented samples, we develop an elaborate strategy for injecting diversity into augmented samples using external counterparts. To validate our model, we conduct radar simulations and measurements to create a variety of datasets. Our model outperforms other comparable models, demonstrating its great potential in improving the performance of human activity classification.