Activity Recognition Research Articles

Deep learning based assembly process action recognition and progress prediction facing human-centric intelligent manufacturing

With the breakthroughs in cutting-edge technologies and the growing demand for personalized services, the approach of full automation alone is no longer sufficient to meet rapidly changing production needs. In light of this, a new human-centric manufacturing (HCM) paradigm has been put forward. Industrial human action recognition, as one of the key means of realizing HCM, has received widespread attention and research in recent years. In this paper, an end-to-end deep learning-based action recognition and progress prediction model was built, which can not only recognize action classes in real-time, but also accurately predict action completion times, to improve the efficiency and safety of HCM systems. First, a non-contact data acquisition system based on Azure Kinect sensor, MediaPipe and You-Only-Look-Once V5 (Yolo V5) algorithm was constructed, achieving comprehensive perception of human and objects in manufacturing system. Second, an Auto-encoder model was established to learn and compress high-dimensional human joint and object data into low-dimensional space to achieve efficient representation of the operation process. Finally, the human action recognition and progress prediction model was built and evaluated in a reducer assembly line, with an accuracy rate for action recognition of 99 %, and a time error for progress prediction of less than 6.9 %. This research aims to provide guidance for improving the level of intelligence in manufacturing and the application of AI technology in the industrial field.

From Recognition to Prediction: Leveraging Sequence Reasoning for Action Anticipation

The action anticipation task refers to predicting what action will happen based on observed videos, which requires the model to have a strong ability to summarize the present and then reason about the future. Experience and common sense suggest that there is a significant correlation between different actions, which provides valuable prior knowledge for the action anticipation task. However, previous methods have not effectively modeled this underlying statistical relationship. To address this issue, we propose a novel end-to-end video modeling architecture that utilizes attention mechanisms, named Anticipation via Recognition and Reasoning (ARR). ARR decomposes the action anticipation task into action recognition and sequence reasoning tasks, and effectively learns the statistical relationship between actions by next action prediction (NAP). In comparison to existing temporal aggregation strategies, ARR is able to extract more effective features from observable videos to make more reasonable predictions. In addition, to address the challenge of relationship modeling that requires extensive training data, we propose an innovative approach for the unsupervised pre-training of the decoder, which leverages the inherent temporal dynamics of video to enhance the reasoning capabilities of the network. Extensive experiments on the Epic-kitchen-100, EGTEA Gaze+, and 50salads datasets demonstrate the efficacy of the proposed methods. The code is available at https://github.com/linuxsino/ARR .

Superclass-aware visual feature disentangling for generalized zero-shot learning

Zero-shot learning (ZSL) aims to learn a model trained on seen samples with the ability to recognize samples from unseen classes, while generalized ZSL (GZSL) takes a step closer to realistic scenarios by recognizing both of seen and unseen samples. The existing methods rely on the semantic descriptions as the side-information and conduct tight alignment between the visual and semantic spaces. However, the tight modality alignment may result in incomplete representations, leading to the loss of originally detailed and discriminative information. In this paper, we propose a simple yet effective superclass-aware visual feature disentangling method termed as SupVFD for GZSL. We use the neighbor relations of the semantic descriptions to define superclass and with the guide of superclass, our method disentangles visual features into discriminative and transferable factors. To this end, the semantic descriptions are used as implicit supervision, which preserves the valuable detailed and discriminative information in the visual features. The extensive experiments in both ZSL and GZSL settings prove our method outperforms the state-of-the-art methods for image object classification as well as video action recognition. Code is available at our github: https://github.com/changniu54/SupVFD-Master.

Holo4Care: a MR framework for assisting in activities of daily living by context-aware action recognition

AbstractThe evolution of virtual and augmented reality devices in recent years has encouraged researchers to develop new systems for different fields. This paper introduces Holo4Care, a context-aware mixed reality framework designed for assisting in activities of daily living (ADL) using the HoloLens 2. By leveraging egocentric cameras embedded in these devices, which offer a close-to-wearer perspective, our framework establishes a congruent relationship, facilitating a deeper understanding of user actions and enabling effective assistance. In our approach, we extend a previously established action estimation architecture after conducting a thorough review of state-of-the-art methods. The proposed architecture utilizes YOLO for hand and object detection, enabling action estimation based on these identified elements. We have trained new models on well-known datasets for object detection, incorporating action recognition annotations. The achieved mean Average Precision (mAP) is 33.2% in the EpicKitchens dataset and 26.4% on the ADL dataset. Leveraging the capabilities of the HoloLens 2, including spatial mapping and 3D hologram display, our system seamlessly presents the output of the action recognition architecture to the user. Unlike previous systems that focus primarily on user evaluation, Holo4Care emphasizes assistance by providing a set of global actions based on the user’s field of view and hand positions that reflect their intentions. Experimental results demonstrate Holo4Care’s ability to assist users in activities of daily living and other domains.

Multihead-Res-SE Residual Network with Attention for Human Activity Recognition

Human activity recognition (HAR) typically uses wearable sensors to identify and analyze the time-series data they collect, enabling recognition of specific actions. As such, HAR is increasingly applied in human–computer interaction, healthcare, and other fields, making accurate and efficient recognition of various human activities. In recent years, deep learning methods have been extensively applied in sensor-based HAR, yielding remarkable results. However, complex HAR research, which involves specific human behaviors in varied contexts, still faces several challenges. To solve these problems, we propose a multi-head neural network based on the attention mechanism. This framework contains three convolutional heads, with each head designed using one-dimensional CNN to extract features from sensory data. The model uses a channel attention module (squeeze–excitation module) to enhance the representational capabilities of convolutional neural networks. We conducted experiments on two publicly available benchmark datasets, UCI-HAR and WISDM, to evaluate our model. The results were satisfactory, with overall recognition accuracies of 96.72% and 97.73% on their respective datasets. The experimental results demonstrate the effectiveness of the network structure for the HAR, which ensures a higher level of accuracy.

Dynamic graph topology generating mechanism: Framework for feature-level multimodal information fusion applied to lower-limb activity recognition

Lower-limb activity recognition (LLAR), which is driven by wearable sensors, has recently become a popular research topic in human assistive devices. Although multimodal information fusion is crucial for improving the accuracy of LLAR, traditional feature-level fusion strategies cannot directly model the relationship between multimodal features according to the lower-limb activity patterns and they realize fusion based on the relationship. To address these limitations, a dynamic graph topology generation (DGTG) mechanism that uses a graph topology to model the relationship between multimodal features mined by a convolutional neural network (CNN) is proposed in this study. A weighted graph isomorphism network (WGIN) that can realize multimodal feature fusion based on the relationship (i.e., graph topology) is also proposed, resulting in a novel feature-level multimodal information fusion framework called CNN-DGTG-WGIN. This proposed method achieved the best average accuracies of 96.3 and 99.7% for two publicly available datasets. This method integrates the determination of the graph topology into an end-to-end learning process without requiring any prior knowledge or original graph structures. Further, CNN-DGTG-WGIN effectively incorporates both linear and nonlinear correlations between features during the DGTG process. Analysis results revealed that the graph topology (multimodal feature relationship) learned by CNN-DGTG-WGIN was closely related to lower-limb activity patterns, encoding relevant information about these patterns, which is the essence of its superior performance.

Device-free Vehicle Type Identification for Next Generations Intelligent Transportation System

With the substantial progress in the automotive industry, intelligent vehicle identification systems have become imperative tools for next generation transportation system planning. The advent of internet of vehicles and intelligent cellular vehicular networks has entered the transportation system into a new era of technology. In smart transportation system of beyond 5G/6G vehicular networks, vehicle type identification is of vital importance for ensuring safe operability of autonomous driving environ ments, electronic toll, and traffic management. It can be used in high efficiency roadway services, vehicle theft prevention, road condition monitoring, automatic driving, access control security, etc. During the recent years, channel state information (CS I) of WiFi signal has gained escalating traction for activity or gesture recognition, and person identification with improved robustness and classification accuracy. Therefore, in this research work, a WiFi based vehicle type identification system is proposed for beyond 5G/6G vehicular networks. In this novel approach, we start from vehicle detection using the variation in CSI of WiFisignals caused by the moving vehicle. For vehicle detection we simply apply a threshold based algorithm and then extract meaningful features. Afterwards, the vehicle type is identified using machine learning algorithms. This is a wireless model that enables commercial WiFi devices to identify different types of vehicles using CSI measurements. The presented framework can identify six different types of vehicles with an accuracy of 91.8%

IoT-FAR: A multi-sensor fusion approach for IoT-based firefighting activity recognition

Inadequate training poses a significant risk of injury among young firefighters. Although Human Activity Recognition (HAR) algorithms have shown potential in monitoring and evaluating performance, most existing studies focus on daily activities and have difficulty distinguishing complex firefighting tasks. This study introduces the Internet of things (IoT)-based wearable firefighting activity recognition (IoT-FAR) system which employs a multi-modal sensor fusion approach to achieve comprehensive activity recognition during firefighting training. The IoT-FAR comprises five wearable body sensor nodes and a coordinator node. This study explores the significance of features extracted from the surface electromyography, heart rate, and inertial measurement units in firefighting training activity recognition. A hybrid machine learning (HML)-based network is proposed, which integrates three models: one trained with all features (MA), another with upper body features (MU), and a third with lower body features (ML). The proposed HML-SVM-RBF1-RF2 network achieves superior performance, with a mean recall of 93.94%, mean precision of 90.94%, and a mean accuracy rate of 98.29%. Additionally, the study introduces the specialized firefighting training associated activities (SFTAA) dataset, which includes endurance training activities involving self-contained breathing apparatus (SCBA) conducted by eighteen firefighters. This dataset represents preliminary work towards building a comprehensive dataset covering various events and scenarios for tracking firefighter activities. The IoT-FAR system also demonstrates the potential use of misclassified activities as evaluation metrics for assessing firefighter training performance.

Automatic recognition of active landslides by surface deformation and deep learning

Catastrophic landslides are generally evolved from potential active landslides, and early identification of active landslides over an extensive region is vital to effective prevention and control of disastrous landslides in urban areas. Interferometric Synthetic Aperture Radar (InSAR) has immense potential in mapping active landslides. However, artificial interpretation of InSAR measurements and manual recognition of active landslides are very laborious and time-consuming, with a relatively high missing and false alarms. That hinders the application of InSAR technique and the identification of active landslides in wide areas. Automatic recognition of active landslides has always been a great challenge and has been relatively rarely investigated by previous studies. This work establishes comprehensive identification indices of geoenvironmental, disaster-triggering, and surface deformation features. Moreover, it suggests a novel deep learning algorithm of SDeepFM to conduct automatic identification of active landslides across a vast and landslide-serious area of Hualong County. Some new viewpoints are suggested as follows. (1) The identification indices consist of disaster-controlling, disaster-inducing, and active deformation characteristics and are constructed in terms of the cause characteristics of active landslides. Thus, it can effectively decrease the false alarms of active landslide identification. (2) The proposed SDeepFM algorithm features a spatial-perception ability and can adequately extract and fuse the low-level and high-level semantic features. It outperforms the classification and regression tree (CART), multi-layer perceptron (MLP), convolutional neural network (CNN), and deep neural network (DNN) algorithms. The test accuracy attains 0.91, 99.73%, 90.21%, 0.92, 0.96, and 0.91 in F1-score, Accuracy, Precision, Recall, AUC, and Kappa, respectively. (3) A total of 164 active landslides are exactly recognized, and 39 active landslides are newly identified in this work. (4) In Hualong County, the characteristics of slope deformation, spatial context, lithology, tectonic movement, human activity, and topography play important roles in active landslide identification. River distribution and rainfall also contribute to active landslide recognition.

AS-APF: Encoding time series as images for human activity recognition with SK-based convolutional networks

AS-APF: Encoding time series as images for human activity recognition with SK-based convolutional networks

RetinaViT: Efficient Visual Backbone for Online Video Streams.

In online video understanding, which has a wide range of real-world applications, inference speed is crucial. Many approaches involve frame-level visual feature extraction, which often represents the biggest bottleneck. We propose RetinaViT, an efficient method for extracting frame-level visual features in an online video stream, aiming to fundamentally enhance the efficiency of online video understanding tasks. RetinaViT is composed of efficiently approximated Transformer blocks that only take changed tokens (event tokens) as queries and reuse the already processed tokens from the previous timestep for the others. Furthermore, we restrict keys and values to the spatial neighborhoods of event tokens to further improve efficiency. RetinaViT involves tuning multiple parameters, which we determine through a multi-step process. During model training, we randomly vary these parameters and then perform black-box optimization to maximize accuracy and efficiency on the pre-trained model. We conducted extensive experiments on various online video recognition tasks, including action recognition, pose estimation, and object segmentation, validating the effectiveness of each component in RetinaViT and demonstrating improvements in the speed/accuracy trade-off compared to baselines. In particular, for action recognition, RetinaViT built on ViT-B16 reduces inference time by approximately 61.9% on the CPU and 50.8% on the GPU, while achieving slight accuracy improvements rather than degradation.

Characterization of biotic and abiotic signatures of modern lake sediments of western India, and its palaeo-environmental implications

A multi-proxy study of biotic and abiotic components was conducted on surface sediment samples from six lakes/wetlands located along the western transitional boundary of the contemporary Indian Summer Monsoon (ISM) in the Indian Subcontinent. The primary goal is to assess the suitability of various proxies as representatives of modern vegetation, environmental and climatic conditions. The collected data indicate significant variations in the composition and density of pollen in response to climate-induced and anthropogenic ecological changes throughout the northwest India transect. The palynological studies from eastern Rajasthan shows high forest elements in comparison to western Haryana and western Uttar Pradesh. In addition, the palynological data was juxtaposed with other biotic proxies such as diatom and isotopic studies, along with geochemical proxies and paleomagnetic data of the surface lake sediments. The presence of marker pollen taxa including Cerealia type Poaceae, Chenopodiaceae, and Brassicaceae, allows for distinct recognition of anthropogenic activities throughout the whole transect. The diversity and distribution of diatoms also support the palynological data in response to climate-induced and anthropogenic ecological changes. Furthermore, grain size, geochemistry (TOC/TN ratio with stable carbon isotope), and magnetic susceptibility data offer crucial insights about the sediment's depositional settings and general mineralogical composition. Stable carbon isotope data shows C3 dominance in relatively humid areas and C4 dominance in semi-arid areas, suggesting climate-driven control over sediment organic matter composition. Canonical correspondence analysis (CCA) indicates that biotic variables (pollen, diatom, stable isotopic composition) are significantly controlled by modern precipitation and temperature. Redundancy analysis reveals a significant influence of current average temperature and precipitation on major element oxide variations in surface lake sediments. Therefore, we propose using palynological, stable carbon isotope, diatom data, along with grain size, environmental magnetism, and geochemistry, to establish a multiproxy modern analogue for quantitative palaeoclimatic reconstructions. As a result, this study provides the first modern analogues from a climate-sensitive region that separates the area under ISM influence from an area with meager precipitation in western India.

Data augmentation aided excavator activity recognition using deep convolutional conditional generative adversarial networks

Excavator activity recognition is a crucial task with significant practical applications such as assessing production efficiency, injury prevention, and enabling intelligent control. However, implementing deep learning-based activity recognition methods presents considerable challenges due to the requirement of large training datasets. And sensor-based activity recognition demands more convenient and reliable sensors. This paper proposes a generalized data augmentation framework for excavator activity recognition. Main pump pressure and flow signals are generated using Deep Convolutional Conditional Generative Adversarial Networks (DC-CGAN). The proposed framework integrates Long Short-Term Memory (LSTM) and Convolutional Neural Network (CNN) architectures for activity recognition. Results indicate that the framework outperforms traditional data augmentation methods regarding the quality of generated data and shallow and single deep networks regarding model accuracy and generalization. Additionally, the framework was implemented on three sizes of excavators and a wheel loader, demonstrating excellent versatility.

CARD8 inflammasome activation during HIV-1 cell-to-cell transmission.

Our previous work demonstrated that CARD8 detects HIV-1 infection by sensing the enzymatic activity of the HIV protease, resulting in CARD8-dependent inflammasome activation (Kulsuptrakul et al., 2023). CARD8 recognition of HIV-1 protease activity is conferred by a HIV protease substrate mimic within the CARD8 N-terminus, which when cleaved by HIV protease triggers CARD8 inflammasome activation. Here, we sought to understand CARD8 responses to HIV-1 when the virus is transmitted through cell-to-cell infection from infected cells to target cells via a viral synapse. We observed that cell-to-cell transmission of HIV-1 induces CARD8 inflammasome activation in immortalized cells and primary human monocyte-derived macrophages in a manner that is dependent on viral protease activity and largely independent of the NLRP3 inflammasome. Additionally, to further evaluate the viral determinants of CARD8 sensing, we tested a panel of HIV protease inhibitor resistant clones to establish how variation in HIV protease affects CARD8 activation. We identified mutant HIV-1 proteases that differentially cleave and activate CARD8 compared to wildtype HIV-1, thus indicating that natural variation in HIV protease affects not only the cleavage of the viral Gag-Pol polyprotein but also likely impacts innate sensing and inflammation.

Distilling interaction knowledge for semi-supervised egocentric action recognition

Egocentric action recognition, the identification of actions within video content obtained from a first-person perspective, is receiving increasing attention due to the widespread adoption of wearable camera technology. Nonetheless, the task of annotating actions within a video characterized by a cluttered background and the presence of various objects is labor-intensive. In this paper, we consider learning for egocentric action recognition in a semi-supervised manner. Inspired by the fact that videos captured from first-person viewpoint usually contain rich contents about how human hands interact with objects, we thus propose to employ a popular teacher–student framework and distill the interaction knowledge between hand and objects for semi-supervised egocentric action recognition. We refer to the proposed method as Interaction Knowledge Distillation or IKD. Specifically, the teacher network takes hands and action-related objects in the labeled videos as input, and uses graph neural networks to capture their spatial–temporal relations as graph edge features. The student network then takes the detected hands/objects from both labeled and unlabeled videos as input and mimics the teacher network to learn from the interactions to improve model performance. Experiments are performed on two popular egocentric action recognition datasets, Something-Something-V2 and EPIC-KITCHENS-100, which show that our proposed approach consistently outperforms recent state-of-the-art methods in typical semi-supervised settings.

Collecting Self-reported Physical Activity and Posture Data Using Audio-based Ecological Momentary Assessment

μEMA is a data collection method that prompts research participants with quick, answer-at-a-glance, single-multiple-choice self-report behavioral questions, thus enabling high-temporal-density self-report of up to four times per hour when implemented on a smartwatch. However, due to the small watch screen, μEMA is better used to select among 2 to 5 multiple-choice answers versus allowing the collection of open-ended responses. We introduce an alternative and novel form of micro-interaction self-report using speech input - audio-μEMA- where a short beep or vibration cues participants to verbally report their behavioral states, allowing for open-ended, temporally dense self-reports. We conducted a one-hour usability study followed by a within-subject, 6-day to 21-day free-living feasibility study in which participants self-reported their physical activities and postures once every 2 to 5 minutes. We qualitatively explored the usability of the system and identified factors impacting the response rates of this data collection method. Despite being interrupted 12 to 20 times per hour, participants in the free-living study were highly engaged with the system, with an average response rate of 67.7% for audio-μEMA for up to 14 days. We discuss the factors that impacted feasibility; some implementation, methodological, and participant challenges we observed; and important considerations relevant to deploying audio-μEMA in real-time activity recognition systems.

Using Human Skeleton Point Repair Methods to Improve the Accuracy of Multi-person Interactive Activity Recognition

Using Human Skeleton Point Repair Methods to Improve the Accuracy of Multi-person Interactive Activity Recognition

WISNet: A deep neural network based human activity recognition system

Nowadays, Human Activity Recognition (HAR) is a key research area with many ubiquitous innovative solutions, where both accelerometer and gyroscope data provide information about an observed person’s physical activity. HAR offers a diverse variety of important applications, including healthcare, burglary detection, workplace monitoring, and emergency identification. Traditional recognition approaches rely on extracting handmade features from the obtained data to identify the typeof human action. Additionally, the efficacy of these works is dependent upon the specific customized features that are chosen. One potential approach to tackle this issue is to utilize Convolutional Neural Networks (CNN) to automatically learn the relevant features. In this paper, we propose a deep learning model, WISNet, a custom 1D-CNN approach to recognize six complex human activities: Jogging, Walking Downstairs, Sitting, Standing, Walking and Climbing Upstairs. The model includes a Convolved Normalized Pooled (CNPM) Block to generate significant features from the initial layers. An Identity and Basic (IDBN) Block is incorporated to extract residual progressive features for capturing complex sequential data dependencies. Channel and Spatial attention (CASb) Block is integrated with the network to prioritize or minimize essential features based on relative weights. The proposed WISNet model achieved an enhanced accuracy and F1-score of 96.41 % and 0.95 for the HAR dataset by surpassing the existing transfer learning architectures such as Gated Recurrent Units (GRU), Long Short-Term Memory (LSTM), and Recurrent Neural Network (SimpleRNN). By strategically integrating the CNPM, IDBN, and CASb blocks, this study aims to tackle distinct challenges encountered in the classification process by enhancing the discernment of features essential for the precise identification of multi-class human activity recognition. The seamless integration of these blocks within the model plays a pivotal role in elevating the overall performance of the WISNet architecture. The work also validates WISNet with similar open-source datasets (UCI-HAR and KU-HAR) and dissimilar open-source datasets (Sleep state detection, Fall detection, and ECG Heartbeat).

IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition

One of the primary challenges in the field of human activity recognition (HAR) is the lack of large labeled datasets. This hinders the development of robust and generalizable models. Recently, cross modality transfer approaches have been explored that can alleviate the problem of data scarcity. These approaches convert existing datasets from a source modality, such as video, to a target modality, such as inertial measurement units (IMUs). With the emergence of generative AI models such as large language models (LLMs) and text-driven motion synthesis models, language has become a promising source data modality as well - as shown in proof of concepts such as IMUGPT. In this work, we conduct a large-scale evaluation of language-based cross modality transfer to determine their effectiveness for HAR. Based on this study, we introduce two new extensions for IMUGPT that enhance its use for practical HAR application scenarios: a motion filter capable of filtering out irrelevant motion sequences to ensure the relevance of the generated virtual IMU data, and a set of metrics that measure the diversity of the generated data facilitating the determination of when to stop generating virtual IMU data for both effective and efficient processing. We demonstrate that our diversity metrics can reduce the effort needed for the generation of virtual IMU data by at least 50%, which opens up IMUGPT for practical use cases beyond a mere proof of concept.

Achieving More with Less: A Lightweight Deep Learning Solution for Advanced Human Activity Recognition (HAR).

Human activity recognition (HAR) is a crucial task in various applications, including healthcare, fitness, and the military. Deep learning models have revolutionized HAR, however, their computational complexity, particularly those involving BiLSTMs, poses significant challenges for deployment on resource-constrained devices like smartphones. While BiLSTMs effectively capture long-term dependencies by processing inputs bidirectionally, their high parameter count and computational demands hinder practical applications in real-time HAR. This study investigates the approximation of the computationally intensive BiLSTM component in a HAR model by using a combination of alternative model components and data flipping augmentation. The proposed modifications to an existing hybrid model architecture replace the BiLSTM with standard and residual LSTM, along with convolutional networks, supplemented by data flipping augmentation to replicate the context awareness typically provided by BiLSTM networks. The results demonstrate that the residual LSTM (ResLSTM) model achieves superior performance while maintaining a lower computational complexity compared to the traditional BiLSTM model. Specifically, on the UCI-HAR dataset, the ResLSTM model attains an accuracy of 96.34% with 576,702 parameters, outperforming the BiLSTM model's accuracy of 95.22% with 849,534 parameters. On the WISDM dataset, the ResLSTM achieves an accuracy of 97.20% with 192,238 parameters, compared to the BiLSTM's 97.23% accuracy with 283,182 parameters, demonstrating a more efficient architecture with minimal performance trade-off. For the KU-HAR dataset, the ResLSTM model achieves an accuracy of 97.05% with 386,038 parameters, showing comparable performance to the BiLSTM model's 98.63% accuracy with 569,462 parameters, but with significantly fewer parameters.