Action Recognition Framework Research Articles

Framework for Simultaneous Indoor Localization, Mapping, and Human Activity Recognition in Ambient Assisted Living Scenarios.

Indoor localization and human activity recognition are two important sources of information to provide context-based assistance. This information is relevant in ambient assisted living (AAL) scenarios, where older adults usually need supervision and assistance in their daily activities. However, indoor localization and human activity recognition have been mostly considered isolated problems. This work presents and evaluates a framework that takes advantage of the relationship between location and activity to simultaneously perform indoor localization, mapping, and human activity recognition. The proposed framework provides a non-intrusive configuration, which fuses data from an inertial measurement unit (IMU) placed in the person’s shoe, with proximity and human activity-related data from Bluetooth low energy beacons (BLE) deployed in the indoor environment. A variant of the simultaneous location and mapping (SLAM) framework was used to fuse the location and human activity recognition (HAR) data. HAR was performed using data streaming algorithms. The framework was evaluated in a pilot study, using data from 22 people, 11 young people, and 11 older adults (people aged 65 years or older). As a result, seven activities of daily living were recognized with an F1 score of 88%, and the in-door location error was 0.98 ± 0.36 m for the young and 1.02 ± 0.24 m for the older adults. Furthermore, there were no significant differences between the groups, indicating that our proposed method works adequately in broad age ranges.

Wearable multi-sensor data fusion approach for human activity recognition using machine learning algorithms

Wearable sensor based human activity recognition (HAR) has a broad range of applications in healthcare, fitness, smart home, and surveillance. In spite of the substantial amount of computational research on HAR, the open challenges in multi-sensor based activity recognition including complex temporal data, identifying discriminative feature vectors from multi-modal data and dimensionality reduction require considerable research attention. Hence, in this paper, using multi-resolution time-frequency analysis of received signal strength (RSS) between the wearable sensors, we present a supervised machine learning (ML) based activity recognition framework. The multi-sensor activity data acquired using the wireless sensor network (WSN) nodes and inertial-sensor embedded in a smartphone are decomposed using the discrete wavelet transform (DWT) and the empirical mode decomposition (EMD) techniques for extracting the prominent feature vector. Using the discriminative statistical features from DWT along with the entropy features from EMD, the four ML classifier models such as support vector machine (SVM), K-nearest neighbour (KNN), ensemble classifier (EC), and decision tree (DT) are trained to classify various human activities. The efficacy of the proposed design is assessed on a publicly available dataset from UCI. The experimental results assessed using the confusion matrix and parallel coordinate plot (PCP) substantiate that the proposed ML based HAR framework can achieve a maximum classification accuracy of 99.63% and is superior to several state-of-the-art ML techniques.

IHAR—A fog‐driven interpretable human activity recognition system

AbstractHuman activity recognition (HAR) is an active research area that is currently being applied to various healthcare applications such as fall detection, assisted living, etc. These applications make use of the Internet of Things which is widespread across today's world. One of the major challenges in these applications is the need for quick intelligent decisions. The deployment of hierarchical edge‐fog‐cloud computing architecture is the potential solution to address the latency issue. In this article, we proposed an interpretable human activity recognition (IHAR) framework that supports HAR, leveraging the advantages of the fog layer. The proposed framework used a deep learning (DL) model at the cloud infrastructure to classify the activities of humans. The trained DL model is made to run on local fog nodes. However, the DL model being black‐box does not provide any explanations for the output to make them acceptable by the users and physicians. Hence, this article also incorporated an explainable artificial intelligence model at the fog layer, to gain insights into the classified output of the DL model. The effective model‐level insights emphasized the need for explainable HAR. The article also included an analysis of the results of the existing explainable AI models such as LIME and SHAP to understand which model leads to better performance in the domain of HAR. Results show that the SHAP model has a 33% higher success rate in generating explanations when compared to the LIME model.

Human Action Recognition and Prediction: A Survey

Derived from rapid advances in computer vision and machine learning, video analysis tasks have been moving from inferring the present state to predicting the future state. Vision-based action recognition and prediction from videos are such tasks, where action recognition is to infer human actions (present state) based upon complete action executions, and action prediction to predict human actions (future state) based upon incomplete action executions. These two tasks have become particularly prevalent topics recently because of their explosively emerging real-world applications, such as visual surveillance, autonomous driving vehicle, entertainment, and video retrieval, etc. Many attempts have been devoted in the last a few decades in order to build a robust and effective framework for action recognition and prediction. In this paper, we survey the complete state-of-the-art techniques in action recognition and prediction. Existing models, popular algorithms, technical difficulties, popular action databases, evaluation protocols, and promising future directions are also provided with systematic discussions.

A human activity recognition framework for grossly labeled smartphone sensing data through combining genetic algorithm with multiple instance multiple label learning

A human activity recognition framework for grossly labeled smartphone sensing data through combining genetic algorithm with multiple instance multiple label learning

Privacy preserving human activity recognition framework using an optimized prediction algorithm

Human activity recognition, in computer vision research, is the area of growing interest as it has plethora of real-world applications. Inferring actions from one or more persons captured through a live video has its immense utility in the contemporary era. Same time, protecting privacy of humans is to be given paramount importance. Many researchers contributed towards this end leading to privacy preserving action recognition systems. However, having an optimized model that can withstand any adversary models that strives to disclose privacy information. To address this problem, we proposed an algorithm known optimized prediction algorithm for privacy preserving activity recognition (OPA-PPAR) based on deep neural networks. It anonymizes video content to have adaptive privacy model that defeats attacks from adversaries. The privacy model enhances the privacy of humans while permitting highly accurate approach towards action recognition. The algorithm is implemented to realize privacy preserving human activity recognition framework (PPHARF). The visual recognition of human actions is made using an underlying adversarial learning process where the anonymization is optimized to have an adaptive privacy model. A dataset named human metabolome database (HMDB51) is used for empirical study. Our experiments with using Python data science platform reveal that the OPA-PPAR outperforms existing methods.

Hybrid handcrafted and learned feature framework for human action recognition

Recognising human actions in video is a challenging task in real-world. Dense trajectory (DT) offers accurate recording of motions over time that is rich in dynamic information. However, DT models lack the mechanism to distinguish dominant motions from secondary ones over separable frequency bands and directions. By contrast, deep learning-based methods are promising over the challenge though still suffering from limited capacity in handling complex temporal information, not mentioning huge datasets needed to guide the training. To take the advantage of semantical meaningful and “handcrafted” video features through feature engineering, this study integrates the discrete wavelet transform (DWT) technique into the DT model for gaining more descriptive human action features. Through exploring the pre-trained dual-stream CNN-RNN models, learned features can be integrated with the handcrafted ones to satisfy stringent analytical requirements within the spatial-temporal domain. This hybrid feature framework generates efficient Fisher Vectors through a novel Bag of Temporal Features scheme and is capable of encoding video events whilst speeding up action recognition for real-world applications. Evaluation of the design has shown superior recognition performance over existing benchmark systems. It has also demonstrated promising applicability and extensibility for solving challenging real-world human action recognition problems.

3d Dual-Stream Convolutional Neural Networks with Simple Recurrent Unit Network: A New Framework for Action Recognition

3d Dual-Stream Convolutional Neural Networks with Simple Recurrent Unit Network: A New Framework for Action Recognition

Pose-Guided Graph Convolutional Networks for Skeleton-Based Action Recognition

Graph convolutional networks (GCN), which can model the human body skeletons as spatial and temporal graphs, have shown remarkable potential in skeleton-based action recognition. However, in the existing GCN-based methods, graph-structured representation of the human skeleton makes it difficult to be fused with other modalities, especially in the early stages. This may limit their scalability and performance in action recognition tasks. In addition, the pose information, which naturally contains informative and discriminative clues for action recognition, is rarely explored together with skeleton data in existing methods. In this work, we proposed pose-guided GCN (PG-GCN), a multi-modal framework for high-performance human action recognition. In particular, a multi-stream network is constructed to simultaneously explore the robust features from both the pose and skeleton data, while a dynamic attention module is designed for early-stage feature fusion. The core idea of this module is to utilize a trainable graph to aggregate features from the skeleton stream with that of the pose stream, which leads to a network with more robust feature representation ability. Extensive experiments show that the proposed PG-GCN can achieve state-of-the-art performance on the NTU RGB+D 60 and NTU RGB+D 120 datasets.

A Hybrid Duo-Deep Learning and Best Features Based Framework for燗ction燫ecognition

Human Action Recognition (HAR) is a current research topic in the field of computer vision that is based on an important application known as video surveillance. Researchers in computer vision have introduced various intelligent methods based on deep learning and machine learning, but they still face many challenges such as similarity in various actions and redundant features. We proposed a framework for accurate human action recognition (HAR) based on deep learning and an improved features optimization algorithm in this paper. From deep learning feature extraction to feature classification, the proposed framework includes several critical steps. Before training fine-tuned deep learning models – MobileNet-V2 and Darknet53 – the original video frames are normalized. For feature extraction, pre-trained deep models are used, which are fused using the canonical correlation approach. Following that, an improved particle swarm optimization (IPSO)-based algorithm is used to select the best features. Following that, the selected features were used to classify actions using various classifiers. The experimental process was performed on six publicly available datasets such as KTH, UT-Interaction, UCF Sports, Hollywood, IXMAS, and UCF YouTube, which attained an accuracy of 98.3%, 98.9%, 99.8%, 99.6%, 98.6%, and 100%, respectively. In comparison with existing techniques, it is observed that the proposed framework achieved improved accuracy.

Hybrid deep learning framework for human activity recognition

The aim of the recognition in the human activity is to recognize the actions of the individuals using a set of observations and their environmental conditions. Since last two decades, the research on this Human Activity Recognition (HAR) has captured the attention of several computer science communities because of the strength to provide support to different applications and the connection to different fields of study such as, human-computer interaction, healthcare, monitoring, entertainment and education. There are many existing methods like deep learning which have been used to develop to recognize the different activities of the human, but couldn’t identify the sudden change of the activities in the human. This paper presents a method using the deep learning methods which can recognize the specific identities and identify a change from one activity to another for the applications of the healthcare. In this method, a deep convolutional neural network is built using which the features are extracted for the collection of the data from the sensors. After which the Gated Recurrent Unit (GRU) captures the long-tern dependency between the different actions which helps to improve the identification rate of the HAR. From the CNN and GRU, a model of wearable sensor can be proposed which can identify the changes of the activities and can accurately recognize these activities. Experiment have been conducted using open-source University of California (UCI) HAR dataset which composed of six different activity such as lying, standing, sitting, walking downstairs, walking upstairs and walking. The CNN-based model achieves a detection accuracy of 95.99% whereas the CNN-GRU model achieves a detection accuracy of 96.79% which is better than most existing HAR methods.

Learning SpatioTemporal and Motion Features in a Unified 2D Network for Action Recognition.

Recent methods for action recognition always apply 3D Convolutional Neural Networks (CNNs) to extract spatiotemporal features and introduce optical flows to present motion features. Although achieving state-of-the-art performance, they are expensive in both time and space. In this paper, we propose to represent both two kinds of features in a unified 2D CNN without any 3D convolution or optical flows calculation. In particular, we first design a channel-wise spatiotemporal module to present the spatiotemporal features and a channel-wise motion module to encode feature-level motion features efficiently. Besides, we provide a distinctive illustration of the two modules from the frequency domain by interpreting them as advanced and learnable versions of frequency components. Second, we combine these two modules and an identity mapping path into one united block that can easily replace the original residual block in the ResNet architecture, forming a simple yet effective network dubbed STM network by introducing very limited extra computation cost and parameters. Third, we propose a novel Twins Training framework for action recognition by incorporating a correlation loss to optimize the inter-class and intra-class correlation and a siamese structure to fully stretch the training data. We extensively validate the proposed STM on both temporal-related datasets (i.e., Something-Something v1 & v2) and scene-related datasets (i.e., Kinetics-400, UCF-101, and HMDB-51). It achieves favorable results against state-of-the-art methods in all the datasets.

A knowledge-driven activity recognition framework for learning unknown activities

Human activity recognition has increasingly received attention in recent years to track regular activities of people. Existing activity recognition approaches considerably contributed to the analysis of human behavior. However, they still confront numerous issues related to the variability of activities performed by people within dynamic environments. Generally, this variability renders the used training or ontology models with predefined activities unsuitable. Therefore, creating an activity recognition approach that is able to leverage dynamically new and unknown activities at runtime becomes important. In this paper, we propose a novel knowledge-driven activity recognition framework using smartphone. This framework envisions taking a knowledge-driven approach to reinforce the recognition accuracy and people's quality of life in the context of dynamic environments at runtime. More specifically, we propose an ontology-based context evolution along with a dynamic decision-making, so that new and unknown performed activities can be accurately recognized. Furthermore, we use a public activity recognition dataset to demonstrate the effectiveness of the proposed framework and show its advantage over a data-driven baselines in terms of accuracy. Experimental results reveal that our framework not only reinforces the accuracy, but also enables an effective activity learning when facing unknown activities at runtime.

Real-time monitoring of occupancy activities and window opening within buildings using an integrated deep learning-based approach for reducing energy demand

Occupancy behaviour in buildings can impact the energy performance and operation of heating, ventilation and air-conditioning (HVAC) systems. HVAC, which uses conventional control strategies or “fixed” setpoint schedules, could not adjust to the conditioned spaces' actual requirements, resulting in building spaces being over or under-conditioned. While the unintended opening of windows can lead to substantial heat loss and consequently raises energy consumption. To optimise building operations, it is necessary to employ solutions such as demand-driven controls, which can monitor the utilisation of indoor spaces and provide the actual thermal comfort requirements of occupants. This study presents a novel vision-based deep learning framework for occupancy activity detection and recognition including the manual window operations in buildings. A region-based Convolutional Neural Network (R-CNN) model was trained and deployed to a camera for real-time detection and recognition. Based on the field experiments conducted within a case study University building, overall accuracy of 85.63% was achieved for occupancy activity detection and 92.20% for window operation detection. Building energy simulation and various scenario-based cases were used to assess the impact of such an approach on the building energy demand and provide insights into how the proposed detection method can enable HVAC systems to respond to dynamic changes within indoor spaces. Results showed that the proposed approach could reduce the over-or under-estimation of occupancy heat gains compared with the use of “fixed” or static profiles. In addition, the approach can help alert building users or managers about windows left open unintentionally, which can reduce unnecessary ventilation heat losses. Furthermore, the approach can also predict the room CO2 concentration and advise occupants about a suitable natural ventilation strategy. The study highlighted the potential of the multi-purpose detection approach, but further development is necessary, including optimisation of the deep learning model, full integration with HVAC controls and further model training and field testing.

An Efficient Human Instance-Guided Framework for Video Action Recognition.

In recent years, human action recognition has been studied by many computer vision researchers. Recent studies have attempted to use two-stream networks using appearance and motion features, but most of these approaches focused on clip-level video action recognition. In contrast to traditional methods which generally used entire images, we propose a new human instance-level video action recognition framework. In this framework, we represent the instance-level features using human boxes and keypoints, and our action region features are used as the inputs of the temporal action head network, which makes our framework more discriminative. We also propose novel temporal action head networks consisting of various modules, which reflect various temporal dynamics well. In the experiment, the proposed models achieve comparable performance with the state-of-the-art approaches on two challenging datasets. Furthermore, we evaluate the proposed features and networks to verify the effectiveness of them. Finally, we analyze the confusion matrix and visualize the recognized actions at human instance level when there are several people.

A Novel Hybrid Deep Learning Model for Human Activity Recognition Based on Transitional Activities.

In recent years, a plethora of algorithms have been devised for efficient human activity recognition. Most of these algorithms consider basic human activities and neglect postural transitions because of their subsidiary occurrence and short duration. However, postural transitions assume a significant part in the enforcement of an activity recognition framework and cannot be neglected. This work proposes a hybrid multi-model activity recognition approach that employs basic and transition activities by utilizing multiple deep learning models simultaneously. For final classification, a dynamic decision fusion module is introduced. The experiments are performed on the publicly available datasets. The proposed approach achieved a classification accuracy of 96.11% and 98.38% for the transition and basic activities, respectively. The outcomes show that the proposed method is superior to the state-of-the-art methods in terms of accuracy and precision.

GeometryMotion-Net: A Strong Two-Stream Baseline for 3D Action Recognition

In this work, we propose a strong two-stream baseline method referred to as GeometryMotion-Net for 3D action recognition. For efficient 3D action recognition, we first represent each point cloud sequence as a limited number of randomly sampled frames with each frame consisting of a sparse set of points. After that, we propose a new two-stream framework for effective 3D action recognition. For the geometry stream, we propose a new module to produce a virtual overall geometry point cloud by first merging all 3D points from these selected frames, and then we exploit local neighborhood information of each point in the feature space. In the motion stream, for any two neighboring point cloud frames, we also propose a new module to generate one virtual forward motion point cloud and one virtual backward motion point cloud. Specifically, for each point in the current frame, we first produce a set of 3D offset features relative to the neighboring points in the reference frame ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , the previous/subsequent frame) and then exploit local neighborhood information of this point in the offset feature space. Based on the newly generated virtual overall geometry point cloud and multiple virtual forward/backward motion point clouds, any existing point cloud analysis methods ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> , PointNet) can be readily adopted for extracting discriminant geometry and bidirectional motion features in the geometry and motion streams, respectively, which are further aggregated to make our two-stream network trainable in an end-to-end fashion. Comprehensive experiments on both large-scale datasets ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> NTU RGB+D 60 and NTU RGB+D 120) and small-scale datasets ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , N-UCLA and UWA3DII) demonstrate the effectiveness and efficiency of our two-stream network for 3D action recognition.

Feasibility Study on Using AI and VR for Decision-Making Training of Basketball Players

Decision-making plays an essential role in basketball offense. Offensive players must make effective decisions to score a basket in various defensive situations. Virtual reality (VR) has been widely used in the training of athletes to strengthen their ability to make optimal decisions by creating controllable repeatable training scenarios. In this article, an action-aware offensive decision-making training system for basketball using VR and artificial intelligence is proposed. The proposed system is composed of different virtual defensive scenarios and an offensive action recognition framework. Trainees wearing head-mounted display and a motion capture suit are trained by intuitively interacting with the VR system and receive decision suggestions when a bad one is made. This study changes the training media and methods to create an immersive training environment during the training phase and evaluates the training effectiveness. These training scenarios are a conventional tactics board, the proposed VR system with a prerecorded <inline-formula><tex-math notation="LaTeX">$360^\circ$</tex-math></inline-formula> panorama video, and the VR system with computer-simulated virtual content. Results indicate that the training scenario affects the training in terms of decision time.

A multi-task learning based hybrid prediction algorithm for privacy preserving human activity recognition framework

There is ever increasing need to use computer vision devices to capture videos as part of many real-world applications. However, invading privacy of people is the cause of concern. There is need for protecting privacy of people while videos are used purposefully based on objective functions. One such use case is human activity recognition without disclosing human identity. In this paper, we proposed a multi-task learning based hybrid prediction algorithm (MTL-HPA) towards realising privacy preserving human activity recognition framework (PPHARF). It serves the purpose by recognizing human activities from videos while preserving identity of humans present in the multimedia object. Face of any person in the video is anonymized to preserve privacy while the actions of the person are exposed to get them extracted. Without losing utility of human activity recognition, anonymization is achieved. Humans and face detection methods file to reveal identity of the persons in video. We experimentally confirm with joint-annotated human motion data base (JHMDB) and daily action localization in YouTube (DALY) datasets that the framework recognises human activities and ensures non-disclosure of privacy information. Our approach is better than many traditional anonymization techniques such as noise adding, blurring, and masking.

DCNN based human activity recognition framework with depth vision guiding

The smartphone-based human activity recognition (HAR) provides abundant healthcare guidance for telemedicine and clinic treatment. The continually increasing daily activities cause many difficulties for recognition and labeling. Although multimodal data fusion and artificial intelligence (AI) techniques can solve these problems, big data collection and labeling are still heavy. This paper proposes a remarkable depth data-guided framework based on smartphones for complex HAR and automatic labeling. The hardware platform is utilized to collect information of depth vision from the Microsoft Kinect camera and Inertial Measurement Unit (IMU) signals from the smartphone simultaneously. This framework consists of five clustering layers and deep learning (DL) based classification model to identify 12 complex daily activities. The results show that the hierarchical k-medoids (Hk-medoids) algorithm obtains the labels with high accuracy (93.89%). Furthermore, the performance evaluation of the DCNN model is evaluated better by comparing it with other machine learning (ML) and DL methods.