Activity Recognition Research Research Articles

Complex Human Action Recognition Using a Hierarchical Feature Reduction and Deep Learning-Based Method

Automated human action recognition is one of the most attractive and practical research fields in computer vision. In such systems, the human action labelling is based on the appearance and patterns of the motions in the video sequences; however, majority of the existing research and most of the conventional methodologies and classic neural networks either neglect or are not able to use temporal information for action recognition prediction in a video sequence. On the other hand, the computational cost of a proper and accurate human action recognition is high. In this paper, we address the challenges of the preprocessing phase, by an automated selection of representative frames from the input sequences. We extract the key features of the representative frame rather than the entire features. We propose a hierarchical technique using background subtraction and HOG, followed by application of a deep neural network and skeletal modelling method. The combination of a CNN and the LSTM recursive network is considered for feature selection and maintaining the previous information; and finally, a Softmax-KNN classifier is used for labelling the human activities. We name our model as “Hierarchical Feature Reduction & Deep Learning”-based action recognition method, or HFR-DL in short. To evaluate the proposed method, we use the UCF101 dataset for the benchmarking which is widely used among researchers in the action recognition research field. The dataset includes 101 complicated activities in the wild. Experimental results show a significant improvement in terms of accuracy and speed in comparison with eight state-of-the-art methods.

Complex System of Vertical Baduanjin Lifting Motion Sensing Recognition under the Background of Big Data

Nowadays, the development of big data is getting faster and faster, and the related research on motion sensing recognition and complex systems under the background of big data is gradually being valued. At present, there are relatively few related researches on vertical Baduanjin in the academic circles; research in this direction can make further breakthroughs in motion sensor recognition. In order to carry out related action recognition research on the lifting action of vertical Baduanjin, this paper uses sensor technology to collect the motion video image of vertical Baduanjin based on the background of big data and uses action recognition technology and related algorithms to obtain the action. Recognize the video image to obtain the data, get the acceleration, angular velocity, and EMG data, and count the end time and duration according to the change of the action. According to the data table and graph change trend compiled at the end of the experiment, we can see the following: after the data is preprocessed, the acceleration signal change range is limited to [−1, 1], and the acceleration change has a clear directionality; and, after 15 lifts of the detected object, its angular velocity in X-axis direction is basically negative. However, when the ninth lift is performed, the angular velocity of the movement in X-axis direction is 36.09, the largest of all angular velocities. When performing the 15th lifting action, the angular velocity of this action in Z-axis direction is −26.05, which is the smallest of all angular velocities. The longest duration of the left muscle discharge during the lifting action of the subject is 15.24 for the tibial anterior muscle and 8.91 for the external oblique muscle with the shortest duration. The longest discharge duration of the right muscle is also the tibial anterior muscle with 12.15, and the shortest duration is the erector spinae with 8.79.

Octave Mix: Data Augmentation Using Frequency Decomposition for Activity Recognition

Research in human activity recognition (HAR) requires a huge amount of data, but it is not easy to collect such measured sensor data. Besides, there is no much application of data augmentation (DA) in HAR. This study proposes Octave Mix as a novel synthetic-style DA method for sensor-based HAR. The proposed method uses frequency decomposition to intersect low- and high-frequency waveforms. In addition, we propose a DA ensemble method and a training algorithm to ensure robustness to the original sensor data while applicable to various feature representations. To evaluate the Octave Mix method’s effectiveness, we conduct experiments using four different benchmark datasets of sensor-based HAR and achieve high estimation accuracy in our results. Furthermore, we demonstrate that ensembling two DA strategies: Octave Mix with rotation and mixup with rotation, achieves higher accuracy.

Method of Analyzing and Managing Volleyball Action by Using Action Sensor of Mobile Device

With the development of electronic technology and sensor technology, more and more intelligent electronic devices integrate micro inertial sensors, which makes the research of human action recognition based on action sensing data have great application value. Data‐based action recognition is a new research direction in the field of pattern recognition, which is essentially a process of action data acquisition, feature extraction, feature extraction, and recognition, the process of classification and recognition. Inertial motion information includes acceleration and angular velocity information, which is ubiquitous in daily life. Compared with motion recognition based on visual information, it can more directly reflect the meaning of action. This study mainly discusses the method of analyzing and managing volleyball action by using the action sensor of mobile device. Based on the motion recognition algorithm of support vector machine, the motion recognition process of support vector machine is constructed. When the data terminal and gateway of volleyball players are not in the same LAN, the classification algorithm classifies the samples to be tested through the characteristic data, which directly affects the recognition results. In this paper, the support vector machine algorithm is selected as the data classification algorithm, and the calculation of the classification process is reduced by designing an appropriate kernel function. For multiclass problems, the hierarchical structure of directed acyclic graph is optimized to improve the recognition rate. We need to bind motion sensors to human joints. In order to realize real‐time recognition of human motion, mobile devices need to add windows to the motion capture data, that is, divide the data into a small sequence of specified length, and provide more application scenarios for the device. This method of embedding motion sensors into devices to read motion information is widely used, which provides a convenient data acquisition method for human motion pattern recognition based on motion information. The multiclassification support vector machine algorithm is used to train the classification algorithm model with action data. When the signal strength of the sensor is 90 t and the speed is 2.0 m/s and 0.5 m/s, the detection accuracy of the adaptive threshold is 93% and 95%, respectively. The results show that the SVM method based on hybrid kernel function can greatly improve the recognition accuracy of volleyball stroke, and the recognition time is short.

Analysis of Deep Neural Networks for Human Activity Recognition in Videos—A Systematic Literature Review

From the past few decades, Human activity recognition (HAR) is one of the vital research areas in computer vision in which much research is ongoing. The researcher’s focus is shifting towards this area due to its vast range of real-life applications to assist in daily living. Therefore, it is necessary to validate its performance on standard benchmark datasets and state-of-the-art systems before applying it in real-life applications. The primary objective of this Systematic Literature Review (SLR) is to collect existing research on video-based human activity recognition, summarize, and analyze the state-of-the-art deep learning architectures regarding various methodologies, challenges, and issues. The top five scientific databases (such as ACM, IEEE, ScienceDirect, SpringerLink, and Taylor & Francis) are accessed to accompany this systematic study by summarizing 70 different research articles on human activity recognition after critical review. Human activity recognition in videos is a challenging problem due to its diverse and complex nature. For accurate video classification, extraction of both spatial and temporal features from video sequences is essential. Therefore, this SLR focuses on reviewing the recent advancements in stratified self-deriving feature-based deep learning architectures. Furthermore, it explores various deep learning techniques available for HAR, challenges researchers to face to build a robust model, and state-of-the-art datasets used for evaluation. This SLR intends to provide a baseline for video-based human activity recognition research while emphasizing several challenges regarding human activity recognition accuracy in video sequences using deep neural architectures.

Sports Dance Action Recognition System Oriented to Human Motion Monitoring and Sensing

Because of its high research value, action recognition has become a very popular research direction in recent years. However, the research on the combination of motion recognition technology and dance movements is still in its infancy. At the same time, due to the high complexity of dance movements and the problems of human body self‐occlusion when performing dances, research on dance video action recognition has been caused. Progress is relatively slow. This article mainly introduces the research of sports dance action recognition system oriented to human motion monitoring and sensing, fully considers the abovementioned problems, and makes in‐depth research and analysis on the current excellent action recognition research content in this field. This paper proposes a research method of sports dance movement recognition for human movement monitoring and sensing, including sports dance movement classification algorithm and sports dance movement preprocessing algorithm, which is used to conduct research experiments on sports dance movement recognition for human movement monitoring and sensing. The experimental results of this article show that the average recognition accuracy of the sports dance action recognition system for human motion monitoring and sensing is 92%, which can be used in daily sports dance training and competition.

TUHAD: Taekwondo Unit Technique Human Action Dataset with Key Frame-Based CNN Action Recognition.

In taekwondo, poomsae (i.e., form) competitions have no quantitative scoring standards, unlike gyeorugi (i.e., full-contact sparring) in the Olympics. Consequently, there are diverse fairness issues regarding poomsae evaluation, and the demand for quantitative evaluation tools is increasing. Action recognition is a promising approach, but the extreme and rapid actions of taekwondo complicate its application. This study established the Taekwondo Unit technique Human Action Dataset (TUHAD), which consists of multimodal image sequences of poomsae actions. TUHAD contains 1936 action samples of eight unit techniques performed by 10 experts and captured by two camera views. A key frame-based convolutional neural network architecture was developed for taekwondo action recognition, and its accuracy was validated for various input configurations. A correlation analysis of the input configuration and accuracy demonstrated that the proposed model achieved a recognition accuracy of up to 95.833% (lowest accuracy of 74.49%). This study contributes to the research and development of taekwondo action recognition.

Complex brain activity analysis and recognition based on multiagent methods

SummaryBrain activity recognition research has been a challenging area for many decades since Hans Berger described electroencephalogram (EEG) in 1929. Many previous researches cannot successfully identify EEG status due to dynamic brain activities and complicated brain correlation. This article adopts multiagent‐based methods to analyze EEG datasets, which can enhance the analytical efficiency through incorporating autonomous, self‐coordination characteristics of agents. Intelligent agents are autonomous applications that can improve system compatibility. The preliminary results indicate that the combination of time‐dependency correlation method with multiagent method is an efficient solution for brain activity recognition.

A Review on Computer Vision-Based Methods for Human Action Recognition.

Human action recognition targets recognising different actions from a sequence of observations and different environmental conditions. A wide different applications is applicable to vision based action recognition research. This can include video surveillance, tracking, health care, and human–computer interaction. However, accurate and effective vision based recognition systems continue to be a big challenging area of research in the field of computer vision. This review introduces the most recent human action recognition systems and provides the advances of state-of-the-art methods. To this end, the direction of this research is sorted out from hand-crafted representation based methods including holistic and local representation methods with various sources of data, to a deep learning technology including discriminative and generative models and multi-modality based methods. Next, the most common datasets of human action recognition are presented. This review introduces several analyses, comparisons and recommendations that help to find out the direction of future research.

Open-Source Data Collection and Data Sets for Activity Recognition in Smart Homes.

As research in smart homes and activity recognition is increasing, it is of ever increasing importance to have benchmarks systems and data upon which researchers can compare methods. While synthetic data can be useful for certain method developments, real data sets that are open and shared are equally as important. This paper presents the E-care@home system, its installation in a real home setting, and a series of data sets that were collected using the E-care@home system. Our first contribution, the E-care@home system, is a collection of software modules for data collection, labeling, and various reasoning tasks such as activity recognition, person counting, and configuration planning. It supports a heterogeneous set of sensors that can be extended easily and connects collected sensor data to higher-level Artificial Intelligence (AI) reasoning modules. Our second contribution is a series of open data sets which can be used to recognize activities of daily living. In addition to these data sets, we describe the technical infrastructure that we have developed to collect the data and the physical environment. Each data set is annotated with ground-truth information, making it relevant for researchers interested in benchmarking different algorithms for activity recognition.

Managing sensor data streams in a smart home application

A challenge in developing an ambient activity recognition system for use in elder care is finding a balance between the sophistication of the system and a cost structure that fits within the budgets of public and private sector healthcare organisations. Much activity recognition research in the context of elder care is based on dense networks of sensors and advanced methods, such as supervised machine learning algorithms. This paper presents the data processing aspects of an activity recognition system based on a simpler, knowledge-based unsupervised approach, designed for a sparse network of sensors. By structuring sensor data management as a streaming system, we provide a simple programming model for the application logic, which facilitates building a fault-tolerant system with the potential for distributed data management within the sensor network. The system, evaluated by a public sector healthcare organisation, constitutes an example of a system that is useful and has a sustainable cost structure.

Coarse-to-Fine Activity Annotation and Recognition Algorithm for Solitary Older Adults

Older adults want to remain independent with dignity for as long as possible, especially the solitary older adults. Activity recognition plays an essential role in elderly care and rehabilitation by detecting activity shifts among the elderly population. Despite over a decade of research and development in activity recognition, accurate and reliable systems for older adults in use are few. We propose an automatic data collecting and labeling system by addressing the annotation issue, and a novel coarse-to-fine activities of daily living(ADLs) recognition algorithm for older adults, by combining supervised and unsupervised machine learning methods. The automatic data collecting and labeling system targets at the annotation issue caused by the diversity of ADLs in free-living situations. Multiple sensors fusion strategy is employed to interpret and annotate the ADLs. Leveraging supervised and unsupervised machine learning methods, we can discover and recognize ambulatory and trivial ADLS for older adults. The performance of the automatic data collecting and labeling system is double-checked in a four days long test. With the reliable ground truth, we evaluate the coarse-to-fine ADLs recognition algorithm. The performance of our algorithm is promising, the recognition accuracy is larger than 91%.

LaM-2SRN: A Method Which Can Enhance Local Features and Detect Moving Objects for Action Recognition

Visual attention mechanism has been widely used in computer vision and plays a vital role in the research of human action recognition. In this paper, we explore a novel moving target detection mechanism for human action recognition and propose a new 3D CNN (3D Convolutional Neural Network) model, dubbed LaM-2SRN (Local Features Enhanced and Moving target detected 2Stream-ResNet) for extracting and learning attention-enhanced spatiotemporal features. The contributions of this paper are as follow: First, the traditional CAM (Class Activation Map) based visual attention algorithm is used to obtain the optical flow information of the human region, thus eliminating the influence of irrelevant optical flow information (such as background clutter). Second, the ViBe algorithm is used to identify the moving target in the continuous frame, retain the optical flow information of the moving target, and make it complement with the optical flow information of the human region, to obtain a complete motion descriptor. After the motion information is marked in the video frame, we put the marked video frames and the original video frames into the 3D CNN and 2D CNN models respectively, to acquired mixed descriptor, which serves as the basis for video classification. Different from most of the previous target detection algorithms, this paper not only detects the human target, but also detects the moving foreground targets in the video, so that the pre-trained CNN model can obtain more complete motion information from videos. We use RGB-only video data for evaluation on two benchmarks: UCF101 and HMDB51, the experimental results demonstrate that our LaM-2SRN is comparable to the previous state-of-the-art algorithms.

ActiPPG: Using deep neural networks for activity recognition from wrist-worn photoplethysmography (PPG) sensors

Abstract Sensor-based activity recognition seeks to provide higher-level knowledge about human activities from multiple sensors such as accelerometer and gyroscope. Thanks to growing ubiquity of sensor-rich smartphones and wearable devices, activity recognition research has made tremendous progress in recent years. Many sensors, such as motion sensors and cameras, have been previously used for this task. In this paper, we present the first work to recognize human activities solely from photoplethysmography (PPG) data. PPG sensors use a light-based technology to sense the rate of blood flow as controlled by the heart's pumping action. Although they are not originally intended to sense body motions and gestures, in this paper, we propose a novel method to extract meaningful features from the PPG to predict human activities. We exploit PPG signals' susceptibility to corruption by noise introduced by motion artifact and extract, as opposed to discard, the motion artifact signals combined with cardiac and respiratory signals to predict the types of activities performed by users. We combine convolutional and recurrent neural networks to predict different types of daily activities (e.g., walking, running, jumping) from the raw PPG signal. Results using data generated by wrist-worn smartwatches of 12 participants demonstrate the feasibility of our approach at predicting five types of activities (standing, walking, jogging, jumping, and sitting), and highlight new insights about how to use PPG sensors for human activity recognition. Results also highlight the importance of extracting motion artifact signals to understand human behaviors.

Advances in human action recognition: an updated survey

Research in human activity recognition (HAR) has seen tremendous growth and continuously receiving attention from both the Computer Vision and the Image Processing communities. Due to the existence of numerous publications in this field, undoubtedly, there have been a number of review papers on this subject that categorise these techniques. Many of the recent works have started to tackle more challenging problems and these proposed techniques are addressing more realistic real-world scenarios. Conspicuously, an updated survey that covers these methods is timely due. To simplify the categorisation, this study takes a two-layer hierarchical approach. At the top level, the categorisation is based on the basic process flow of HAR, i.e. input data-type, features-type, descriptor-type, and classifier-type. At the second layer, each of these components is further subcategorised based on the diversity of the proposed methods. Finally, a remark on the coming popularity of deep learning approach in this field is also given.

Designing Videogames to Crowdsource Accelerometer Data Annotation for Activity Recognition Research.

Human activity recognition using wearable accelerometers can enable in-situ detection of physical activities to support novel human-computer interfaces and interventions. However, developing valid algorithms that use accelerometer data to detect everyday activities often requires large amounts of training datasets, precisely labeled with the start and end times of the activities of interest. Acquiring annotated data is challenging and time-consuming. Applied games, such as human computation games (HCGs) have been used to annotate images, sounds, and videos to support advances in machine learning using the collective effort of "non-expert game players." However, their potential to annotate accelerometer data has not been formally explored. In this paper, we present two proof-of-concept, web-based HCGs aimed at enabling game players to annotate accelerometer data. Using results from pilot studies with Amazon Mechanical Turk players, we discuss key challenges, opportunities, and, more generally, the potential of using applied videogames for annotating raw accelerometer data to support activity recognition research.

UP-Fall Detection Dataset: A Multimodal Approach

Falls, especially in elderly persons, are an important health problem worldwide. Reliable fall detection systems can mitigate negative consequences of falls. Among the important challenges and issues reported in literature is the difficulty of fair comparison between fall detection systems and machine learning techniques for detection. In this paper, we present UP-Fall Detection Dataset. The dataset comprises raw and feature sets retrieved from 17 healthy young individuals without any impairment that performed 11 activities and falls, with three attempts each. The dataset also summarizes more than 850 GB of information from wearable sensors, ambient sensors and vision devices. Two experimental use cases were shown. The aim of our dataset is to help human activity recognition and machine learning research communities to fairly compare their fall detection solutions. It also provides many experimental possibilities for the signal recognition, vision, and machine learning community.

Contextual Action Cues from Camera Sensor for Multi-Stream Action Recognition.

In action recognition research, two primary types of information are appearance and motion information that is learned from RGB images through visual sensors. However, depending on the action characteristics, contextual information, such as the existence of specific objects or globally-shared information in the image, becomes vital information to define the action. For example, the existence of the ball is vital information distinguishing “kicking” from “running”. Furthermore, some actions share typical global abstract poses, which can be used as a key to classify actions. Based on these observations, we propose the multi-stream network model, which incorporates spatial, temporal, and contextual cues in the image for action recognition. We experimented on the proposed method using C3D or inflated 3D ConvNet (I3D) as a backbone network, regarding two different action recognition datasets. As a result, we observed overall improvement in accuracy, demonstrating the effectiveness of our proposed method.

A Comprehensive Survey of Vision-Based Human Action Recognition Methods.

Although widely used in many applications, accurate and efficient human action recognition remains a challenging area of research in the field of computer vision. Most recent surveys have focused on narrow problems such as human action recognition methods using depth data, 3D-skeleton data, still image data, spatiotemporal interest point-based methods, and human walking motion recognition. However, there has been no systematic survey of human action recognition. To this end, we present a thorough review of human action recognition methods and provide a comprehensive overview of recent approaches in human action recognition research, including progress in hand-designed action features in RGB and depth data, current deep learning-based action feature representation methods, advances in human–object interaction recognition methods, and the current prominent research topic of action detection methods. Finally, we present several analysis recommendations for researchers. This survey paper provides an essential reference for those interested in further research on human action recognition.

Domain learning joint with semantic adaptation for human action recognition

Action recognition is a challenging task in the field of computer vision. The deficiency in training samples is a bottleneck problem in the current action recognition research. With the explosive growth of Internet data, some researchers try to use prior knowledge learned from various video sources to assist in recognizing the action video of the target domain, which is called knowledge adaptation. Based on this idea, we propose a novel framework for action recognition, called Semantic Adaptation based on the Vector of Locally Max Pooled deep learned Features (SA-VLMPF). The proposed framework consists of three parts: Two-Stream Fusion Network (TSFN), Vector of Locally Max-Pooled deep learned Features (VLMPF) and Semantic Adaptation Model (SAM). TSFN adopts a cascaded convolution fusion strategy to combine the convolutional features extracted from two-stream network. VLMPF retains the long-term information in videos and removes the irrelevant information by capturing multiple local features and extracting the features with the highest response to action category. SAM first maps the data of the auxiliary domain and the target domain into the high-level semantic representation through the deep network. Then the obtained high-level semantic representations from auxiliary domain are adapted into target domain in order to optimize the target classifier. Compared with the existing methods, the proposed methods can utilize the advantages of deep learning methods in obtaining the high-level semantic information to improve the performance of knowledge adaptation. At the same time, SA-VLMPF can make full use of the auxiliary data to make up for the insufficiency of training samples. Multiple experiments are conducted on several couples of datasets to validate the effectiveness of the proposed framework. The results show that the proposed SA-VLMPF outperforms the state-of-the-art knowledge adaptation methods.