Activity Recognition Algorithms Research Articles

Improvement of the Human Action Recognition Algorithm by the Pre-processing of Input Data

The paper presents an approach to recognizing human actions using an additional preprocessing stage of input data. The growing volumes of video information do not always allow support the quality of data at a high level; this can cause limitations in the further processing of digital data. In this regard, it becomes urgent to introduce an additional stage of image enhancement into the algorithm for recognizing actions in video. The proposed method includes three main steps: image enhancement, constructing a descriptor, and classification. The presented image enhancement stage is based on the combined local and global image processing in the frequency domain. The basic idea in using local alfa-rooting method is to apply it to different disjoint blocks with different sizes. To solve the problem of constructing a descriptor, a three-dimensional microblock dense difference (3D DMD) algorithm is used, which provides a highly oriented representation of image regions by tightly capturing microblocks within each region in several orientations and scales. 3D DMD has several advantages over other methods: higher efficiency compared to existing methods; minimal computational costs when using an integrated image; low dimension; ease of implementation; does not require settings. The presented modification allows to increase productivity by 2-4%.

Human Action Recognition in Smart Cultural Tourism Based on Fusion Techniques of Virtual Reality and SOM Neural Network.

Smart cultural tourism is the development trend of the future tourism industry. Virtual reality is an important tool to realize smart tourism. The reality of virtual reality mainly comes from human-computer interaction, which is closely related to human action recognition technology. Therefore, the research takes human action recognition as the research direction, uses a self-organizing mapping network (SOM) neural network to extract the key frame of action video, combines it with multi-feature vector method to recognize human action, and compares the recognition rate and user satisfaction of different recognition methods. The results show that the recognition rate of multi-feature voting human action recognition algorithm based on SOM neural network is 93.68% on UT-Kinect action, 59.06% on MSRDailyActivity3D, and the overall action recognition time is only 3.59 s. Within six months, the total profit of human-computer interactive virtual reality tourism project with SOM neural network multi-eigenvector as the core algorithm reached 422,000 yuan, and 88% of users expressed satisfaction after use. It shows that the proposed method has a good recognition rate and can give users effective feedback in time. It is hoped that this research has a certain reference value in promoting the development of human motion recognition technology.

Versatile and non-versatile occupational back-support exoskeletons: A comparison in laboratory and field studies.

Assistive strategies for occupational back-support exoskeletons have focused, mostly, on lifting tasks. However, in occupational scenarios, it is important to account not only for lifting but also for other activities. This can be done exploiting human activity recognition algorithms that can identify which task the user is performing and trigger the appropriate assistive strategy. We refer to this ability as exoskeleton versatility. To evaluate versatility, we propose to focus both on the ability of the device to reduce muscle activation (efficacy) and on its interaction with the user (dynamic fit). To this end, we performed an experimental study involving healthy subjects replicating the working activities of a manufacturing plant. To compare versatile and non-versatile exoskeletons, our device, XoTrunk, was controlled with two different strategies. Correspondingly, we collected muscle activity, kinematic variables and users' subjective feedbacks. Also, we evaluated the task recognition performance of the device. The results show that XoTrunk is capable of reducing muscle activation by up to in lifting and in carrying. However, the non-versatile control strategy hindered the users' natural gait (e.g., reduction of hip flexion), which could potentially lower the exoskeleton acceptance. Detecting carrying activities and adapting the control strategy, resulted in a more natural gait (e.g., increase of hip flexion). The classifier analyzed in this work, showed promising performance (online accuracy>91%). Finally, we conducted 9hours of field testing, involving four users. Initial subjective feedbacks on the exoskeleton versatility, are presented at the end of this work.

Deep Learning for Activity Recognition in Older People Using a Pocket-Worn Smartphone.

Activity recognition can provide useful information about an older individual’s activity level and encourage older people to become more active to live longer in good health. This study aimed to develop an activity recognition algorithm for smartphone accelerometry data of older people. Deep learning algorithms, including convolutional neural network (CNN) and long short-term memory (LSTM), were evaluated in this study. Smartphone accelerometry data of free-living activities, performed by 53 older people (83.8 ± 3.8 years; 38 male) under standardized circumstances, were classified into lying, sitting, standing, transition, walking, walking upstairs, and walking downstairs. A 1D CNN, a multichannel CNN, a CNN-LSTM, and a multichannel CNN-LSTM model were tested. The models were compared on accuracy and computational efficiency. Results show that the multichannel CNN-LSTM model achieved the best classification results, with an 81.1% accuracy and an acceptable model and time complexity. Specifically, the accuracy was 67.0% for lying, 70.7% for sitting, 88.4% for standing, 78.2% for transitions, 88.7% for walking, 65.7% for walking downstairs, and 68.7% for walking upstairs. The findings indicated that the multichannel CNN-LSTM model was feasible for smartphone-based activity recognition in older people.

Optimal sampling strategies for characterizing behavior and affect from ambulatory audio recordings.

Advances in mobile and wearable technologies mean it is now feasible to record hours to days of participant behavior in its naturalistic context, a great boon for psychologists interested in family processes and development. While automated activity recognition algorithms exist for a limited set of behaviors, time-consuming human annotations are still required to robustly characterize the vast majority of behavioral and affective markers of interest. This report is the first to date which systematically tests the efficacy of different sampling strategies for characterizing behavior from audio recordings to provide practical guidelines for researchers. Using continuous audio recordings of the daily lives of 11 preschool-aged children, we compared sampling techniques to determine the most accurate and efficient approach. Results suggest that sampling both low and high frequency verbal and overt behaviors is best if samples are short in duration, systematically rather than randomly selected, and sampled to cover at least 12.5% of recordings. Implications for assessment of real-world behavior are discussed. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

MV2Flow

In video action recognition, motion is a very crucial clue, which is usually represented by optical flow. However, optical flow is computationally expensive to obtain, which becomes the bottleneck for the efficiency of traditional action recognition algorithms. In this article, we propose a network called MV2Flow to learn motion representation efficiently from the signals in the compressed domain. To learn the network, three losses are defined. First, we select the classical TV-L1 flow as proxy ground truth to guide the learning. Besides, an unsupervised image reconstruction loss is proposed to further refine it. Moreover, toward the task of action recognition, the above two losses are combined with a motion content loss. To evaluate our approach, extensive experiments on two benchmark datasets UCF-101 and HMDB-51 are conducted. The motion representation generated with our MV2Flow has shown comparable classification performance on action recognition with TV-L1 flow, while operating at an over 200× faster speed. Based on our MV2Flow and 2D-CNN-based network, we have achieved state-of-the-art performance in the compressed domain. With 3D-CNN-based network, we also achieve comparable accuracy with higher inference speed than methods in the decoded domain setting.

Random forest algorithms for recognizing daily life activities using plantar pressure information: a smart-shoe study.

BackgroundWearable activity trackers are regarded as a new opportunity to deliver health promotion interventions. Indeed, while the prediction of active behaviors is currently primarily relying on the processing of accelerometer sensor data, the emergence of smart clothes with multi-sensing capacities is offering new possibilities. Algorithms able to process data from a variety of smart devices and classify daily life activities could therefore be of particular importance to achieve a more accurate evaluation of physical behaviors. This study aims to (1) develop an activity recognition algorithm based on the processing of plantar pressure information provided by a smart-shoe prototype and (2) to determine the optimal hardware and software configurations.MethodSeventeen subjects wore a pair of smart-shoe prototypes composed of plantar pressure measurement insoles, and they performed the following nine activities: sitting, standing, walking on a flat surface, walking upstairs, walking downstairs, walking up a slope, running, cycling, and completing office work. The insole featured seven pressure sensors. For each activity, at least four minutes of plantar pressure data were collected. The plantar pressure data were cut in overlapping windows of different lengths and 167 features were extracted for each window. Data were split into training and test samples using a subject-wise assignment method. A random forest model was trained to recognize activity. The resulting activity recognition algorithms were evaluated on the test sample. A multi hold-out procedure allowed repeating the operation with 5 different assignments. The analytic conditions were modulated to test (1) different window lengths (1–60 seconds), (2) some selected sensor configurations and (3) different numbers of data features.ResultsA window length of 20 s was found to be optimum and therefore used for the rest of the analysis. Using all the sensors and all 167 features, the smart shoes predicted the activities with an average success of 89%. “Running” demonstrated the highest sensitivity (100%). “Walking up a slope” was linked with the lowest performance (63%), with the majority of the false negatives being “walking on a flat surface” and “walking upstairs.” Some 2- and 3-sensor configurations were linked with an average success rate of 87%. Reducing the number of features down to 20 does not alter significantly the performance of the algorithm.ConclusionHigh-performance human behavior recognition using plantar pressure data only is possible. In the future, smart-shoe devices could contribute to the evaluation of daily physical activities. Minimalist configurations integrating only a small number of sensors and computing a reduced number of selected features could maintain a satisfying performance. Future experiments must include a more heterogeneous population.

Vehicle theft recognition from surveillance video based on spatiotemporal attention

Frequent vehicle thefts have a highly detrimental impact on public safety. Thanks to surveillance equipment distributed throughout a city, a large number of videos that can be used to recognize vehicle theft are available. However, vehicle theft behavior has the characteristics of a small criminal target and small movement. Hence, the existing action recognition algorithms cannot be directly applied for the recognition of vehicle theft. In this paper, we propose a method for vehicle theft recognition based on a spatiotemporal attention mechanism. First, a database of vehicle theft is established by collecting videos from the Internet and an existing dataset. Then, we establish a vehicle theft recognition network and introduce a spatiotemporal attention mechanism for application when extracting the spatiotemporal features of theft. Through the learning of adaptive feature weights, the features that contribute most greatly to recognition are emphasized. Simulation experiments show that our proposed algorithm can achieve 97.04% accuracy on the collected vehicle theft database.

Characterizing Peaks in Acceleration Signals–Application to Physical Activity Detection Using Wearable Sensors

The world is getting older by the minute due to rising life expectancy, leading to an urgent need for the continuous monitoring of patients. Tracking human activities without hospitalization has been tackled in the past decade thanks to the advancement of sensing technologies and wearable devices. However, the limited power resources of microcontrollers and the power consumption due to the use of different sensors are two issues that make the recognition process via embedded systems an open research topic to date. Consequently, this paper proposes a low-cost machine learning-based human activity recognition algorithm. It detects cyclic activities from wrist-worn tri-axial accelerometer data and classifies them into four classes. Specifically, a novel and smart peak detection technique is proposed, followed by the extraction of small handcrafted feature vectors, representing the input of a novel machine-learning architecture. These three contributions are approved by experimental results on a 3300-file dataset, showing an accuracy of 99.21% with a low computational cost thanks to an efficient implementation for feature extraction. The effectiveness of the proposed recognition process is validated in real world conditions.

Hip-hop action image recognition based on symmetric algorithm and iterative weighting of dense sampling

With the rapid increase of hip-hop video data, facing a large number of rapidly updated data, automatic, accurate and fast analysis and recognition of relevant information in hip-hop video has become an urgent problem to be solved. An image recognition method of hip-hop action based on symmetric algorithm and iterative weighting of dense sampling is proposed. Firstly, the symmetric action feature of human hip-hop action image is extracted by the feature extraction method based on symmetric algorithm. Then, the symmetric action feature is obtained by fusion of the hip-hop action recognition algorithm based on iterative weighting of dense sampling. Finally, the action feature is obtained based on the fused action feature. The K-nearest neighbor classification learning method is used to recognize the hip-hop action image. The experimental results show that, under the same sample number, the number of omissions of children is 3, the extraction rate is 92.5%; the number of omissions of adults is 0, the extraction rate is 100%. This method can extract the human hip-hop image well, which lays a good foundation for the subsequent image recognition.

Deep CNN, Body Pose, and Body-Object Interaction Features for Drivers’ Activity Monitoring

Automatic recognition and prediction of in-vehicle human activities has a significant impact on the next generation of driver assistance and intelligent autonomous vehicles. In this article, we present a novel single image driver action recognition algorithm inspired by human perception that often focuses selectively on parts of the images to acquire information at specific places which are distinct to a given task. Unlike existing approaches, we argue that human activity is a combination of pose and semantic contextual cues. In detail, we model this by considering the configuration of body joints, their interaction with objects being represented as a pairwise relation to capture the structural information. Our body-pose and body-object interaction representation is built to be semantically rich and meaningful, which is highly discriminative even though it is coupled with a basic linear SVM classifier. We also propose a Multi-stream Deep Fusion Network (MDFN) for combining high-level semantics with CNN features. Our experimental results demonstrate that the proposed approach significantly improves the drivers’ action recognition accuracy on two exacting datasets.

Integrally Cooperative Spatio-Temporal Feature Representation of Motion Joints for Action Recognition

In contemporary research on human action recognition, most methods separately consider the movement features of each joint. However, they ignore that human action is a result of integrally cooperative movement of each joint. Regarding the problem, this paper proposes an action feature representation, called Motion Collaborative Spatio-Temporal Vector (MCSTV) and Motion Spatio-Temporal Map (MSTM). MCSTV comprehensively considers the integral and cooperative between the motion joints. MCSTV weighted accumulates limbs’ motion vector to form a new vector to account for the movement features of human action. To describe the action more comprehensively and accurately, we extract key motion energy by key information extraction based on inter-frame energy fluctuation, project the energy to three orthogonal axes and stitch them in temporal series to construct the MSTM. To combine the advantages of MSTM and MCSTV, we propose Multi-Target Subspace Learning (MTSL). MTSL projects MSTM and MCSTV into a common subspace and makes them complement each other. The results on MSR-Action3D and UTD-MHAD show that our method has higher recognition accuracy than most existing human action recognition algorithms.

Application of Crowd Simulations in the Evaluation of Tracking Algorithms.

Tracking and action-recognition algorithms are currently widely used in video surveillance, monitoring urban activities and in many other areas. Their development highly relies on benchmarking scenarios, which enable reliable evaluations/improvements of their efficiencies. Presently, benchmarking methods for tracking and action-recognition algorithms rely on manual annotation of video databases, prone to human errors, limited in size and time-consuming. Here, using gained experiences, an alternative benchmarking solution is presented, which employs methods and tools obtained from the computer-game domain to create simulated video data with automatic annotations. Presented approach highly outperforms existing solutions in the size of the data and variety of annotations possible to create. With proposed system, a potential user can generate a sequence of random images involving different times of day, weather conditions, and scenes for use in tracking evaluation. In the design of the proposed tool, the concept of crowd simulation is used and developed. The system is validated by comparisons to existing methods.

Energy-Guided Temporal Segmentation Network for Multimodal Human Action Recognition.

To achieve the satisfactory performance of human action recognition, a central task is to address the sub-action sharing problem, especially in similar action classes. Nevertheless, most existing convolutional neural network (CNN)-based action recognition algorithms uniformly divide video into frames and then randomly select the frames as inputs, ignoring the distinct characteristics among different frames. In recent years, depth videos have been increasingly used for action recognition, but most methods merely focus on the spatial information of the different actions without utilizing temporal information. In order to address these issues, a novel energy-guided temporal segmentation method is proposed here, and a multimodal fusion strategy is employed with the proposed segmentation method to construct an energy-guided temporal segmentation network (EGTSN). Specifically, the EGTSN had two parts: energy-guided video segmentation and a multimodal fusion heterogeneous CNN. The proposed solution was evaluated on a public large-scale NTU RGB+D dataset. Comparisons with state-of-the-art methods demonstrate the effectiveness of the proposed network.

Human body activity recognition using wearable inertial sensors integrated with a feature extraction–based machine-learning classification algorithm

This study employed wearable inertial sensors integrated with an activity-recognition algorithm to recognize six types of daily activities performed by humans, namely walking, ascending stairs, descending stairs, sitting, standing, and lying. The sensor system consisted of a microcontroller, a three-axis accelerometer, and a three-axis gyro; the algorithm involved collecting and normalizing the activity signals. To simplify the calculation process and to maximize the recognition accuracy, the data were preprocessed through linear discriminant analysis; this reduced their dimensionality and captured their features, thereby reducing the feature space of the accelerometer and gyro signals; they were then verified through the use of six classification algorithms. The new contribution is that after feature extraction, data classification results indicated that an artificial neural network was the most stable and effective of the six algorithms. In the experiment, 20 participants equipped the wearable sensors on their waists to record the aforementioned six types of daily activities and to verify the effectiveness of the sensors. According to the cross-validation results, the combination of linear discriminant analysis and an artificial neural network was the most stable classification algorithm for data generalization; its activity-recognition accuracy was 87.37% on the training data and 80.96% on the test data.

FSD-10: A fine-grained classification dataset for figure skating

Action recognition is an important and challenging problem in video analysis. Although the past decade has witnessed progress in action recognition with the development of deep learning, such process has been slow in competitive sports content analysis. To promote the research on action recognition from competitive sports video clips, we introduce a Figure Skating Dataset (FSD-10) for fine-grained sports content analysis. To this end, we collect 1484 clips from the worldwide figure skating championships in 2017–2018, which consist of 10 different actions in men/ladies programs. Each clip is at a rate of 30 frames per second with resolution 1080 × 720, which are annotated by experts. To build a baseline for action recognition in figure skating, we evaluate state-of-the-art action recognition methods on FSD-10. Motivated by the idea that domain knowledge is of great concern in sports field, we propose a keyframe based temporal segment network (KTSN) for classification and achieve remarkable performance. Experimental results demonstrate that FSD-10 is an ideal dataset for benchmarking action recognition algorithms, as it requires to accurately extract action motions rather than action poses. We hope FSD-10, which is designed to have a large collection of finegrained actions, can serve as a new challenge to develop more robust and advanced action recognition models.

Human Action Recognition Algorithm Based on Improved ResNet and Skeletal Keypoints in Single Image

Human action recognition is an important part for computers to understand the behavior of people in pictures or videos. In a single image, there is no context information for recognition, so its accuracy still needs to be greatly improved. In this paper, a single-image human action recognition method based on improved ResNet and skeletal keypoints is proposed, and the accuracy is improved by several methods. We improved the backbone network ResNet-50 and CPN to a certain extent and constructed a multitask network to suit the human action recognition task, which not only improves the accuracy but also balances the total number of parameters and solves the problem of large network and slow operation. In this paper, the improvement methods of ResNet-50, CPN, and whole network are tested, respectively. The results show that the single-image human action recognition based on improved ResNet and skeletal keypoints can accurately identify human action in the case of different human movements, different background light, and occlusion. Compared with the original network and the main human action recognition algorithms, the accuracy of our method has its certain advantages.

Wearable Internet-of-Things platform for human activity recognition and health care

We propose to perform wearable sensors-based human physical activity recognition. This is further extended to an Internet-of-Things (IoT) platform which is based on a web-based application that integrates wearable sensors, smartphones, and activity recognition. To this end, a smartphone collects the data from wearable sensors and sends it to the server for processing and recognition of the physical activity. We collect a novel data set of 13 physical activities performed both indoor and outdoor. The participants are from both the genders where their number per activity varies. During these activities, the wearable sensors measure various body parameters via accelerometers, gyroscope, magnetometers, pressure, and temperature. These measurements and their statistical are then represented in features vectors that used to train and test supervised machine learning algorithms (classifiers) for activity recognition. On the given data set, we evaluate a number of widely known classifiers such random forests, support vector machine, and many others using the WEKA machine learning suite. Using the default settings of these classifiers in WEKA, we attain the highest overall classification accuracy of 90%. Consequently, such a recognition rate is encouraging, reliable, and effective to be used in the proposed platform.

Deep-Aligned Convolutional Neural Network for Skeleton-Based Action Recognition and Segmentation

Convolutional neural networks (CNNs) are deep learning frameworks which are well known for their notable performance in classification tasks. Hence, many skeleton-based action recognition and segmentation (SBARS) algorithms benefit from them in their designs. However, a shortcoming of such applications is the general lack of spatial relationships between the input features in such data types. Besides, non-uniform temporal scalings are a common issue in skeleton-based data streams which leads to having different input sizes even within one specific action category. In this work, we propose a novel deep-aligned convolutional neural network (DACNN) to tackle the above challenges for the particular problem of SBARS. Our network is designed by introducing a new type of filters in the context of CNNs which are trained based on their alignments to the local subsequences in the inputs. These filters result in efficient predictions as well as learning interpretable patterns in the data. Also, our DACNN framework can incrementally expand its deep structure based on the learning progress, which makes it flexible regarding different SBARS datasets. We empirically evaluate our framework on real-world benchmarks showing that the proposed DACNN algorithm obtains a competitive performance compared to the state of the art while benefiting from a less complicated yet more interpretable model.

Research on Human Action Recognition Algorithm Based on Sine Feature

Recent studies have shown that exploring features of the skeleton data is vital for human action recognition. Nevertheless, how to effectively extract discriminative features is still a challenging work. In this paper, we propose a novel method that extracts a sine feature for human action recognition from skeleton data. Kinect is used to extract human skeleton information (3D coordinates of each joint point) firstly, then two joint points are connected to define a skeleton vector according to the principle of human body structure, and the sine feature of each skeleton vector is calculated as a new pose description feature, SVM is used to classify the obtained features for human action classification. Experimental results on Cornell Activity Database are provided, and the results demonstrate the effectiveness of our approach.