Visual Action Recognition Research Articles

Skeleton-based Human Activity Classification in Sparse Image Sequences

Research results on human activity classification in video are described, based on initial human skeleton estimation in video frames. Both single person actions and two-person interactions are considered. The initial skeleton data is estimated in selected video frames by OpenPose, HRNet or other dedicated library. Important contributions of presented work are computational steps of skeleton tracking and -refinement, and relational feature extraction from pairs of skeleton joints. It is shown, that this feature engineering significantly increases the classification accuracy. Regarding the final neural network encoder-classifier, two different architectures are designed and tested. The first solution is a lightweight MLP network, implementing the idea of a "mixture of pose experts". Several pose classifiers (experts) are trained independently on different time periods (snapshots) of single-person visual actions (or 2-person interactions), while the final classification is a time-related pooling of weighted expert classifications. All pose experts use the same deep encoding network. The second (middle weight) solution is based on a LSTM network.Both solutions are trained and tested on the action set of the well-known NTU RGB+D dataset, although only 2D data are used.Our results show comparable performance with some of the best reported STM- and CNN-based classifiers for this dataset. We conclude that by reducing the noise of skeleton data, highly successful lightweight- and midweight-approaches to visual activity recognition in image sequences can be achieved.

Action Recognition for Human–Robot Teaming: Exploring Mutual Performance Monitoring Possibilities

Human–robot teaming (HrT) is being adopted in an increasing range of industries and work environments. Effective HrT relies on the success of complex and dynamic human–robot interaction. Although it may be optimal for robots to possess all the social and emotional skills to function as productive team members, certain cognitive capabilities can enable them to develop attitude-based competencies for optimizing teams. Despite the extensive research into the human–human team structure, the domain of HrT research remains relatively limited. In this sense, incorporating established human–human teaming (HhT) elements may prove practical. One key element is mutual performance monitoring (MPM), which involves the reciprocal observation and active anticipation of team members’ actions within the team setting, fostering enhanced team coordination and communication. By adopting this concept, this study uses ML-based visual action recognition as a potential tool for developing an effective way to monitor the human component in HrT. This study utilizes a data modeling approach on an existing dataset, the “Industrial Human Action Recognition Dataset” (InHARD), curated specifically for human action recognition assembly tasks in industrial environments involving human–robot collaborations. This paper presents the results of this modeling approach in analyzing the dataset to implement a theoretical concept that can be a first step toward enabling the system to adapt dynamically. The outcomes emphasize the significance of implementing state-of-the-art team concepts by integrating modern technologies and assessing the possibility of advancing HrT in this direction.

Retracted: Construction of Intelligent Nursing System Based on Visual Action Recognition Algorithm.

[This retracts the article DOI: 10.1155/2022/8311371.].

Interaction-Aware Spatio-Temporal Pyramid Attention Networks for Action Classification.

For CNN-based visual action recognition, the accuracy may be increased if local key action regions are focused on. The task of self-attention is to focus on key features and ignore irrelevant information. So, self-attention is useful for action recognition. However, current self-attention methods usually ignore correlations among local feature vectors at spatial positions in CNN feature maps. In this paper, we propose an effective interaction-aware self-attention model which can extract information about the interactions between feature vectors to learn attention maps. Since the different layers in a network capture feature maps at different scales, we introduce a spatial pyramid with the feature maps at different layers for attention modeling. The multi-scale information is utilized to obtain more accurate attention scores. These attention scores are used to weight the local feature vectors of the feature maps and then calculate attentional feature maps. Since the number of feature maps input to the spatial pyramid attention layer is unrestricted, we easily extend this attention layer to a spatio-temporal version. Our model can be embedded in any general CNN to form a video-level end-to-end attention network for action recognition. Several methods are investigated to combine the RGB and flow streams to obtain accurate predictions of human actions. Experimental results show that our method achieves state-of-the-art results on the datasets UCF101, HMDB51, Kinetics-400, and untrimmed Charades.

Construction of Intelligent Nursing System Based on Visual Action Recognition Algorithm.

Due to advancements in scientific research and technological innovation, as well as the proliferation of the Internet of things, the Internet, and big data, the general public has gradually become aware of a new type of intelligent nursing system model known as the smart nursing system. The smart nursing system is a sensing system and information platform for the elderly in their homes, communities, and institutions for elderly care. Based on this, it provides timely, efficient, and cost-effective elderly care services in real time. These services utilize the Internet of things and the Internet as well. Through the monitoring of video data, we are able to differentiate the visual motions of these elderly individuals and determine whether they are in a normal life state or a fall state. This has the potential to better meet the diverse and multifaceted needs of senior citizens, enhance the quality of life of senior citizens in their final years, and provide senior citizens with greater humanistic care and spiritual solace. Our team has developed an intelligent nursing system based on the visual action recognition algorithm, also known as the deep learning (DL) algorithm. As a result of our simulation tests, we discovered that the algorithm can accurately identify the living situations of elderly individuals at home.

Long Jump Action Recognition Based on Deep Convolutional Neural Network.

Long jump is a test item of national student physical health monitoring, which can reflect the quality of students' lower limb strength. Long jump is a highly technical activity, which includes four basic movements: running aid, jumping, vacating, and landing. Many students have problems with the technical aspects, resulting in test scores that do not objectively reflect the true physical condition of the students, which affects the accuracy of the test results. From the perspective of rapid diagnostic feedback of students' long jump movements, we design and develop a long jump movement recognition method based on deep convolutional neural network. In this paper, we firstly summarize the traditional visual action recognition algorithm, then apply 3D convolution to extract the spatiotemporal features of long jump action from three directions of the video block, and fuse the spatiotemporal features of the three directions in different ways to achieve feature complementation; finally, using the multimodality of long jump action data, we use 3D convolutional neural network to train the RGB images and then train the depth. This joint training method can accelerate the convergence speed and improve the accuracy of the network on both depth and edge images. The experiments compared the recognition effects of the tandem fusion of features, the maximum fusion, and the multiplicative fusion in the scoring layer, and the highest accuracy of 82.3% was achieved by the tandem fusion of features with the fusion of three modalities.

Privacy preserving human activity recognition framework using an optimized prediction algorithm

Human activity recognition, in computer vision research, is the area of growing interest as it has plethora of real-world applications. Inferring actions from one or more persons captured through a live video has its immense utility in the contemporary era. Same time, protecting privacy of humans is to be given paramount importance. Many researchers contributed towards this end leading to privacy preserving action recognition systems. However, having an optimized model that can withstand any adversary models that strives to disclose privacy information. To address this problem, we proposed an algorithm known optimized prediction algorithm for privacy preserving activity recognition (OPA-PPAR) based on deep neural networks. It anonymizes video content to have adaptive privacy model that defeats attacks from adversaries. The privacy model enhances the privacy of humans while permitting highly accurate approach towards action recognition. The algorithm is implemented to realize privacy preserving human activity recognition framework (PPHARF). The visual recognition of human actions is made using an underlying adversarial learning process where the anonymization is optimized to have an adaptive privacy model. A dataset named human metabolome database (HMDB51) is used for empirical study. Our experiments with using Python data science platform reveal that the OPA-PPAR outperforms existing methods.

Retraction Note to: Image recognition of coastal environment and aerobics sports based on remote sensing images based on deep learning

When analyzing the environmental development of coastal areas, the main basis used by researchers for analysis is the data information collected by remote sensing sensors. In the process of research and analysis, experts divided the areas displayed by remote sensing images into three types based on the characteristics of ocean areas, land, and intermediate zones, namely, sea areas, coastal beaches, and land areas. Today, with the continuous development of science and technology, people can choose the most appropriate image research method according to the development characteristics of different regions to better study the spatial distribution and geographic distribution of images. In the current era of development, the level of development of robot services is constantly improving. In order to improve the intelligence of robot service functions, people have begun to study the ability of machines to recognize human activities, and strive to meet the needs of development in all aspects. This article analyzes the characteristics of robot visual action recognition, compares and analyzes three different action representation methods, and uses deep learning related technologies to improve the level of action recognition.

Human Behavior Analysis: A Survey on Action Recognition

The visual recognition and understanding of human actions remain an active research domain of computer vision, being the scope of various research works over the last two decades. The problem is challenging due to its many interpersonal variations in appearance and motion dynamics between humans, without forgetting the environmental heterogeneity between different video images. This complexity splits the problem into two major categories: action classification, recognising the action being performed in the scene, and spatiotemporal action localisation, concerning recognising multiple localised human actions present in the scene. Previous surveys mainly focus on the evolution of this field, from handcrafted features to deep learning architectures. However, this survey presents an overview of both categories and respective evolution within each one, the guidelines that should be followed and the current benchmarks employed for performance comparison between the state-of-the-art methods.

RETRACTED ARTICLE: Image recognition of coastal environment and aerobics sports based on remote sensing images based on deep learning

When analyzing the environmental development of coastal areas, the main basis used by researchers for analysis is the data information collected by remote sensing sensors. In the process of research and analysis, experts divided the areas displayed by remote sensing images into three types based on the characteristics of ocean areas, land, and intermediate zones, namely, sea areas, coastal beaches, and land areas. Today, with the continuous development of science and technology, people can choose the most appropriate image research method according to the development characteristics of different regions to better study the spatial distribution and geographic distribution of images. In the current era of development, the level of development of robot services is constantly improving. In order to improve the intelligence of robot service functions, people have begun to study the ability of machines to recognize human activities, and strive to meet the needs of development in all aspects. This article analyzes the characteristics of robot visual action recognition, compares and analyzes three different action representation methods, and uses deep learning related technologies to improve the level of action recognition.

Minimal videos: Trade-off between spatial and temporal information in human and machine vision

Objects and their parts can be visually recognized from purely spatial or purely temporal information but the mechanisms integrating space and time are poorly understood. Here we show that visual recognition of objects and actions can be achieved by efficiently combining spatial and motion cues in configurations where each source on its own is insufficient for recognition. This analysis is obtained by identifying minimal videos: these are short and tiny video clips in which objects, parts, and actions can be reliably recognized, but any reduction in either space or time makes them unrecognizable. Human recognition in minimal videos is invariably accompanied by full interpretation of the internal components of the video. State-of-the-art deep convolutional networks for dynamic recognition cannot replicate human behavior in these configurations. The gap between human and machine vision demonstrated here is due to critical mechanisms for full spatiotemporal interpretation that are lacking in current computational models.

Histogram of Fuzzy Local Spatio-Temporal Descriptors for Video Action Recognition

Feature extraction plays a vital role in visual action recognition. Many existing gradient-based feature extractors, including histogram of oriented gradients, histogram of optical flow, motion boundary histograms, and histogram of motion gradients, build histograms for representing different actions over the spatio-temporal domain in a video. However, these methods require to set the number of bins for information aggregation in advance. Varying numbers of bins usually lead to inherent uncertainty within the process of pixel voting with regard to the bins in the histogram. This article proposes a novel method to handle such uncertainty by fuzzifying these feature extractors. The proposed approach has two advantages: it better represents the ambiguous boundaries between the bins and, thus, the fuzziness of the spatio-temporal visual information entailed in videos; and the contribution of each pixel is flexibly controlled by a fuzziness parameter for various scenarios. The proposed family of fuzzy descriptors and a combination of them are evaluated on two publicly available datasets, demonstrating that the proposed approach outperforms the original counterparts and other state-of-the-art methods.

Exploring the effects of non-local blocks on video captioning networks

In addition to visual features, video also contains temporal information that contributes to semantic meaning regarding the relationships between objects and scenes. There have been many attempts to describe spatial and temporal relationships in video, but simple encoder-decoder models are not sufficient for capturing detailed relationships in video clips. A video clip often consists of several shots that seem to be unrelated, and simple recurrent models suffer from these changes in shots. In other fields, including visual question answering and action recognition, researchers began to have interests in describing visual relations between the objects. In this paper, we introduce a video captioning method to capture temporal relationships with a non-local block and boundary-aware system. We evaluate our approach on a Microsoft video description Corpus (MSVD, YouTube2Text) dataset and a Microsoft research-video to text (MSR-VTT) dataset. The experimental results show that a non-local block applied along a temporal axis can improve video captioning performance on video captioning datasets.

Recognizing Human Activities From Video Using Weakly Supervised Contextual Features

Human activity recognition from the video is an important problem due to its potential applications in remote surveillance, content-based video retrieval, and in humanoid robots. Most of the visual activity recognition research in the last decade focus on recognizing basic human actions from well-constrained laboratory videos and depend upon fully annotated video dataset for model training. Whereas activity recognition from unconstrained videos is still very challenging due to large variations in object appearance and pose, occlusion, and inter and intraclass variations. It is an extremely laborious task to prepare large scale realistic video activity dataset with detailed annotations of the human, object, and their mutual interactions in each frame. Although it is intuitive to model contextual relationships form fully annotated dataset, however, it is unknown as to how reliably multilevel contextual features can be extracted in the absence of annotated dataset. To mitigate these challenges, we propose a weakly supervised approach for complex human activity recognition from realistic videos. The proposed approach requires only activity labels for each video to train the model. A novel multilevel contextual features and context estimation procedure from the un-annotated dataset is also introduced. Restricted Boltzman machine is used to systematically integrate multilevel contextual features. We evaluate the proposed approach on benchmark realistic surveillance video datasets for human-human and human-object interaction activity recognition. The experimental results show improved accuracies on benchmark datasets without using fully annotated datasets.

A self-organizing neural network architecture for learning human-object interactions

Abstract The visual recognition of transitive actions comprising human-object interactions is a key component for artificial systems operating in natural environments. This challenging task requires jointly the recognition of articulated body actions as well as the extraction of semantic elements from the scene such as the identity of the manipulated objects. In this paper, we present a self-organizing neural network for the recognition of human-object interactions from RGB-D videos. Our model consists of a hierarchy of Grow-When-Required (GWR) networks that learn prototypical representations of body motion patterns and objects, accounting for the development of action-object mappings in an unsupervised fashion. We report experimental results on a dataset of daily activities collected for the purpose of this study as well as on a publicly available benchmark dataset. In line with neurophysiological studies, our self-organizing architecture exhibits higher neural activation for congruent action-object pairs learned during training sessions with respect to synthetically created incongruent ones. We show that our unsupervised model shows competitive classification results on the benchmark dataset with respect to strictly supervised approaches.

Online Recognition of Daily Activities by Color-Depth Sensing and Knowledge Models.

Visual activity recognition plays a fundamental role in several research fields as a way to extract semantic meaning of images and videos. Prior work has mostly focused on classification tasks, where a label is given for a video clip. However, real life scenarios require a method to browse a continuous video flow, automatically identify relevant temporal segments and classify them accordingly to target activities. This paper proposes a knowledge-driven event recognition framework to address this problem. The novelty of the method lies in the combination of a constraint-based ontology language for event modeling with robust algorithms to detect, track and re-identify people using color-depth sensing (Kinect® sensor). This combination enables to model and recognize longer and more complex events and to incorporate domain knowledge and 3D information into the same models. Moreover, the ontology-driven approach enables human understanding of system decisions and facilitates knowledge transfer across different scenes. The proposed framework is evaluated with real-world recordings of seniors carrying out unscripted, daily activities at hospital observation rooms and nursing homes. Results demonstrated that the proposed framework outperforms state-of-the-art methods in a variety of activities and datasets, and it is robust to variable and low-frame rate recordings. Further work will investigate how to extend the proposed framework with uncertainty management techniques to handle strong occlusion and ambiguous semantics, and how to exploit it to further support medicine on the timely diagnosis of cognitive disorders, such as Alzheimer’s disease.

Action recognition is sensitive to the identity of the actor

Recognizing who is carrying out an action is essential for successful human interaction. The cognitive mechanisms underlying this ability are little understood and have been subject of discussions in embodied approaches to action recognition. Here we examine one solution, that visual action recognition processes are at least partly sensitive to the actor’s identity. We investigated the dependency between identity information and action related processes by testing the sensitivity of neural action recognition processes to clothing and facial identity information with a behavioral adaptation paradigm. Our results show that action adaptation effects are in fact modulated by both clothing information and the actor’s facial identity. The finding demonstrates that neural processes underlying action recognition are sensitive to identity information (including facial identity) and thereby not exclusively tuned to actions. We suggest that such response properties are useful to help humans in knowing who carried out an action.

Visual adaptation dominates bimodal visual-motor action adaptation.

A long standing debate revolves around the question whether visual action recognition primarily relies on visual or motor action information. Previous studies mainly examined the contribution of either visual or motor information to action recognition. Yet, the interaction of visual and motor action information is particularly important for understanding action recognition in social interactions, where humans often observe and execute actions at the same time. Here, we behaviourally examined the interaction of visual and motor action recognition processes when participants simultaneously observe and execute actions. We took advantage of behavioural action adaptation effects to investigate behavioural correlates of neural action recognition mechanisms. In line with previous results, we find that prolonged visual exposure (visual adaptation) and prolonged execution of the same action with closed eyes (non-visual motor adaptation) influence action recognition. However, when participants simultaneously adapted visually and motorically – akin to simultaneous execution and observation of actions in social interactions - adaptation effects were only modulated by visual but not motor adaptation. Action recognition, therefore, relies primarily on vision-based action recognition mechanisms in situations that require simultaneous action observation and execution, such as social interactions. The results suggest caution when associating social behaviour in social interactions with motor based information.

Action Recognition and Movement Direction Discrimination Tasks Are Associated with Different Adaptation Patterns.

The ability to discriminate between different actions is essential for action recognition and social interactions. Surprisingly previous research has often probed action recognition mechanisms with tasks that did not require participants to discriminate between actions, e.g., left-right direction discrimination tasks. It is not known to what degree visual processes in direction discrimination tasks are also involved in the discrimination of actions, e.g., when telling apart a handshake from a high-five. Here, we examined whether action discrimination is influenced by movement direction and whether direction discrimination depends on the type of action. We used an action adaptation paradigm to target action and direction discrimination specific visual processes. In separate conditions participants visually adapted to forward and backward moving handshake and high-five actions. Participants subsequently categorized either the action or the movement direction of an ambiguous action. The results showed that direction discrimination adaptation effects were modulated by the type of action but action discrimination adaptation effects were unaffected by movement direction. These results suggest that action discrimination and direction categorization rely on partly different visual information. We propose that action discrimination tasks should be considered for the exploration of visual action recognition mechanisms.

Neural model for multi-stability in visual action recognition

The visual perception of body movements shows interesting dynamical properties. Biological motion perception can show perceptual multi-stability with respect to the perceived walking direction [1]. In addition, body motion perception is subject to adaptation, as is demonstrated by the existence of high-level after-effects and of fMRI repetition suppression in relevant areas. Existing neural models for action recognition do not account for these phenomena. We present neurodynamical model that reproduces these phenomena, and which allows to study the interplay between multi-stability, adaptation and intrinsic fluctuations in body motion perception.