Human Interaction Recognition Research Articles

Interactive semantics neural networks for skeleton-based human interaction recognition

Interactive semantics neural networks for skeleton-based human interaction recognition

Proxemics-net++: classification of human interactions in still images

Human interaction recognition (HIR) is a significant challenge in computer vision that focuses on identifying human interactions in images and videos. HIR presents a great complexity due to factors such as pose diversity, varying scene conditions, or the presence of multiple individuals. Recent research has explored different approaches to address it, with an increasing emphasis on human pose estimation. In this work, we propose Proxemics-Net++, an extension of the Proxemics-Net model, capable of addressing the problem of recognizing human interactions in images through two different tasks: the identification of the types of “touch codes” or proxemics and the identification of the type of social relationship between pairs. To achieve this, we use RGB and body pose information together with the state-of-the-art deep learning architecture, ConvNeXt, as the backbone. We performed an ablative analysis to understand how the combination of RGB and body pose information affects these two tasks. Experimental results show that body pose information contributes significantly to proxemic recognition (first task) as it allows to improve the existing state of the art, while its contribution in the classification of social relations (second task) is limited due to the ambiguity of labelling in this problem, resulting in RGB information being more influential in this task.

Multi-Modal Enhancement Transformer Network for Skeleton-Based Human Interaction Recognition.

Skeleton-based human interaction recognition is a challenging task in the field of vision and image processing. Graph Convolutional Networks (GCNs) achieved remarkable performance by modeling the human skeleton as a topology. However, existing GCN-based methods have two problems: (1) Existing frameworks cannot effectively take advantage of the complementary features of different skeletal modalities. There is no information transfer channel between various specific modalities. (2) Limited by the structure of the skeleton topology, it is hard to capture and learn the information about two-person interactions. To solve these problems, inspired by the human visual neural network, we propose a multi-modal enhancement transformer (ME-Former) network for skeleton-based human interaction recognition. ME-Former includes a multi-modal enhancement module (ME) and a context progressive fusion block (CPF). More specifically, each ME module consists of a multi-head cross-modal attention block (MH-CA) and a two-person hypergraph self-attention block (TH-SA), which are responsible for enhancing the skeleton features of a specific modality from other skeletal modalities and modeling spatial dependencies between joints using the specific modality, respectively. In addition, we propose a two-person skeleton topology and a two-person hypergraph representation. The TH-SA block can embed their structural information into the self-attention to better learn two-person interaction. The CPF block is capable of progressively transforming the features of different skeletal modalities from low-level features to higher-order global contexts, making the enhancement process more efficient. Extensive experiments on benchmark NTU-RGB+D 60 and NTU-RGB+D 120 datasets consistently verify the effectiveness of our proposed ME-Former by outperforming state-of-the-art methods.

WiFi-TCN: Temporal Convolution for Human Interaction Recognition Based on WiFi Signal

WiFi-TCN: Temporal Convolution for Human Interaction Recognition Based on WiFi Signal

Secure and Privacy-Preserving Human Interaction Recognition of Pervasive Healthcare Monitoring

A cloud-based Artificial Intelligence (AI) service has recently empowered the Internet of Medical Things (IoMT) in many applications on the remote Human Interaction Recognition of Pervasive Healthcare Monitoring (HIR-PHM). In this work, a framework of the skeleton-based HIR-PHM under secure Edge-Fog-Cloud computing (EFCC) was proposed to manage the computation and storage resources, latency, cyber-attack, and privacy-preserving simultaneously. At the Edge, IoMT with a camera, record the human interaction as videos and sends them to the Fog, which installed a human pose estimation model, PoseNet, to convert the videos into human skeleton data. At the Cloud, a skeleton-based Spatial-Temporal Graph Convolution Network with Pairwise Adjacency Matrix (STGCN-PAM) was employed to recognize the human interaction. A Hybrid of One Time Password (OTP), Hashed Messages Authentication Code (HMAC), and symmetric cipher were employed to secure the skeleton data. The UT-Interaction dataset was used to evaluate the proposed framework. Besides, the computation performance and latency under EFCC were compared with Edge-Fog and Edge-Cloud deployments. Experimental results confirmed two major contributions: 1) the proposed framework has promising performance compared with other methods on the dataset. 2) The skeleton-based HIR-PHM with secure EFCC shows the advantages of better computation performance in terms of frame per second.

Two-Person Graph Convolutional Network for Skeleton-Based Human Interaction Recognition

Graph convolutional networks (GCNs) have been the predominant methods in skeleton-based human action recognition, including human-human interaction recognition. However, when dealing with interaction sequences, current GCN-based methods simply split the two-person skeleton into two discrete graphs and perform graph convolution separately as done for single-person action classification. Such operations ignore rich interactive information and hinder effective spatial inter-body relationship modeling. To overcome the above shortcoming, we introduce a novel unified two-person graph to represent inter-body and intra-body correlations between joints. Experimental results show accuracy improvements in recognizing both interactions and individual actions when utilizing the proposed two-person graph topology. In addition, several graph labeling strategies are designed to supervise the model to learn discriminant spatial-temporal interactive features. Finally, we propose a two-person graph convolutional network (2P-GCN). Our model outperforms state-of-the-art methods on four benchmarks of three interaction datasets: SBU, interaction subsets of NTU-RGB+D and NTU-RGB+D 120.

Recognition of Human Interactions in Still Images using AdaptiveDRNet with Multi-level Attention

Recognition of Human Interactions in Still Images using AdaptiveDRNet with Multi-level Attention

Standard Deviation Labeling A New Partitioning Strategy for Spatial Temporal Graph Convolutional Networks in Human Interaction Recognition

Standard Deviation Labeling A New Partitioning Strategy for Spatial Temporal Graph Convolutional Networks in Human Interaction Recognition

Human gesture interaction recognition algorithm based on augmented reality in smart wristband equipment

Human gesture interaction recognition algorithm based on augmented reality in smart wristband equipment

Human gesture interaction recognition algorithm based on augmented reality in smart wristband equipment

Human gesture interaction recognition algorithm based on augmented reality in smart wristband equipment

A Multi-Stream Sequence Learning Framework for Human Interaction Recognition

Human interaction recognition (HIR) is challenging due to multiple humans’ involvement and their mutual interaction in a single frame, generated from their movements. Mainstream literature is based on three-dimensional (3-D) convolutional neural networks (CNNs), processing only visual frames, where human joints data play a vital role in accurate interaction recognition. Therefore, this article proposes a multistream network for HIR that intelligently learns from skeletons’ key points and spatiotemporal visual representations. The first stream localises the joints of the human body using a pose estimation model and transmits them to a 1-D CNN and bidirectional long short-term memory to efficiently extract the features of the dynamic movements of each human skeleton. The second stream feeds the series of visual frames to a 3-D convolutional neural network to extract the discriminative spatiotemporal features. Finally, the outputs of both streams are integrated via fully connected layers that precisely classify the ongoing interactions between humans. To validate the performance of the proposed network, we conducted a comprehensive set of experiments on two benchmark datasets, UT-interaction and TV human interaction, and found 1.15% and 10.0% improvement in the accuracy.

Human interaction recognition framework based on interacting body part attention

Human activity recognition in videos has been widely studied and has recently gained significant advances with deep learning approaches; however, it remains a challenging task. In this paper, we propose a novel framework that simultaneously considers both implicit and explicit representations of human interactions by fusing information of local image where the interaction actively occurred, primitive motion with the posture of individual subject’s body parts, and the co-occurrence of overall appearance change. Human interactions change, depending on how the body parts of each human interact with the other. The proposed method captures the subtle difference between different interactions using interacting body part attention. Semantically important body parts that interact with other objects are given more weight during feature representation. The combined feature of interacting body part attention-based individual representation and the co-occurrence descriptor of the full-body appearance change is fed into long short-term memory to model the temporal dynamics over time in a single framework. The experimental results on five widely used public datasets demonstrate the effectiveness of the proposed method to recognize human interactions from videos.

Skeleton-Based ST-GCN for Human Action Recognition With Extended Skeleton Graph and Partitioning Strategy

Skeleton-based Graph Convolutional Networks (GCN) for human action and interaction recognition have received considerable attention of researchers due to its compact and view-invariant nature of skeleton data. However, the static skeleton graph topology in conventional GCNs does not reflect the implicit relationships of non-adjacent joints, which contain vital latent information for a skeleton pose in an action sequence. Moreover, traditional tri-categorical node partitioning strategy discards much of the motion dependencies along temporal dimension for non-physically connected edges. We propose an extended skeleton graph topology along with extended partitioning strategy to extract much of the non-adjacent joint relational information in the model for robust discriminative features. Extended skeleton graph represents joints as vertices and weighted edges represent intrinsic and extrinsic relationships between physically connected and non-physically connected joints respectively. Furthermore, extended partitioning strategy divides the input graph for GCN as five-categorical fixed-length tensor to encompass maximal motion dependencies. Finally, the extended skeleton graph and partitioning strategy are realized by adopting Spatio-Temporal Graph Convolutional Network (ST-GCN). The experiments carried out over three large scale datasets NTU-RGB+D, NTU-RGB+D 120 and Kinetics-Skeleton show improved performance over conventional state-of-the-art ST-GCNs.

An Intelligent Framework for Recognizing Social Human-Object Interactions

Human object interaction (HOI) recognition plays an important role in the designing of surveillance and monitoring systems for healthcare, sports, education, and public areas. It involves localizing the human and object targets and then identifying the interactions between them. However, it is a challenging task that highly depends on the extraction of robust and distinctive features from the targets and the use of fast and efficient classifiers. Hence, the proposed system offers an automated body-parts-based solution for HOI recognition. This system uses RGB (red, green, blue) images as input and segments the desired parts of the images through a segmentation technique based on the watershed algorithm. Furthermore, a convex hull-based approach for extracting key body parts has also been introduced. After identifying the key body parts, two types of features are extracted. Moreover, the entire feature vector is reduced using a dimensionality reduction technique called t-SNE (t-distributed stochastic neighbor embedding). Finally, a multinomial logistic regression classifier is utilized for identifying class labels. A large publicly available dataset, MPII (Max Planck Institute Informatics) Human Pose, has been used for system evaluation. The results prove the validity of the proposed system as it achieved 87.5% class recognition accuracy.

Performance Evaluation of Entropy Based Cuboid Selection Technique For Human Interaction Recognition

Performance Evaluation of Entropy Based Cuboid Selection Technique For Human Interaction Recognition

Performance evaluation of entropy-based cuboid selection technique for human interaction recognition

Local feature-based approaches are proven to be most successful for the application of human action recognition. This work aims to study the effect of the number of cuboids chosen on the performance of human action recognition. A 3D region, called cuboid, is extracted around every spatio-temporal interest point and an entropy-based cuboid selection method is implemented for choosing the cuboids having maximum information. The proposed method is evaluated using UT interaction and SBU kinect interaction datasets. Results show that maximum classification accuracy and F1 Score are obtained when the top 40 cuboids are selected for the feature extraction. The classification accuracy and F1 Score increase when the number of selected cuboids is increased from 10 to 40 and remain constant when the number is increased beyond 40. The selected 40 cuboids are then used for feature extraction. To carry out this experiment, wavelet coefficients extracted from selected cuboids are used as features.

Human interaction recognition method based on parallel multi-feature fusion network

Human activity recognition is a key technology in intelligent video surveillance and an important research direction in the field of computer vision. However, the complexity of human interaction features and the differences in motion characteristics at different time periods have always existed. In this paper, a human interaction recognition algorithm based on parallel multi-feature fusion network is proposed. First of all, in view of the different amount of information provided by the different time periods of action, an improved time-phased video down sampling method based on Gaussian model is proposed. Second, the Inception module uses different scale convolution kernels for feature extraction. It can improve network performance and reduce the amount of network parameters at the same time. The ResNet module mitigates degradation problem due to increased depth of neural networks and achieves higher classification accuracy. The amount of information provided in the motion video in different stages of motion time is also different. Therefore, we combine the advantages of the Inception network and ResNet to extract feature information, and then we integrate the extracted features. After the extracted features are merged, the training is continued to realize parallel connection of the multi-feature neural network. In this paper, experiments are carried out on the UT dataset. Compared with the traditional activity recognition algorithm, this method can accomplish the recognition tasks of six kinds of interactive actions in a better way, and its accuracy rate reaches 88.9%.

Multi-stream pose convolutional neural networks for human interaction recognition in images

Recognizing human interactions in still images is quite a challenging task since compared to videos, there is only a glimpse of interaction in a single image. This work investigates the role of human poses in recognizing human–human interactions in still images. To this end, a multi-stream convolutional neural network architecture is proposed, which fuses different levels of human pose information to recognize human interactions better. In this context, several pose-based representations are explored. Experimental evaluations in an extended benchmark dataset show that the proposed multi-stream pose Convolutional Neural Network is successful in discriminating a wide range of human–human interactions and human poses when used in conjunction with the overall context provides discriminative cues about human–human interactions.

Dyadic relational graph convolutional networks for skeleton-based human interaction recognition

Skeleton-based human interaction recognition is a challenging task requiring all abilities to recognize spatial, temporal, and interactive features. These abilities rarely co-exist in existing methods. Graph convolutional network (GCN) based methods fail to extract interactive features. Traditional interaction recognition methods cannot effectively capture spatial features from skeletons. Toward this end, we propose a novel Dyadic Relational Graph Convolutional Network (DR-GCN) for interaction recognition. Specifically, we make four contributions: (i) we design a Relational Adjacency Matrix (RAM) that represents dynamic relational graphs. These graphs are constructed combining both geometric features and relative attention from the two skeleton sequences; (ii) we propose a Dyadic Relational Graph Convolution Block (DR-GCB) that extracts spatial-temporal interactive features; (iii) we stack the proposed DR-GCBs to build DR-GCN and integrate our methods with an advanced model. (iv) Our models achieve state-of-the-art results on SBU and significant improvements on the mutual action sub-datasets of NTU-RGB+D and NTU-RGB+D 120.

Hierarchical Long Short-Term Concurrent Memory for Human Interaction Recognition.

In this work, we aim to address the problem of human interaction recognition in videos by exploring the long-term inter-related dynamics among multiple persons. Recently, Long Short-Term Memory (LSTM) has become a popular choice to model individual dynamic for single-person action recognition due to its ability to capture the temporal motion information in a range. However, most existing LSTM-based methods focus only on capturing the dynamics of human interaction by simply combining all dynamics of individuals or modeling them as a whole. Such methods neglect the inter-related dynamics of how human interactions change over time. To this end, we propose a novel Hierarchical Long Short-Term Concurrent Memory (H-LSTCM) to model the long-term inter-related dynamics among a group of persons for recognizing human interactions. Specifically, we first feed each person's static features into a Single-Person LSTM to model the single-person dynamic. Subsequently, at one time step, the outputs of all Single-Person LSTM units are fed into a novel Concurrent LSTM (Co-LSTM) unit, which mainly consists of multiple sub-memory units, a new cell gate, and a new co-memory cell. In the Co-LSTM unit, each sub-memory unit stores individual motion information, while this Co-LSTM unit selectively integrates and stores inter-related motion information between multiple interacting persons from multiple sub-memory units via the cell gate and co-memory cell, respectively. Extensive experiments on several public datasets validate the effectiveness of the proposed H-LSTCM by comparing against baseline and state-of-the-art methods.