Human-object Interaction Detection Research Articles

Automatically detecting human-object interaction by an instance part-level attention deep framework

Automatically detecting human-object interactions (HOIs) from an image is a very important but challenging task in computer vision. One of the significant problems in HOI detection is that similar human-object interactions are difficult to distinguish. Recently, many instance-centric HOI detection schemes, based on appearance features and coarse spatial information, have been proposed. These methods, however, lack the capacity of capturing and analyzing the fine-grained context between human poses and object parts, which plays a crucial role in HOI detection. To address these problems, we propose a novel instance part-level attention deep framework for HOI detection. Specifically, our approach consists of a human/object-part detection phase and an HOI detection phase. In the former phase, a part-level visual pattern estimation model is designed for capturing the fine-grained human body parts and object parts. In the latter phase, a self-attention-based deep network is proposed to learn the visual composite around the human-object pair that implicitly expresses the consistent spatial, scale, co-occurrence, and viewpoint relationships among human body parts and object parts across images, which are effective for predicting HOI. To the best of our knowledge, we are the first to propose a framework where the fine-grained part-level mutual context of a human-object pair is extracted to improve HOI detection. By comparing our approach with state-of-the-art HOI detection methods on benchmark datasets, we demonstrated that our proposed framework outperformed the existing HOI detection methods, such as significantly improving the performance of part-level visual pattern estimation, HOI detection, and the quality of the self-attention-based deep network structure.

Human–Object Interaction Detection with Ratio-Transformer

Human–object interaction (HOI) is a human-centered object detection task that aims to identify the interactions between persons and objects in an image. Previous end-to-end methods have used the attention mechanism of a transformer to spontaneously identify the associations between persons and objects in an image, which effectively improved detection accuracy; however, a transformer can increase computational demands and slow down detection processes. In addition, the end-to-end method can result in asymmetry between foreground and background information. The foreground data may be significantly less than the background data, while the latter consumes more computational resources without significantly improving detection accuracy. Therefore, we proposed an input-controlled transformer, “ratio-transformer” to solve an HOI task, which could not only limit the amount of information in the input transformer by setting a sampling ratio, but also significantly reduced the computational demands while ensuring detection accuracy. The ratio-transformer consisted of a sampling module and a transformer network. The sampling module divided the input feature map into foreground versus background features. The irrelevant background features were a pooling sampler, which were then fused with the foreground features as input data for the transformer. As a result, the valid data input into the Transformer network remained constant, while irrelevant information was significantly reduced, which maintained the foreground and background information symmetry. The proposed network was able to learn the feature information of the target itself and the association features between persons and objects so it could query to obtain the complete HOI interaction triplet. The experiments on the VCOCO dataset showed that the proposed method reduced the computational demand of the transformer by 57% without any loss of accuracy, as compared to other current HOI methods.

Human–Object Interaction Detection Based on Star Graph

Graph attention networks (GATs)-based method performs well in Human–Object Interaction (HOI) detection due to its ability to aggregate contextual information. However, the traditional GATs-based methods are computationally intensive, and the graph’s structure cannot effectively represent the HOI in an image. Meanwhile, the graph models are unable to predict the interactions that involve less contextual information correctly. In this paper, we design a method based on graph models called V-SGATs, which stands for Visual Branch and Star Graph Attention Networks. The human-centric star graph and object-centric star graph are adopted to reduce computational complexity, and the logical structure of the star graph can represent HOI more reasonably. Meanwhile, the visual branch that recognizes the interaction without utilizing the contextual information is designed to aid the graph model in predicting the interactions that involve less contextual information. Experiments are carried out on two large-scale HOI public benchmarks V-COCO and HICO-DET, and the results show that the proposed method performs better than most of the existing methods based on GATs.

ICGPN: Interaction-centric graph parsing network for human-object interaction detection

Human-Object Interaction (HOI) detection aims to infer different interactions, which occur between humans and related objects of images. HOI is usually represented by a triplet human,action,object and can be modeled as a graph. Thus, with global structural information of images, graph-based methods can detect interactions. However, in existing graph networks, although different fully-connected graphs are built, all detected bounding boxes are regarded as graph nodes equally or different types of nodes according to the category, thereby the dominant role of humans in HOI is ignored. In addition, object node representations mainly focus on appearance features, contributing little to HOI inference. To address these issues, a novel graph-based HOI detection model, named interaction-centric graph parsing network (iCGPN), models one human node as a central node, and other nodes as semantic nodes. Firstly, for each detected human instance, a human-centric fully-connected graph is constructed to learn related HOIs. Secondly, in order to reflect the difference between central nodes and semantic nodes, we design different feature representations and model different edge relationships. Through introducing the attention mechanism, global information related to human-object interaction is explored to enrich the semantic node representation, in which spatial layout, relative locations and object categories information are also combined. Finally, a multi-relation graph convolutional network is applied to update the node feature and infer the HOI. Furthermore, a multi-IOU random shift scheme is proposed to augment the data of the training set to fit the object detection deviation and enhance the generalization ability of our network. Extensive experimental results show that iCGPN achieves very competitive results in comparison with state-of-the-arraph-based methods on the V-COCO and HICO-DET datasets, which demonstrate the effectiveness of the proposed method.

Detecting Human-Object Interactions with Object-Guided Cross-Modal Calibrated Semantics

Human-Object Interaction (HOI) detection is an essential task to understand human-centric images from a fine-grained perspective. Although end-to-end HOI detection models thrive, their paradigm of parallel human/object detection and verb class prediction loses two-stage methods' merit: object-guided hierarchy. The object in one HOI triplet gives direct clues to the verb to be predicted. In this paper, we aim to boost end-to-end models with object-guided statistical priors. Specifically, We propose to utilize a Verb Semantic Model (VSM) and use semantic aggregation to profit from this object-guided hierarchy. Similarity KL (SKL) loss is proposed to optimize VSM to align with the HOI dataset's priors. To overcome the static semantic embedding problem, we propose to generate cross-modality-aware visual and semantic features by Cross-Modal Calibration (CMC). The above modules combined composes Object-guided Cross-modal Calibration Network (OCN). Experiments conducted on two popular HOI detection benchmarks demonstrate the significance of incorporating the statistical prior knowledge and produce state-of-the-art performances. More detailed analysis indicates proposed modules serve as a stronger verb predictor and a more superior method of utilizing prior knowledge. The codes are available at https://github.com/JacobYuan7/OCN-HOI-Benchmark.

Improving Human-Object Interaction Detection via Phrase Learning and Label Composition

Human-Object Interaction (HOI) detection is a fundamental task in high-level human-centric scene understanding. We propose PhraseHOI, containing a HOI branch and a novel phrase branch, to leverage language prior and improve relation expression. Specifically, the phrase branch is supervised by semantic embeddings, whose ground truths are automatically converted from the original HOI annotations without extra human efforts. Meanwhile, a novel label composition method is proposed to deal with the long-tailed problem in HOI, which composites novel phrase labels by semantic neighbors. Further, to optimize the phrase branch, a loss composed of a distilling loss and a balanced triplet loss is proposed. Extensive experiments are conducted to prove the effectiveness of the proposed PhraseHOI, which achieves significant improvement over the baseline and surpasses previous state-of-the-art methods on Full and NonRare on the challenging HICO-DET benchmark.

Highlighting Object Category Immunity for the Generalization of Human-Object Interaction Detection

Human-Object Interaction (HOI) detection plays a core role in activity understanding. As a compositional learning problem (human-verb-object), studying its generalization matters. However, widely-used metric mean average precision (mAP) fails to model the compositional generalization well. Thus, we propose a novel metric, mPD (mean Performance Degradation), as a complementary of mAP to evaluate the performance gap among compositions of different objects and the same verb. Surprisingly, mPD reveals that previous methods usually generalize poorly. With mPD as a cue, we propose Object Category (OC) Immunity to boost HOI generalization. The idea is to prevent model from learning spurious object-verb correlations as a short-cut to over-fit the train set. To achieve OC-immunity, we propose an OC-immune network that decouples the inputs from OC, extracts OC-immune representations, and leverages uncertainty quantification to generalize to unseen objects. In both conventional and zero-shot experiments, our method achieves decent improvements. To fully evaluate the generalization, we design a new and more difficult benchmark, on which we present significant advantage. The code is available at https://github.com/Foruck/OC-Immunity.

Human-object interaction detection with depth-augmented clues

Human object interaction (HOI) detection aims to localize and classify triplets of human, object and relationship from a given image. Different from previous methods that only extract vision information in RGB images, we propose a Depth-augmented Relationship Reasoning (DRR) method that focuses on the RGB images and corresponding depth messages simultaneously. Rethinking principles of photography, we argue that RGB images discard spatial depth carrying third dimension relative distance information between instances. In light of this, we beforehand estimate the depth information for each image, yielding a corresponding depth map. Then we leverage multiple representations encoded by depth information and RGB images to enrich semantic interpretation. Subsequently, we explore a hierarchical attention strategy to fuse these semantic representations and further generate depth-augmented features, being used to reason about fine-grained human-object interactions. Extensive experiments on the benchmark datasets V-COCO, HICO-DET and HCVRD verify the effectiveness of our method and demonstrate the importance of spatial depth information for HOI.

Multi-stream feature refinement network for human object interaction detection

Human–Object Interaction (HOI) detection is a crucial problem for comprehensive visual understanding, which aims to detect <human,action,object> triplets within an image. Many existing methods often exploit to integrate the human and object visual features , the spatial layout of human–object pairs, human poses, contextual information, and even object semantic information into a framework to infer the interactions, proving that all these components can contribute to improve the HOI detection. However, most methods simply concatenate these components that are not explicitly embedded in the feature learning for HOI detection. In this paper, we are trying to fuse these components explicitly using a multi-stream feature refinement network. The network extracts the visual features of humans, contexts, and objects, which receives the attentions from human poses, spatial configurations, and semantic prior knowledge of objects to refine these visual features, respectively. In addition, an additional graph neural network is employed here to learn the structural features of human–object pairs. We verify our method on V-COCO and HICO-DET datasets with extensive experiments. The experimental results demonstrate that our method is a simple yet effective for HOI detection, achieving superior performance to those state-of-the-art methods.

Pose attention and object semantic representation-based human-object interaction detection network

Pose attention and object semantic representation-based human-object interaction detection network

Human-object interaction detection via interactive visual-semantic graph learning

Human-object interaction detection via interactive visual-semantic graph learning

Effective actor-centric human-object interaction detection

While Human-Object Interaction (HOI) Detection has achieved tremendous advances in recent, it still remains challenging due to complex interactions with multiple humans and objects occurring in images, which would inevitably lead to ambiguities. Most existing methods either generate all human-object pair candidates and infer their relationships by cropped local features successively in a two-stage manner, or directly predict interaction points in a one-stage procedure. However, the lack of spatial configurations or reasoning steps of two- or one- stage methods respectively limits their performance in such complex scenes. To avoid this ambiguity, we propose a novel actor-centric framework. The main ideas are that when inferring interactions: 1) the non-local features of the entire image guided by actor position are obtained to model the relationship between the actor and context, and then 2) we use an object branch to generate pixel-wise interaction area prediction, where the interaction area denotes the object central area. Moreover, we also use an actor branch to get interaction prediction of the actor and propose a novel composition strategy based on center-point indexing to generate the final HOI prediction. Thanks to the usage of the non-local features and the partly-coupled property of the human-objects composition strategy, our proposed framework can detect HOI more accurately especially for complex images. Extensive experimental results show that our method achieves the state-of-the-art on the challenging V-COCO and HICO-DET benchmarks and is more robust especially in multiple persons and/or objects scenes.

Image2Triplets: A computer vision-based explicit relationship extraction framework for updating construction activity knowledge graphs

Knowledge graph (KG) is an effective tool for knowledge management, particularly in the architecture, engineering and construction (AEC) industry, where knowledge is fragmented and complicated. However, research on KG updates in the industry is scarce, with most current research focusing on text-based KG updates. Considering the superiority of visual data over textual data in terms of accuracy and timeliness, the potential of computer vision technology for explicit relationship extraction in KG updates is yet to be explored. This paper combines zero-shot human-object interaction detection techniques with general KGs to propose a novel framework called Image2Triplets that can extract explicit visual relationships from images to update the construction activity KG. Comprehensive experiments on the images of architectural decoration processes have been performed to validate the proposed framework. The results and insights will contribute new knowledge and evidence to human-object interaction detection, KG update and construction informatics from the theoretical perspective.

Spatial-Net for Human-Object Interaction Detection

Human-object interaction (HOI) detection is the detection of a human&#x2019;s relationship with an object in still images and videos. The majority of HOI detection methods rely on appearance features as the primary feature for detecting the relationship between humans and objects. Furthermore, the model&#x2019;s performance is affected by the abundance of false-positive pairs generated by the image&#x2019;s non-interactive human-object pairs and human-object mis-grouping. In this paper, we propose "Spatial-Net", a new HOI detection approach in still images. In the proposed approach, the HOI problem is divided into two main tasks, namely pair-prediction and global-rejection. In the pair-prediction task, the spatial relationship is adopted to predict the human-object interaction for each human-object pair using spatial features that contains spatial map which is a single channel image that represents human-object pairs including body parts and object masks, relative geometry features such as relative size, relative distance, and intersection-over-union between body part and objects, and weighted distance that is used as body part attention deterministic model. In the global-rejection task, an augmented model is employed to reject false positive pairs. We use the Hungarian matching technique to assign human-object pairs for each action and human-centric model to reject the non-interaction human-object pairs according to semantic co-occurrence between human and object. The experimental results on the V-COCO dataset demonstrate that the proposed Spatial-Net outperforms many state-of-the-art HOI models with less inference time.

Distance-Aware Occlusion Detection with Focused Attention.

For humans, understanding the relationships between objects using visual signals is intuitive. For artificial intelligence, however, this task remains challenging. Researchers have made significant progress studying semantic relationship detection, such as human-object interaction detection and visual relationship detection. We take the study of visual relationships a step further from semantic to geometric. In specific, we predict relative occlusion and relative distance relationships. However, detecting these relationships from a single image is challenging. Enforcing focused attention to task-specific regions plays a critical role in successfully detecting these relationships. In this work, (1) we propose a novel three-decoder architecture as the infrastructure for focused attention; 2) we use the generalized intersection box prediction task to effectively guide our model to focus on occlusion-specific regions; 3) our model achieves a new state-of-the-art performance on distance-aware relationship detection. Specifically, our model increases the distance F1-score from 33.8% to 38.6% and boosts the occlusion F1-score from 34.4% to 41.2%. Our code and data will be publicly available.

A review of deep learning based human-object interaction detection

A review of deep learning based human-object interaction detection

Human Pose Estimation and Object Interaction for Sports Behaviour

In the new era of technology, daily human activities are becoming more challenging in terms of monitoring complex scenes and backgrounds. To understand the scenes and activities from human life logs, human-object interaction (HOI) is important in terms of visual relationship detection and human pose estimation. Activities understanding and interaction recognition between human and object along with the pose estimation and interaction modeling have been explained. Some existing algorithms and feature extraction procedures are complicated including accurate detection of rare human postures, occluded regions, and unsatisfactory detection of objects, especially small-sized objects. The existing HOI detection techniques are instance-centric (object-based) where interaction is predicted between all the pairs. Such estimation depends on appearance features and spatial information. Therefore, we propose a novel approach to demonstrate that the appearance features alone are not sufficient to predict the HOI. Furthermore, we detect the human body parts by using the Gaussian Matric Model (GMM) followed by object detection using YOLO. We predict the interaction points which directly classify the interaction and pair them with densely predicted HOI vectors by using the interaction algorithm. The interactions are linked with the human and object to predict the actions. The experiments have been performed on two benchmark HOI datasets demonstrating the proposed approach.

Pose graph parsing network for human-object interaction detection

The detection of interactions between humans and objects is one of the core issues in the area of scene understanding in image analysis. The conventional method is to pair the human body with the object as an entity and pay attention to the human spatial area and object. However, this method does not consider two key aspects: humans use certain body parts to interact with objects, and correlations exist between different body parts. Thus, in this paper, we propose a pose graph parsing network (PGPN) for human-object interaction detection. Specifically, we construct a multibranch network to study high-level semantic features. In addition to emphasizing the appearance area of each instance in an image, feature propagation based on a pose graph is further adopted to consider the features of correlation between different body parts. Furthermore, a branch refines and captures the relationship between human parts and an object using a human pose. We validate this approach on the V-COCO and HICO-DET datasets and compare it with the state-of-the-arts. In comparison to other models, the PGPN superior and significantly improves the performance of human-object interaction detection.

Human object interaction detection using two-direction spatial enhancement and exclusive object prior

Human-Object Interaction (HOI) detection aims to detect visual relations between humans and objects in images. One significant problem of HOI detection is that non-interactive human-object pair can be easily mis-grouped and misclassified as an action, especially when the humans are close and performing similar actions in the scene. To address the mis-grouping problem, we propose a spatial enhancement approach to enforce fine-level spatial constraints in two directions between human body parts and object parts. At inference, we propose a human-object regrouping approach for object-exclusive actions by considering the object-exclusive property of the interactive object, where the target object should not be shared by more than one human. By suppressing non-interactive pairs, our approach can decrease the false positives. Experiments on V-COCO and HICO-DET datasets demonstrate our approach is more robust compared to the existing methods under the presence of multiple humans and objects in the scene.

Human–object interaction detection with missing objects

Human–object interaction (HOI) detection is an important vision task that requires the detection of individual object instances and reasoning of their relations. Despite encouraging advancement in recent years, past methods are still limited to relatively simple images where the human and object instances can be detected without difficulties. HOI in the wild should work even when the objects that a person is interacting with are not visible or hard to detect in the image. In this paper, we formulate HOI with missing objects (HOI-MO) as a research problem, and show that it is indeed critical as many such instances can be found even in the commonly-used public HOI detection datasets. We label these to compose new test sets for the proposed method. To our knowledge, we introduce the first method for such challenging HOI detection that incorporates global scene information. The effectiveness and superiority of the proposed method are demonstrated through extensive experiments and comparisons.