Hierarchical Knowledge-Based Graph Embedding Model for Image–Text Matching in IoTs

Abstract

The development of Internet of Things systems (IoTs) and 5G technology has allowed image and text information to be collected and spread at an unprecedentedly high speed. To improve the data processing capabilities of IoTs, the semantic relations between images and text should be extracted efficiently and accurately. Therefore, to reduce the enormous semantic differences between images and text, existing methods introduce consensus knowledge graphs into image–text matching tasks. However, these methods result in noisy edges during the graph construction stage and overlook detailed knowledge extraction, leading to reduced performance in semantic matching. In this article, a two-layer heterogeneous knowledge graph network is proposed to solve the above problems. The proposed model incorporates category knowledge and local knowledge for improved data representation. Specifically, a category-based hierarchical knowledge graph is constructed to learn representations of knowledge concepts through a hierarchical correlation graph embedding (HCGE) module. Then, a globally guided local attention (GLA) module is used to extract fine-grained local knowledge. Finally, the similarity between the input image and text is calculated based on knowledge-fused features to complete the matching process. Extensive experiments show that the proposed model can learn more effective knowledge features to improve the efficacy of image–text matching in.