Abstract
This paper mainly studies the problem of image-text matching in order to make image and text better match. Existing cross-media retrieval methods only make use of the information of image and part of text, that is, matching the whole image with the whole sentence, or matching some image areas with some words. In order to better reveal the potential connection between image and text semantics, this paper proposes a fusion of two levels of similarity across media images-text retrieval method, constructed the cross-media two-level network to explore the better matching between images and texts, it contains two subnets for dealing with global features and local characteristics. Specifically, in this method, the image is divided into the whole picture and some image area, the text is divided into the whole sentences and words, to study respectively, to explore the full potential alignment of images and text, and then use a two-level alignment framework is used to promote each other, fusion of two kinds of similarity can learn to complete representation of cross-media retrieval. Through the experimental evaluation on Flickr30K and MS-COCO datasets, the results show that the method in this paper can make the semantic matching of image and text more accurate, and is superior to the international popular cross-media retrieval method in various evaluation indexes.
Highlights
Cross-media represent that coexistence of complex media objects such as web text, images, audio, video, and so on. It is manifested in the complex relation and organization structure formed by various media objects
By ‘‘crossmedia’’, the same semantic information can be expressed from the respective sides, so specific content information can be more comprehensively reflected than a single media object and its specific modality
By integrating and analyzing multiple modes, we can understand the content information contained in cross-media retrieval
Summary
Cross-media represent that coexistence of complex media objects such as web text, images, audio, video, and so on. The main idea of Lee et al [19] is to use the attention mechanism on the text and image side to learn better text and image representation, and use hard triplet loss to measure the similarity between text and image in the common subspace This approach uses the attention mechanism to obtain better potential semantic alignment. We were inspired by the ideas proposed in some rankingbased methods, in which loss function based on triplet loss can be used to increase the distance representation of different semantics of different media modes In this way, the result is a better modal matching representation of the image and text
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
