Abstract
A new approach that drastically improves cross-modal retrieval performance in vision and language (hereinafter referred to as “vision and language retrieval”) is proposed in this paper. Vision and language retrieval takes data of one modality as a query to retrieve relevant data of another modality, and it enables flexible retrieval across different modalities. Most of the existing methods learn optimal embeddings of visual and lingual information to a single common representation space. However, we argue that the forced embedding optimization results in loss of key information for sentences and images. In this paper, we propose an effective utilization of representation spaces in a simple but robust vision and language retrieval method. The proposed method makes use of multiple individual representation spaces through text-to-image and image-to-text models. Experimental results showed that the proposed approach enhances the performance of existing methods that embed visual and lingual information to a single common representation space.
Highlights
Single-modal retrieval such as document retrieval from keyword queries [1] and image retrieval from an image query [2] has been traditionally conducted
In each method, we can see that the proposed approach drastically enhances the mean and median ranks compared to the state-of-the-art methods. This means that the proposed approach is effective for various conventional embedding methods to improve the retrieval performance of vision and language retrieval
The best median rank is obtained around the settings α = 0.3, β = 0.5 and γ = 0.2. These results mean that the similarity sEn is the most important information; the similarities sVn and sLn are important information, and we can enhance the performance for vision and language retrieval by using spaces
Summary
Single-modal retrieval such as document retrieval from keyword queries [1] and image retrieval from an image query [2] has been traditionally conducted. In a text-to-image retrieval scenario, a lingual feature from a query in space L and visual features from candidate images in space V are projected into a learned common representation space E that can compare the two different modalities. This embedding approach is currently one of the most popular approaches. We enhance the retrieval performance of conventional embedding methods that only utilize space E utilizing the text-to-image and image-to-text models.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
