Semantic Clues Research Articles

A Study of Occluded Person Re-Identification for Shared Feature Fusion with Pose-Guided and Unsupervised Semantic Segmentation

The human body is often occluded by a variety of obstacles in the monitoring system, so occluded person re-identification is still a long-standing challenge. Recent methods based on pose guidance or external semantic clues have improved the representation and related performance of features; there are still problems, such as weak model representation and unreliable semantic clues. To solve the above problems, we proposed a feature extraction network, named shared feature fusion with pose-guided and unsupervised semantic segmentation (SFPUS). This network will extract more discriminative features and reduce the occlusion noise on pedestrian matching. Firstly, the multibranch joint feature extraction module (MFE) is used to extract feature sets containing pose information and high-order semantic information. This module not only provides robust extraction capabilities but can also precisely segment occlusion and the body. Secondly, in order to obtain multiscale discriminant features, the multiscale correlation feature matching fusion module (MCF) is used to match the two feature sets, and the Pose–Semantic Fusion Loss is designed to calculate the similarity of the feature sets between different modes and fuse them into a feature set. Thirdly, to solve the problem of image occlusion, we use unsupervised cascade clustering to better prevent occlusion interference. Finally, performances of the proposed method and various existing methods are compared on the Occluded-Duke, Occluded-ReID, Market-1501 and Duke-MTMC datasets. The accuracy of Rank-1 reached 65.7%, 80.8%, 94.8% and 89.6%, respectively, and the mAP accuracy reached 58.8%, 72.5%, 91.8% and 80.1%. The experiment results demonstrate that our proposed SFPUS holds promising prospects and performs admirably compared with state-of-the-art methods.

Few‐Shot Contrastive Learning‐Based Multi‐Round Dialogue Intent Classification Method

ABSTRACTTraditional text classification models face challenges in handling long texts and understanding topic transitions in dialogue scenarios, leading to suboptimal performance in automatic speech recognition (ASR)‐based multi‐round dialogue intent classification. In this article, we propose a few‐shot contrastive learning‐based multi‐round dialogue intent classification method. First, the ASR texts are partitioned, and role‐based features are extracted using a Transformer encoder. Second, refined sample pairs are forward‐propagated, adversarial samples are generated by perturbing word embedding matrices and contrastive loss is applied to positive sample pairs. Then, positive sample pairs are input into a multi‐round reasoning module to learn semantic clues from the entire scenario through multiple dialogues, obtain reasoning features, input them into a classifier to obtain classification results, and calculate multi‐task loss. Finally, a prototype update module (PUM) is introduced to rectify the biased prototypes by using gated recurrent unit (GRU) to update the prototypes stored in the memory bank and few‐shot learning (FSL) task. Experimental evaluations demonstrate that the proposed method outperforms state‐of‐the‐art methods on two public datasets (DailyDialog and CM) and a private real‐world dataset.

Correlation-aware Cross-modal Attention Network for Fashion Compatibility Modeling in UGC Systems

Empowered by the continuous integration of social multimedia and artificial intelligence, the application scenarios of information retrieval (IR) progressively tend to be diversified and personalized. Currently, User-Generated Content (UGC) systems have great potential to handle the interactions between large-scale users and massive media contents. As an emerging multimedia IR, Fashion Compatibility Modeling (FCM) aims to predict the matching degree of each given outfit and provide complementary item recommendation for user queries. Although existing studies attempt to explore the FCM task from a multimodal perspective with promising progress, they still fail to fully leverage the interactions between multimodal information or ignore the item-item contextual connectivities of intra-outfit. In this paper, a novel fashion compatibility modeling scheme is proposed based on Correlation-aware Cross-modal Attention Network. To better tackle these issues, our work mainly focuses on enhancing comprehensive multimodal representations of fashion items by integrating the cross-modal collaborative contents and uncovering the contextual correlations. Since the multimodal information of fashion items can deliver various semantic clues from multiple aspects, a modality-driven collaborative learning module is presented to explicitly model the interactions of modal consistency and complementarity via a co-attention mechanism. Considering the rich connections among numerous items in each outfit as contextual cues, a correlation-aware information aggregation module is further designed to adaptively capture significant intra-correlations of item-item for characterizing the content-aware outfit representations. Experiments conducted on two real-world fashion datasets demonstrate the superiority of our approach over state-of-the-art methods.

Effective type label-based synergistic representation learning for biomedical event trigger detection

BackgroundDetecting event triggers in biomedical texts, which contain domain knowledge and context-dependent terms, is more challenging than in general-domain texts. Most state-of-the-art models rely mainly on external resources such as linguistic tools and knowledge bases to improve system performance. However, they lack effective mechanisms to obtain semantic clues from label specification and sentence context. Given its success in image classification, label representation learning is a promising approach to enhancing biomedical event trigger detection models by leveraging the rich semantics of pre-defined event type labels.ResultsIn this paper, we propose the Biomedical Label-based Synergistic representation Learning (BioLSL) model, which effectively utilizes event type labels by learning their correlation with trigger words and enriches the representation contextually. The BioLSL model consists of three modules. Firstly, the Domain-specific Joint Encoding module employs a transformer-based, domain-specific pre-trained architecture to jointly encode input sentences and pre-defined event type labels. Secondly, the Label-based Synergistic Representation Learning module learns the semantic relationships between input texts and event type labels, and generates a Label-Trigger Aware Representation (LTAR) and a Label-Context Aware Representation (LCAR) for enhanced semantic representations. Finally, the Trigger Classification module makes structured predictions, where each label is predicted with respect to its neighbours. We conduct experiments on three benchmark BioNLP datasets, namely MLEE, GE09, and GE11, to evaluate our proposed BioLSL model. Results show that BioLSL has achieved state-of-the-art performance, outperforming the baseline models.ConclusionsThe proposed BioLSL model demonstrates good performance for biomedical event trigger detection without using any external resources. This suggests that label representation learning and context-aware enhancement are promising directions for improving the task. The key enhancement is that BioLSL effectively learns to construct semantic linkages between the event mentions and type labels, which provide the latent information of label-trigger and label-context relationships in biomedical texts. Moreover, additional experiments on BioLSL show that it performs exceptionally well with limited training data under the data-scarce scenarios.

Dual-modal non-local context guided multi-stage fusion for indoor RGB-D semantic segmentation

Complementarily fusing RGB and depth images while effectively suppressing task-irrelevant noise is crucial for achieving accurate indoor RGB-D semantic segmentation. In this paper, we propose a novel deep model that leverages dual-modal non-local context to guide the aggregation of complementary features and the suppression of noise at multiple stages. Specifically, we introduce a dual-modal non-local context encoding (DNCE) module to learn global representations for each modality at each stage, which are then utilized to facilitate cross-modal complementary clue aggregation (CCA). Subsequently, the enhanced features from both modalities are merged together. Additionally, we propose a semantic guided feature rectification (SGFR) module to exploit rich semantic clues in the top-level merged features for suppressing noise in the lower-stage merged features. Both the DNCE-CCA and the SGFR modules provide dual-modal global views that are essential for effective RGB-D fusion. Experimental results on two public indoor datasets, NYU Depth V2 and SUN-RGBD, demonstrate that our proposed method outperforms state-of-the-art models of similar complexity.

Multi-task hierarchical convolutional network for visual-semantic cross-modal retrieval

Bridging visual and textual representations plays a central role in delving into multimedia data understanding. The main challenge arises from that images and texts exist in heterogeneous spaces, leading to the difficulty to preserve the semantic consistency between both modalities. To narrow the modality gap, most recent methods resort to extra object detectors or parsers to obtain the hierarchical representations. In this work, we address this problem by introducing our Multi-Task Hierarchical Convolutional Neural Network (MT-HCN). It is characterized by mining the hierarchical semantic information without the aid of any extra supervisions. Firstly, from the perspective of representing architecture, we leverage the intrinsic hierarchical structure of Convolutional Neural Networks (CNNs) to decompose the representations of both modalities into two semantically complementary levels, i.e., exterior representations and concept representations. The former focuses on discovering the fine-grained low-level associations between both modalities, meanwhile the latter underlines capturing more high-level abstract semantics. Specifically, we present a Self-Supervised Clustering (SSC) loss to preserve more fine-grained semantic clues in exterior representations. It is constituted on the basis of viewing multiple image/text pairs with similar exterior as a category. In addition, a novel harmonious bidirectional triplet ranking (HBTR) loss is proposed, which mitigate the adverse effects brought about by the biased and noisy negative samples. Besides hardest negatives, it also imposes the constraints on the distance between the positive pairs and the centroid of negative pairs. Extensive experimental results on two popular cross-modal retrieval benchmarks demonstrate our proposed MT-HCN can achieve the competitive results compared with the state-of-the-art methods.

Exploring varying color spaces through representative forgery learning to improve deepfake detection

In the digital age, the rise of deepfake technology has brought unprecedented challenges to multimedia content authentication. The existing deepfake detection methods generally perform well in known settings. However, generalization and robustness are still challenging tasks. Observing that most conventional methods adopt the RGB color space, we introduce a novel deepfake detection approach by utilizing multiple color spaces to enhance the identification of deepfakes. Overall, our proposed detection framework comprises two primary stages, i.e., representative forgery learning through multi-color space reasoning and the color spaces-based forgery detection network (FDN). The representative forgery learning task is realized in succession through the manipulation cue boosting network (MCBN), color space transformations, and the forgery highlighting network (FHN). MCBN improves the feature representation, alternate color spaces provide distinctive advantages over traditional RGB color space, while FHN plays an auxiliary role, where it not only mines the texture inconsistency but also points out high-level semantic forgery clues, aiding in the robustness ability of FDN to discern subtle alterations in digital imagery accurately. Through rigorous evaluation on the benchmark datasets, including the FaceForensics++, DFDC, and CelebDF datasets, our proposed approach exhibits promising results in identifying forged multimedia content across varying color representations, outperforming the state-of-the-art methods.

Effects of grammatical and semantic clues on verb acquisition in Chinese-speaking children

Effects of grammatical and semantic clues on verb acquisition in Chinese-speaking children

Diffusion Models for Imperceptible and Transferable Adversarial Attack.

Many existing adversarial attacks generate Lp-norm perturbations on image RGB space. Despite some achievements in transferability and attack success rate, the crafted adversarial examples are easily perceived by human eyes. Towards visual imperceptibility, some recent works explore unrestricted attacks without Lp-norm constraints, yet lacking transferability of attacking black-box models. In this work, we propose a novel imperceptible and transferable attack by leveraging both the generative and discriminative power of diffusion models. Specifically, instead of direct manipulation in pixel space, we craft perturbations in the latent space of diffusion models. Combined with well-designed content-preserving structures, we can generate human-insensitive perturbations embedded with semantic clues. For better transferability, we further "deceive" the diffusion model which can be viewed as an implicit recognition surrogate, by distracting its attention away from the target regions. To our knowledge, our proposed method, DiffAttack, is the first that introduces diffusion models into the adversarial attack field. Extensive experiments conducted across diverse model architectures (CNNs, Transformers, and MLPs), datasets (ImageNet, CUB-200, and Standford Cars), and defense mechanisms underscore the superiority of our attack over existing methods such as iterative attacks, GAN-based attacks, and ensemble attacks. Furthermore, we provide a comprehensive discussion on future research avenues in diffusion-based adversarial attacks, aiming to chart a course for this burgeoning field.

Dynamic Spatio-Temporal Graph Reasoning for VideoQA With Self-Supervised Event Recognition.

Video question answering (VideoQA) requires the ability of comprehensively understanding visual contents in videos. Existing VideoQA models mainly focus on scenarios involving a single event with simple object interactions and leave event-centric scenarios involving multiple events with dynamically complex object interactions largely unexplored. These conventional VideoQA models are usually based on features extracted from the global visual signals, making it difficult to capture the object-level and event-level semantics. Although there exists a recent work utilizing a static spatio-temporal graph to explicitly model object interactions in videos, it ignores the dynamic impact of questions for graph construction and fails to exploit the implicit event-level semantic clues in questions. To overcome these limitations, we propose a Self-supervised Dynamic Graph Reasoning (SDGraphR) model for video question answering (VideoQA). Our SDGraphR model learns a question-guided spatio-temporal graph that dynamically encodes intra-frame spatial correlations and inter-frame correspondences between objects in the videos. Furthermore, the proposed SDGraphR model discovers event-level cues from questions to conduct self-supervised learning with an auxiliary event recognition task, which in turn helps to improve its VideoQA performances without using any extra annotations. We carry out extensive experiments to validate the substantial improvements of our proposed SDGraphR model over existing baselines.

Dynamic Interaction Dilation for Interactive Human Parsing

Interactive segmentation pursues generating high-quality pixel-level predictions with a few user-provided clicks, which is gaining attention for its convenience in segmentation data annotation. Users are allowed to iteratively refine the prediction by adding clicks until the result is satisfactory. Existing interactive methods usually transform the clicks into a set of localization maps by Euclidian distance computation or RGB texture extraction to guide the segmentation, which makes the click transformation a core module in interactive segmentation networks. However, when adopted in human images where large poses, occlusions, and bad illuminations are prevailing, prior transformation methods tend to cause uncorrectable overlapping across localization maps, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , one click corresponds to multiple transformed values at the same position in different localization map channels, which are difficult to form a good match among human parts and limit the interaction efficiency. Furthermore, the inappropriately transformed information is hard to be refined with the static transformation manner, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , based on the fixed formulas / RGB textures, which is out of tune with the dynamically refined interaction process. Hence, we design a dynamic transformation scheme for interactive human parsing (IHP) named Dynamic Interaction Dilation Net ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DID-Net</b> ), which serves as an initial attempt to break the limitations of static transformation while capturing long-range dependencies of clicks within each human part. Specifically, we construct a Dynamic Dilation Module ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DD-Module</b> ) to dilate clicks radially in several directions assisted by human body edge detection. The continually refined edges guide to improve the dilation quality in each interaction iteration, thereby better fitting user intention. Furthermore, we propose an Adaptive Interaction Excitation Block ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">AIE-Block</b> ) to exploit potential semantic clues buried in the dilated clicks and emphasize semantic expression for each human part by feature recalibration. Our DID-Net achieves state-of-the-art performance on 3 public human parsing benchmarks.

Relative Linguistic Clues to Fancy Expressions

Objectives: The study aims at a relative description that can be relied upon for the purpose of a consistent and appropriate placement of these terms in their context on the approach of the Arabs in their speech and poetry, and to put them down to the places in which they were originally placed, and to indicate their discriminatory features, which helps to use them accurately in expression.&#x0D; Methods: The research took a descriptive and analytical approach based on tracking the positions of these words in some poetic models in the era of protest, and analyzing them in the light of verbal and moral clues. I came to her.&#x0D; Results: The study revealed the linguistic criteria that predict the emergence of contextual semantic components, and showed that similar utterances in predicates increase their chances of succession in the same context, and the contextual significance is gained through the regularity of utterances in contexts controlled by verbal and semantic clues, and that they dominate the context.&#x0D; Conclusion: The study concluded that a relative linguistic standard for the expressions of passion in the light of the verbal and intangible clues, emphasizing the attention of the Arabs to the meanings and expressions indicated by them through their grammatical positions describing some of their contextual cases, revealing the semantic features they carry, and clarifying the relationships of convergence or divergence between them, which helps to Employing it accurately in the appropriate context, and coming up with a relative linguistic standard for each word, especially since each word has a life story in its consistency with these standards, which had a role in producing the semantic difference displacing it from the rest of its sisters of the words of passion.

Driving Environment Inference from POI of Navigation Map: Fuzzy Logic and Machine Learning Approaches.

To adapt vehicle control and plan strategies in a predictive manner, it is usually desired to know the context of a driving environment. This paper aims at efficiently inferring the following five driving environments around vehicle's vicinity: shopping zone, tourist zone, public station, motor service area, and security zone, whose existences are not necessarily mutually exclusive. To achieve that, we utilize the Point of Interest (POI) data from a navigation map as the semantic clue, and solve the inference task as a multilabel classification problem. Specifically, we first extract all relevant POI objects from a map, then transform these discrete POI objects into numerical POI features. Based on these POI features, we finally predict the occurrence of each driving environment via an inference engine. To calculate representative POI features, a statistical approach is introduced. To composite an inference engine, three inference systems are investigated: fuzzy inference system (FIS), support vector machine (SVM), and multilayer perceptron (MLP). In total, we implement 11 variants of inference engine following two inference strategies: independent and unified inference strategies, and conduct comprehensive evaluation on a manually collected dataset. The result shows that the proposed inference framework generalizes well on different inference systems, where the best overall F1 score 0.8699 is achieved by the MLP-based inference engine following the unified inference strategy, along with the fastest inference time of 0.0002 millisecond per sample. Hence, the generalization ability and efficiency of the proposed inference framework are proved.

Semantic-Context Graph Network for Point-Based 3D Object Detection

Point-based indoor 3D object detection has received increasing attention with the large demand for augmented reality, autonomous driving, and robot technology in the industry. However, the detection precision suffers from inputs with semantic ambiguity, i.e., shape symmetries, occlusion, and texture missing, which would lead that different objects appearing similar from different viewpoints and then confusing the detection model. Typical point-based detectors relieve this problem via learning proposal representations with both geometric and semantic information, while the entangled representation may cause a reduction in both semantic and spatial discrimination. In this paper, we focus on alleviating the confusion from entanglement and then enhancing the proposal representation by considering the proposal’s semantics and the context in one scene. A semantic-context graph network (SCGNet) is proposed, which mainly includes two modules: a category-aware proposal recoding module (CAPR) and a proposal context aggregation module (PCAg). To produce semantically clear features from entanglement representation, the CAPR module learns a high-level semantic embedding for each category to extract discriminative semantic clues. In view of further enhancing the proposal representation and leveraging the semantic clues, the PCAg module builds a graph to mine the most relevant context in the scene. With few bells and whistles, the SCGNet achieves SOTA performance and obtains consistent gains when applying to different backbones (0.9% ~ 2.4% on ScanNet V2 and 1.6% ~ 2.2% on SUN RGB-D for mAP@0.25). Code is available at https://github.com/dsw-jlu-rgzn/SCGNet.

FineStyle: Semantic-Aware Fine-Grained Motion Style Transfer with Dual Interactive-Flow Fusion.

We present FineStyle, a novel framework for motion style transfer that generates expressive human animations with specific styles for virtual reality and vision fields. It incorporates semantic awareness, which improves motion representation and allows for precise and stylish animation generation. Existing methods for motion style transfer have all failed to consider the semantic meaning behind the motion, resulting in limited controls over the generated human animations. To improve, FineStyle introduces a new cross-modality fusion module called Dual Interactive-Flow Fusion (DIFF). As the first attempt, DIFF integrates motion style features and semantic flows, producing semantic-aware style codes for fine-grained motion style transfer. FineStyle uses an innovative two-stage semantic guidance approach that leverages semantic clues to enhance the discriminative power of both semantic and style features. At an early stage, a semantic-guided encoder introduces distinct semantic clues into the style flow. Then, at a fine stage, both flows are further fused interactively, selecting the matched and critical clues from both flows. Extensive experiments demonstrate that FineStyle outperforms state-of-the-art methods in visual quality and controllability. By considering the semantic meaning behind motion style patterns, FineStyle allows for more precise control over motion styles. Source code and model are available on https://github.com/XingliangJin/Fine-Style.git.

PANet: An End-to-end Network Based on Relative Motion for Online Multi-object Tracking

The popular tracking-by-detection paradigm of multi-object tracking (MOT) takes detections of each frame as the input and associates detections from one frame to another. Existing association methods based on the relative motion have attracted attention, because they restrain the effect of noisy detections and improve the performance of MOT. However, these methods depend only on the immediately previous frame, which may easily lead to inaccurate matches and even large accumulated errors. Furthermore, multiple objects involved in occlusions are not fully exploited in these existing methods, which leads to the aggravation of inaccurate matches. Motivated by these issues, we design the pivot to represent each object and propose a novel pivot association network (PANet) for the MOT task. Specifically, pivots are learned from spatial semantic and historical contextual clues, which alleviates the dependency on the immediately previous frame. Our online tracker PANet employs pivots and a lightweight associator to localize tracklets of objects, which can inhibit noise detections and improve the accuracy of tracklet prediction by learning the correlation responses between pivots and spatial search areas. Extensive experiments conducted on two-dimensional MOT15, MOT16, MOT17, and MOT20 demonstrate the effectiveness of the proposed method against numerous state-of-the-art MOT trackers.

High-order paired-ASPP for deep semantic segmentation networks

Current semantic segmentation models only exploit first-order information, while rarely exploring high-order semantics. However, traditional first-order statistics are insufficient to support a solid unanimous representation. In this paper, we propose High-Order Paired-ASPP (Atrous Spatial Pyramid Pooling) Network to exploit high-order statistics from various feature levels. The network first introduces a High-Order Representation module to extract the contextual high-order information from all stages of the backbone. They can provide more semantic clues and discriminative information than the first-order ones. Besides, a Paired-ASPP module is proposed to embed high-order statistics of the early stages into the last stage. It can further preserve the boundary-related and spatial context in the low-level features for final prediction. Our experiments show that the high-order semantics significantly boost the performance on confusing objects. Our method achieves competitive performance without bells and whistles on three benchmarks, i.e., Cityscapes, ADE20K and Pascal-Context.

Intertwined grammaticalization, degrammaticalization, and pragmaticalization: The case of ppwun ‘share’ in Korean

This paper investigates the developmental paths and synchronic functions of one of the Korean defective nouns, i.e., ppwun ‘the only thing’, which originated from a full noun pwun ‘share’. Since the earliest data already show a number of grammaticalized functions, this paper attempts to reconstruct the developmental paths based on the semantic and morphosyntactic clues, making use of the synchrony-for-diachrony approach of grammaticalization. The noun pwun, through coalescence with the adjacent genitive marker -s, grammaticalized into a postpositional particle of focus -spwun ‘only’. When the etymologically bi-morphemic -s.pwun was reanalyzed as mono-morphemic, the dependent particle became reanalyzed again, presumably via analogy of nominal compounding, as a full-fledged, independent noun denoting ‘the only thing’, an instance of degrammaticalization by virtue of an increase of morpho-syntactic independence. Further, some such forms were pragmaticalized by acquiring discourse marker functions, i.e., the speaker’s negative stance toward the state of affairs alerting the addressee that something more serious is about to follow. A series of morphosyntactic and semantic-pragmatic changes shows how the linguistic forms have continuously been subject to morphosyntactic reanalysis and semantic reinterpretation resulting in multiply intertwined scenarios of grammaticalization, degrammaticalization, and pragmaticalization, often with their collocating forms.

Understanding L2-derived words in context: Is complete receptive morphological knowledge necessary?

AbstractThe study investigates whether comprehension of derived words in text context requires a complete understanding of word parts. It explores comprehension of derived words as a function of learner proficiency and contextual clues. Ninety English-as-a-foreign-language learners at three proficiency levels participated in three successive tests representing three clues conditions, absence of clues, availability of syntactic clues, and availability of syntactic and semantic clues. They had to supply the meaning of 22 derived pseudowords constructed with nonword stems and 22 frequent affixes—for example, stacement, gummful. The meanings of the nonword stems were provided. Test scores were compared by 3 (proficiency level) × 3 (clue condition) analysis of variance with repeated measures. The results showed effects of both variables, proficiency and clues. The largest increase in comprehension scores occurred with the addition of syntactic clues. The results imply that derived forms of familiar base words can be understood even when learners’ receptive morphological knowledge is not complete.

Multi-view inter-modality representation with progressive fusion for image-text matching

Recently, image-text matching has been intensively explored to bridge vision and language. Previous methods explore an inter-modality relationship between an image-text pair from the single-view feature. However, it is difficult to discover all the abundant information based on a single inter-modality relationship. In this paper, a novel Multi-View Inter-Modality Representation with Progressive Fusion (MIRPF) is developed to explore inter-modality relationships from multi-view features. The multi-view strategy provides more complementary and global semantic clues than single-view approaches. In particular, the multi-view inter-modality representation network is constructed to generate multiple inter-modality representations, which provide diverse views to discover the latent image-text relationships. Furthermore, the progressive fusion module is performed to fuse inter-modality features stepwise, which fully uses the inherent complementary between different views. Extensive experiments on Flickr30K and MSCOCO verify the superiority of MIRPF compared with several existing approaches. The code is available at: https://github.com/jasscia18/MIRPF.