Pairwise Ranking Loss Research Articles

Image–Text Cross-Modal Retrieval with Instance Contrastive Embedding

Image–text cross-modal retrieval aims to bridge the semantic gap between different modalities, allowing for the search of images based on textual descriptions or vice versa. Existing efforts in this field concentrate on coarse-grained feature representation and then utilize pairwise ranking loss to pull image–text positive pairs closer, pushing negative ones apart. However, using pairwise ranking loss directly on coarse-grained representation lacks reliability as it disregards fine-grained information, posing a challenge in narrowing the semantic gap between image and text. To this end, we propose an Instance Contrastive Embedding (IConE) method for image–text cross-modal retrieval. Specifically, we first transfer the multi-modal pre-training model to the cross-modal retrieval task to leverage the interactive information between image and text, thereby enhancing the model’s representational capabilities. Then, to comprehensively consider the feature distribution of intra- and inter-modality, we design a novel two-stage training strategy that combines instance loss and contrastive loss, dedicated to extracting fine-grained representation within instances and bridging the semantic gap between modalities. Extensive experiments on two public benchmark datasets, Flickr30k and MS-COCO, demonstrate that our IConE outperforms several state-of-the-art (SoTA) baseline methods and achieves competitive performance.

Using Multimodal Contrastive Knowledge Distillation for Video-Text Retrieval

Cross-modal retrieval aims to enable a flexible bi-directional retrieval experience across different modalities ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g</i> ., searching for videos with texts). Many existing efforts tend to learn a common semantic representation embedding space in which items of different modalities can be directly compared, wherein the positive global representations of video-text pairs are pulled close while the negative ones are pushed apart via pair-wise ranking loss. However, such a vanilla loss would unfortunately yield ambiguous feature embeddings for texts of different videos, causing inaccurate cross-modal matching and unreliable retrievals. Toward this end, we propose a multimodal contrastive knowledge distillation method for instance video-text retrieval, called MCKD, by adaptively using the general knowledge of self-supervised model (teacher) to calibrate mixed boundaries. Specifically, the teacher model is tailored for robust (less-ambiguous) visual-text joint semantic space by maximizing mutual information of co-occurred modalities during multimodal contrastive learning. This robust and structural inter-instance knowledge is then distilled, with the help of explicit discrimination loss, to a student model for improved matching performance. Extensive experiments on four public benchmark video-text datasets (MSR-VTT, TGIF, VATEX, and Youtube2Text) demonstrate that our MCKD can achieve at most 8.8%, 6.4%, 5.9%, and 5.3% improvement in text-to-video performance by the R@1 metric, compared with 14 SoTA baselines.

PCCT: Progressive Class-Center Triplet Loss for Imbalanced Medical Image Classification.

Imbalanced training data in medical image diagnosis is a significant challenge for diagnosing rare diseases. For this purpose, we propose a novel two-stage Progressive Class-Center Triplet (PCCT) framework to overcome the class imbalance issue. In the first stage, PCCT designs a class-balanced triplet loss to coarsely separate distributions of different classes. Triplets are sampled equally for each class at each training iteration, which alleviates the imbalanced data issue and lays solid foundation for the successive stage. In the second stage, PCCT further designs a class-center involved triplet strategy to enable a more compact distribution for each class. The positive and negative samples in each triplet are replaced by their corresponding class centers, which prompts compact class representations and benefits training stability. The idea of class-center involved loss can be extended to the pair-wise ranking loss and the quadruplet loss, which demonstrates the generalization of the proposed framework. Extensive experiments support that the PCCT framework works effectively for medical image classification with imbalanced training images. On four challenging class-imbalanced datasets (two skin datasets Skin7 and Skin 198, one chest X-ray dataset ChestXray-COVID, and one eye dataset Kaggle EyePACs), the proposed approach respectively obtains the mean F1 score 86.20, 65.20, 91.32, and 87.18 over all classes and 81.40, 63.87, 82.62, and 79.09 for rare classes, achieving state-of-the-art performance and outperforming the widely used methods for the class imbalance issue.

Sequential Recommendation with Multiple Contrast Signals

Sequential recommendation has become a trending research topic for its capability to capture dynamic user intents based on historical interaction sequence. To train a sequential recommendation model, it is a common practice to optimize the next-item recommendation task with a pairwise ranking loss. In this paper, we revisit this typical training method from the perspective of contrastive learning and find it can be taken as a specialized contrastive learning task conceptually and mathematically, named context-target contrast . Further, to leverage other self-supervised signals in user interaction sequences, we propose another contrastive learning task to encourage sequences after augmentation, as well as sequences with the same target item, to have similar representations, called context-context contrast . A general framework, ContraRec, is designed to unify the two kinds of contrast signals, leading to a holistic joint-learning framework for sequential recommendation with different contrastive learning tasks. Besides, various sequential recommendation methods (e.g., GRU4Rec, Caser, and BERT4Rec) can be easily integrated as the base sequence encoder in our ContraRec framework. Extensive experiments on three public datasets demonstrate that ContraRec achieves superior performance compared to state-of-the-art sequential recommendation methods.

Confidence-Aware Active Feedback for Interactive Instance Search

Online relevance feedback (RF) is widely utilized in instance search (INS) tasks to further refine imperfect ranking results, but it often has low interaction efficiency. The active learning (AL) technique addresses this problem by selecting valuable feedback candidates. However, mainstream AL methods require an initial labeled set for a cold start and are often computationally complex to solve. Therefore, they cannot fully satisfy the requirements for online RF in interactive INS tasks. To address this issue, we propose a confidence-aware active feedback method (CAAF) that is specifically designed for online RF in interactive INS tasks. Inspired by the explicit difficulty modeling scheme in self-paced learning, CAAF utilizes a pairwise manifold ranking loss to evaluate the ranking confidence of each unlabeled sample. The ranking confidence improves not only the interaction efficiency by indicating valuable feedback candidates but also the ranking quality by modulating the diffusion weights in manifold ranking. In addition, we design two acceleration strategies, an approximate optimization scheme and a top-K search scheme, to reduce the computational complexity of CAAF. Extensive experiments on both image INS tasks and video INS tasks searching for buildings, landscapes, persons, and human behaviors demonstrate the effectiveness of the proposed method. Notably, in the real-world, large-scale video INS task of NIST TRECVID 2021, CAAF uses 25% fewer feedback samples to achieve a performance that is nearly equivalent to the champion solution. Moreover, with the same number of feedback samples, CAAF's mAP is 51.9%, significantly surpassing the champion solution by 5.9%. Code is available at https://github.com/nercms-mmap/caaf.

Successively focused details: attention-based recurrent multiscale network for sketch-based image retrieval

Sketch-based image retrieval (SBIR) is a cross-domain retrieval problem whose main challenge is that sketches and images differ in many aspects, such as visual content and abstract degree. Existing approaches solve cross-domain gaps using deep neural networks (DNNs) to learn a joint global feature representation for sketches and images. To improve the retrieval effect using local features, some researchers have introduced attention mechanisms and multi-scale inputs. However, these cannot effectively locate regions of detail difference between sketch–image pairs, and the discriminative ability of these models is insufficient. Just as human observational habits range from global to local, this paper constructs a multiscale network with the help of attention modules. The network is stacked in a recurrent way and focuses stepwise on the most discriminative regions from coarse to fine through region-specific attention models. The part-aware attention module is specifically designed for each scale network to make it robust against local spatial misalignment. To address the challenges of SBIR, we extend the network into a three-channel structure that integrates the sketch, edge map, and RGB photo branches to acquire a more discriminative cross-domain feature representation. The proposed network is alternately optimized by weighted regularization triplet loss, classification loss, and pairwise ranking loss. We conduct extensive experiments on several commonly used SBIR datasets, quantitative, and qualitative results demonstrate that our network learns more fine-grained features and outperforms all baselines.

Large Scale Ranking Using Stochastic Gradient Descent

A system of linear equations can represent any ranking problem that minimizes a pairwise ranking loss. We utilize a fast version of gradient descent algorithm with a near-optimal learning rate and momentum factor to solve this linear equations system iteratively. Tikhonov regularization is also integrated into this framework to avoid overfitting problems where we have very large and high dimensional but sparse data.

A Deep Ranking Weighted Multihashing Recommender System for Item Recommendation.

Collaborative filtering (CF) techniques are used in recommender systems to provide users with specialised recommendations on social websites and in e-commerce. But they suffer from sparsity and cold start problems (CSP) and fail to interpret why they recommend a new item. A novel deep ranking weighted multihash recommender (DRWMR) system is designed to suppress sparsity and CSP. The proposed DRWMR system contains two stages: the neighbours' formation and recommendation phases. Initially, the data is fed to the deep convolutional neural network (CNN). The significant features are extracted from CNN. The CNN contains an additional layer; the hash code is generated by minimising pairwise ranking loss and classification loss. Therefore, a weight is assigned to different hash tables and hash bits for a recommendation. Then, the similarity between users is obtained based on the weighted hammering distance; the similarity between users helps to form the neighbourhood for the active user. Finally, the rating for unknown items can be obtained by taking the weighted average rating of the neighbourhood, and a list of the top n items can be produced. The effectiveness and accuracy of the proposed DRWMR system are tested on the MovieLens 100 K dataset and compared with the existing methods. Based on the evaluation results, the proposed DRWMR system gives precision (0.16), the root mean squared error (RMSE) of 0.73 and the recall (0.08), the mean absolute error (MAE) of 0.57, and the F − 1 measure (0.101).

Pedestrian Flow Prediction and Route Recommendation with Business Events.

Due to the potential economic benefits, pedestrian flow is considered an essential indication of public spaces. Pedestrian flow prediction is designed to assist operators in making decisions (such as shopping center owners). Operators hold certain events, such as sales promotions, to attract surrounding pedestrians; we refer to this type of event as a business event. Business events attract pedestrian flows, which means business opportunities for the merchants. Moreover, their placement will affect the distributions of the pedestrian flows. However, deciding which route is chosen for a specified event is difficult. To the best of our knowledge, we are the first to consider business events when predicting pedestrian flow. In this paper, we investigate two problems: one is pedestrian flow prediction with business events, and the other is route recommendation for business events. First, we propose an Attraction-Based Matrix Factorization model (ABMF) to efficiently predict the pedestrian flow with business events, which introduces the attraction index of different categories to pedestrians in matrix factorization. Second, we leverage the Skip-gram mode to learn the latent representations and improve the pair-wise ranking loss to a flow-aware-based method (SG-FWARP), which aims to learn events’ latent representations for route recommendation. Compared with other state-of-the-art methods, the experimental results show ABMF can predict pedestrian flow matrix with a similarity of over compared with the ground truth, and SG-FWARP can recommend routes for business events with high accuracy.

Temporal Information-Guided Generative Adversarial Networks for Stimuli Image Reconstruction From Human Brain Activities

Understanding how the human brain works has attracted increasing attention in both fields of neuroscience and machine learning. Previous studies use autoencoder and generative adversarial networks (GANs) to improve the quality of stimuli image reconstruction from functional magnetic resonance imaging (fMRI) data. However, these methods mainly focus on acquiring relevant features between two different modalities of data, i.e., stimuli images and fMRI, while ignoring the temporal information of fMRI data, thus leading to suboptimal performance. To address this issue, in this article, we propose a temporal information-guided GAN (TIGAN) to reconstruct visual stimuli from human brain activities. Specifically, the proposed method consists of three key components, including: 1) an image encoder for mapping the stimuli images into latent space; 2) a long short-term memory (LSTM) generator for fMRI feature mapping, which is used to capture temporal information in fMRI data; and 3) a discriminator for image reconstruction, which is used to make the reconstructed image more similar to the original image. In addition, to better measure the relationship of two different modalities of data (i.e., fMRI and natural images), we leverage a pairwise ranking loss to rank the stimuli images and fMRI to ensure strongly associated pairs at the top and weakly related ones at the bottom. The experimental results on real-world data sets suggest that the proposed TIGAN achieves better performance in comparison with several state-of-the-art image reconstruction approaches.

Learning term weights by overfitting pairwise ranking loss

A search engine strikes a balance between effectiveness and efficiency to retrieve the best documents in a scalable way. Recent deep learning-based ranker methods are proving to be effective and improving the state-of-the-art in relevancy metrics. However, as opposed to index-based retrieval methods, neural rankers like bidirectional encoder representations from transformers (BERT) do not scale to large datasets. In this article, we propose a query term weighting method that can be used with a standard inverted index without modifying it. Query term weights are learned using relevant and irrelevant document pairs for each query, using a pairwise ranking loss. The learned weights prove to be more effective than term recall which is a probabilistic relevance feedback, previously used for the task. We further show that these weights can be predicted with a BERT regression model and improve the performance of both a BM25 based index and an index already optimized with a term weighting function.

Disentangled Graph Recurrent Network for Document Ranking

BERT-based ranking models are emerging for its superior natural language understanding ability. All word relations and representations in the concatenation of query and document are modeled in the self-attention matrix as latent knowledge. However, some latent knowledge has none or negative effect on the relevance prediction between query and document. We model the observable and unobservable confounding factors in a causal graph and perform do-query to predict the relevance label given an intervention over this graph. For the observed factors, we block the back door path by an adaptive masking method through the transformer layer and refine word representations over this disentangled word graph through the refinement layer. For the unobserved factors, we resolve the do-operation query from the front door path by decomposing word representations into query related and unrelated parts through the decomposition layer. Pairwise ranking loss is mainly used for the ad hoc document ranking task, triangle distance loss is introduced to both the transformer and refinement layers for more discriminative representations, and mutual information constraints are put on the decomposition layer. Experimental results on public benchmark datasets TREC Robust04 and WebTrack2009-12 show that DGRe outperforms state-of-the-art baselines more than 2% especially for short queries.

A framework for unbiased explainable pairwise ranking for recommendation

<h2>Abstract</h2> Recent research in recommender systems has demonstrated the advantages of pairwise ranking in recommendation. In this work, we focus on the state-of-the-art pairwise ranking loss function, Bayesian Personalized Ranking (BPR), and aim to address two of its limitations, namely: (1) the lack of explainability and (2) exposure bias. We propose a recommendation framework that encompasses various loss functions that are based on BPR and which aim to mitigate the aforementioned limitations. Our open-source framework includes code to train and tune state-of-the-art pairwise ranking recommender systems on benchmark datasets and evaluate them based on the three criteria of ranking accuracy, explainability, and popularity debiasing.

Learning Hierarchical Video Graph Networks for One-Stop Video Delivery

The explosive growth of video data has brought great challenges to video retrieval, which aims to find out related videos from a video collection. Most users are usually not interested in all the content of retrieved videos but have a more fine-grained need. In the meantime, most existing methods can only return a ranked list of retrieved videos lacking a proper way to present the video content. In this paper, we introduce a distinctively new task, namely One-Stop Video Delivery (OSVD) aiming to realize a comprehensive retrieval system with the following merits: it not only retrieves the relevant videos but also filters out irrelevant information and presents compact video content to users, given a natural language query and video collection. To solve this task, we propose an end-to-end Hierarchical Video Graph Reasoning framework (HVGR) , which considers relations of different video levels and jointly accomplishes the one-stop delivery task. Specifically, we decompose the video into three levels, namely the video-level, moment-level, and the clip-level in a coarse-to-fine manner, and apply Graph Neural Networks (GNNs) on the hierarchical graph to model the relations. Furthermore, a pairwise ranking loss named Progressively Refined Loss is proposed based on prior knowledge that there is a relative order of the similarity of query-video, query-moment, and query-clip due to the different granularity of matched information. Extensive experimental results on benchmark datasets demonstrate that the proposed method achieves superior performance compared with baseline methods.

Regression Guided by Relative Ranking Using Convolutional Neural Network (R3 CNN) for Facial Beauty Prediction

Facial beauty prediction (FBP) aims to automatically assess facial attractiveness consistently with judgements based on human perception. Most of previous methods formulate FBP as a classification, regression or ranking problem of machine learning. However, humans not only represent facial attractiveness as a score, but also perceive the relative aesthetics of faces. Inspired by this observation, we formulate FBP as a specific regression problem guided by ranking information. Specifically, we propose a general CNN architecture, called R <inline-formula><tex-math notation="LaTeX">$^3$</tex-math></inline-formula> CNN, to integrate the relative ranking of faces in terms of aesthetics to improve performance of FBP. As R <inline-formula><tex-math notation="LaTeX">$^3$</tex-math></inline-formula> CNN consists of both regression and ranking components, it is challenging to train and fine-tune it by existing techniques. To tackle this problem, we propose the following learning schemes for R <inline-formula><tex-math notation="LaTeX">$^3$</tex-math></inline-formula> CNN: 1) a hard pair sampling strategy that generates challenging-to-predicted image pairs and pseudo ranking labels from true rating scores; 2) an assemble loss function that combines regression loss and pairwise ranking loss (PR-Loss), where PR-Loss can be a hinge-form loss or a log-sum-exp pairwise loss; 3) a cascaded fine-tuning method that further improves prediction. Moreover, we build a benchmark dataset, called SCUT-FBP5500, containing 5,500 facial images with diverse properties (male/female, Asian/Caucasian, ages) and labels (face landmarks, rating scores within [1, 5], rating score distribution). Experiments were performed on both the SCUT-FBP and the SCUT-FBP5500 benchmark datasets, where our method achieves state-of-the-art performance on different evaluation settings. Comparisons with related CNN models highlight the effectiveness of the R <inline-formula><tex-math notation="LaTeX">$^3$</tex-math></inline-formula> CNN architecture for FBP.

Learning over No-Preferred and Preferred Sequence of Items for Robust Recommendation

In this paper, we propose a theoretically supported sequential strategy for training a large-scale Recommender System (RS) over implicit feedback, mainly in the form of clicks. The proposed approach consists in minimizing pairwise ranking loss over blocks of consecutive items constituted by a sequence of non-clicked items followed by a clicked one for each user. We present two variants of this strategy where model parameters are updated using either the momentum method or a gradient-based approach. To prevent updating the parameters for an abnormally high number of clicks over some targeted items (mainly due to bots), we introduce an upper and a lower threshold on the number of updates for each user. These thresholds are estimated over the distribution of the number of blocks in the training set. They affect the decision of RS by shifting the distribution of items that are shown to the users. Furthermore, we provide a convergence analysis of both algorithms and demonstrate their practical efficiency over six large-scale collections with respect to various ranking measures and computational time.

User preference and embedding learning with implicit feedback for recommender systems

In this paper, we propose a novel ranking framework for collaborative filtering with the overall aim of learning user preferences over items by minimizing a pairwise ranking loss. We show the minimization problem involves dependent random variables and provide a theoretical analysis by proving the consistency of the empirical risk minimization in the worst case where all users choose a minimal number of positive and negative items. We further derive a Neural-Network model that jointly learns a new representation of users and items in an embedded space as well as the preference relation of users over the pairs of items. The learning objective is based on three scenarios of ranking losses that control the ability of the model to maintain the ordering over the items induced from the users' preferences, as well as, the capacity of the dot-product defined in the learned embedded space to produce the ordering. The proposed model is by nature suitable for implicit feedback and involves the estimation of only very few parameters. Through extensive experiments on several real-world benchmarks on implicit data, we show the interest of learning the preference and the embedding simultaneously when compared to learning those separately. We also demonstrate that our approach is very competitive with the best state-of-the-art collaborative filtering techniques proposed for implicit feedback.

Neural Collaborative Preference Learning With Pairwise Comparisons

Collaborative Ranking (CR), as an effective recommendation framework, has attracted increasing attention in recent years. Most CR methods simply adopt the inner product between user/item embeddings as the rating score function, with an assumption that the interacted items are preferred to non-interacted ones. However, such fixed score functions and assumption might not be sufficient to capture the real preference ranking list from the complicated interactions in real-world data. To alleviate this issue, we develop a novel collaborative ranking framework that learns an arbitrary utility function for item ranking with user preference concerned. In the core of our framework, a neural network is employed to model the utility function for personalized ranking with the strength of its nonlinearity. On top of this, we further adopt a pairwise ranking loss for user-item pairs to preserve the preference order of items for users. Besides, such utility function enables us to generate the final top-K preference list in a much easier way. Finally, extensive experiments on four real-world datasets show the validity of our proposed method.

Water level prediction from social media images with a multi-task ranking approach

Floods are among the most frequent and catastrophic natural disasters and affect millions of people worldwide. It is important to create accurate flood maps to plan (offline) and conduct (real-time) flood mitigation and flood rescue operations. Arguably, images collected from social media can provide useful information for that task, which would otherwise be unavailable. We introduce a computer vision system that estimates water depth from social media images taken during flooding events, in order to build flood maps in (near) real-time. We propose a multi-task (deep) learning approach, where a model is trained using both a regression and a pairwise ranking loss. Our approach is motivated by the observation that a main bottleneck for image-based flood level estimation is training data: it is difficult and requires a lot of effort to annotate uncontrolled images with the correct water depth. We demonstrate how to efficiently learn a predictor from a small set of annotated water levels and a larger set of weaker annotations that only indicate in which of two images the water level is higher, and are much easier to obtain. Moreover, we provide a new dataset, named DeepFlood, with 8145 annotated ground-level images, and show that the proposed multi-task approach can predict the water level from a single, crowd-sourced image with ≈11 cm root mean square error.

Learning with privileged information for photo aesthetic assessment

Privileged information (PI), known as teacher providing students helpful comments, comparisons, and explanations to improve students’ performance, has been widely applied in various machine learning tasks, resulting in great success. Existing approaches utilizing attributes either fail to leveraging the attributes information thoroughly, or suffer from the complex network structures for automatically attributes learning. Therefore, we propose a new Deep Convolutional Neural Network with Privileged Information (PI-DCNN) for photo aesthetic assessment by utilizing the prior knowledge of photo and photographic elements as privileged information. This paper is the first to systematically summarize all the attributes (i.e., photo and photographic attributes) related to aesthetics assessment. Specifically, we first explore the privileged information of photo and photography attributes, which is available at the training stage but it is not available for the test set. After that, we transfer the probabilistic dependency relations as constraints, and formulate photo aesthetics assessment in a deep convolutional neural network. Lastly, we propose a new pair-wise ranking loss that can exploit the relationship of photo aesthetic quality within a pair of photos. Experimental results on two widely benchmark databases of aesthetic assessment, AADB and AVA, demonstrate the effectiveness of the proposed PI-DCNN method on photo aesthetic assessment task.