Hard Negative Samples Research Articles

Efficient Session-based Recommendation with Contrastive Graph-based Shortest Path Search

Session-based recommendation aims to predict the next item based on a set of anonymous sessions. Capturing user intent from a short interaction sequence imposes a variety of challenges since no user profiles are available and interaction data is naturally sparse. Recent approaches relying on graph neural networks (GNNs) for session-based recommendation use global item relations to explore collaborative information from different sessions. These methods capture the topological structure of the graph and rely on multi-hop information aggregation in GNNs to exchange information along edges. Consequently, graph-based models suffer from noisy item relations in the training data and introduce high complexity for large item catalogs. We propose to explicitly model the multi-hop information aggregation mechanism over multiple layers via shortest-path edges based on knowledge from the sequential recommendation domain. Our approach does not require multiple layers to exchange information and ignores unreliable item-item relations. Furthermore, to address inherent data sparsity, we are the first to apply supervised contrastive learning by mining data-driven positive and hard negative item samples from the training data. Extensive experiments on four different datasets show that the proposed approach outperforms almost all of the state-of-the-art methods.

GCLmf: A Novel Molecular Graph Contrastive Learning Framework Based on Hard Negatives and Application in Toxicity Prediction.

In silico methods for prediction of chemical toxicity can decrease the cost and increase the efficiency in the early stage of drug discovery. However, due to low accessibility of sufficient and reliable toxicity data, constructing robust and accurate prediction models is challenging. Contrastive learning, a type of self-supervised learning, leverages large unlabeled data to obtain more expressive molecular representations, which can boost the prediction performance on downstream tasks. While molecular graph contrastive learning has gathered growing attentions, current models neglect the quality of negative data set. Here, we proposed a self-supervised pretraining deep learning framework named GCLmf. We first utilized molecular fragments that meet specific conditions as hard negative samples to boost the quality of the negative set and thus increase the difficulty of the proxy tasks during pre-training to learn informative representations. GCLmf has shown excellent predictive power on various molecular property benchmarks and demonstrates high performance in 33 toxicity tasks in comparison with multiple baselines. In addition, we further investigated the necessity of introducing hard negatives in model building and the impact of the proportion of hard negatives on the model.

Contrastive Speaker Representation Learning with Hard Negative Sampling for Speaker Recognition.

Speaker recognition is a technology that identifies the speaker in an input utterance by extracting speaker-distinguishable features from the speech signal. Speaker recognition is used for system security and authentication; therefore, it is crucial to extract unique features of the speaker to achieve high recognition rates. Representative methods for extracting these features include a classification approach, or utilizing contrastive learning to learn the speaker relationship between representations and then using embeddings extracted from a specific layer of the model. This paper introduces a framework for developing robust speaker recognition models through contrastive learning. This approach aims to minimize the similarity to hard negative samples-those that are genuine negatives, but have extremely similar features to the positives, leading to potential mistaken. Specifically, our proposed method trains the model by estimating hard negative samples within a mini-batch during contrastive learning, and then utilizes a cross-attention mechanism to determine speaker agreement for pairs of utterances. To demonstrate the effectiveness of our proposed method, we compared the performance of a deep learning model trained with a conventional loss function utilized in speaker recognition with that of a deep learning model trained using our proposed method, as measured by the equal error rate (EER), an objective performance metric. Our results indicate that when trained with the voxceleb2 dataset, the proposed method achieved an EER of 0.98% on the voxceleb1-E dataset and 1.84% on the voxceleb1-H dataset.

Denoising and Augmented Negative Sampling for Collaborative Filtering

Negative sampling plays a crucial role in implicit-feedback-based collaborative filtering, where it leverages massive unlabeled data to generate negative signals for guiding supervised learning. The current state-of-the-art approaches focus on utilizing hard negative samples that contain more information to establish a better decision boundary. To strike a balance between efficiency and effectiveness, most existing methods adopt a two-pass approach: in the first pass, a fixed number of unobserved items are sampled using a simple static distribution, while, in the second pass, a more sophisticated negative sampling strategy is employed to select the final negative items. However, selecting negative samples solely from the original items in a dataset is inherently restricted due to the limited available choices, and thus may not be able to effectively contrast positive samples. In this paper, we empirically validate this observation through meticulously designed experiments and identify three major limitations of existing solutions: ambiguous trap, information discrimination, and false negative samples. Our response to such limitations is to introduce “denoised” and “augmented” negative samples that may not exist in the original dataset. This direction renders a few substantial technical challenges. First, constructing augmented negative samples may introduce excessive noise that eventually distorts the decision boundary. Second, the scarcity of supervision signals hampers the denoising process. To this end, we introduce a novel generic denoising and augmented negative sampling (DANS) paradigm and provide a concrete instantiation. First, we disentangle the hard and easy factors of negative items. Then, we regulate the augmentation of easy factors by carefully considering the direction and magnitude. Next, we propose a reverse attention mechanism to learn a user’s negative preference, which allows us to perform a dimension-level denoising procedure on hard factors. Finally, we design an advanced negative sampling strategy to identify the final negative samples, taking into account both the score function used in existing methods and a novel metric called synthesization gain. Through extensive experiments on real-world datasets, we demonstrate that our method substantially outperforms state-of-the-art baselines. Our code is publicly available at https://github.com/Asa9aoTK/ANS-Recbole.

Contrastive learning with hard negative samples for chest X-ray multi-label classification

Contrastive learning has gained significant popularity and achieved remarkable success in learning meaningful representations in various domains. This study addresses the significant problem of dependency on labeled data in chest radiography (CXR) images, which are crucial for diagnosing respiratory diseases such as pneumonia but are both time-consuming and expensive to annotate. Despite extensive research, existing studies on CXR largely depend on labeled data. To overcome this challenge, we propose a framework named SURE (similarity, uncertainty, and representativeness) for multi-label classification with hard negatives in contrastive learning. The proposed framework incorporates these aspects when handling hard negatives and effectively combines contrastive learning and downstream tasks for robust representation learning and multi-label classification. Experimental validation using three distinct CXR datasets demonstrates that our approach significantly reduces the dependency on labeled data while achieving notable performance improvements over existing methods, highlighting its potential effectiveness and efficiency in the CXR domain.

Rehabilitation exercise quality assessment through supervised contrastive learning with hard and soft negatives.

Exercise-based rehabilitation programs have proven to be effective in enhancing the quality of life and reducing mortality and rehospitalization rates. AI-driven virtual rehabilitation, which allows patients to independently complete exercises at home, utilizes AI algorithms to analyze exercise data, providing feedback to patients and updating clinicians on their progress. These programs commonly prescribe a variety of exercise types, leading to a distinct challenge in rehabilitation exercise assessment datasets: while abundant in overall training samples, these datasets often have a limited number of samples for each individual exercise type. This disparity hampers the ability of existing approaches to train generalizable models with such a small sample size per exercise type. Addressing this issue, this paper introduces a novel supervised contrastive learning framework with hard and soft negative samples that effectively utilizes the entire dataset to train a single model applicable to all exercise types. This model, with a Spatial-Temporal Graph Convolutional Network (ST-GCN) architecture, demonstrated enhanced generalizability across exercises and a decrease in overall complexity. Through extensive experiments on three publicly available rehabilitation exercise assessment datasets, UI-PRMD, IRDS, and KIMORE, our method has proven to surpass existing methods, setting a new benchmark in rehabilitation exercise quality assessment.

OR[formula omitted]Net: Online Re-weighting Relation Network for kinship verification

Kinship verification aims to infer whether there is a kin relation between different individuals from facial images. However, popular kinship datasets are often small and suffer from data imbalance. Most existing methods build complex networks to extract features but ignore some implicit information, like family information. They use balanced datasets with fixed negative samples for training, which overlooks valuable information from multiple negative samples, leading to poor performance and robustness. To address these issues, we propose a novel end-to-end framework for kinship verification called Online Re-weighting Relation Network (OR2Net) based on an online re-weighting strategy of meta-learning and relation network. Our novel relation network aims to extract fine-grained features and reduce differences between generations by using multi-scale features and mining family information from kinship datasets. Additionally, we design a lightweight meta re-weighting network that uses a small, clean meta-set to guide the adaptive weighing of training examples. This is done using one-step stochastic gradient descent (SGD) based on an online re-weighting strategy from meta-learning. This helps find effective hard negative samples and reduces the imbalance problem. Extensive experiments on three public kinship verification datasets show that our proposed method is more effective compared to state-of-the-art methods. The code is publicly available on https://github.com/XinZhao-dlnu/OR2N.

SDNet: Spatial adversarial perturbation local descriptor learned with the dynamic probabilistic weighting loss

Local descriptor is an important upstream component in computer vision tasks. Despite considerable advances with deep learning-based descriptors, recent descriptors are not robust enough to handle widespread viewpoint changes in image matching tasks such as localization and 3D reconstruction. In this study, SDNet, a robust descriptor utilizing spatial adversarial perturbations, trained with a novel dynamic probabilistic weighting loss to enhance performance under such challenges. First, to increase the robustness and generalization ability of the network across spatially transformed instances, a innovative module for generating hard negative samples via spatial adversarial perturbations is designed. By maximizing adversarial loss, this module generates more complex patches, significantly enhancing the geometric robustness of the descriptor. Importantly, this module integrates seamlessly with existing patch-based descriptors without necessitating extra training data. Second, to mitigate the imbalance in the matching relationship between generated positive and negative pairs, the label weighted triplet loss is proposed, which markedly improves descriptor performance. Third, a comprehensive theoretical analysis of preceding studies is carried out from a gradient perspective, and a probabilistic dynamic weighting approach that adaptively emphasizes weighting functions with higher likelihoods is proposed to improve training performance of the descriptor. Extensive experiments are carried out on mainstream datasets. These comprehensive experiments demonstrate the effectiveness of SDNet, and the proposed method achieves significant improvements on the UBC, HPatches and ETH datasets, outperforming current state-of-the-art methods. The code is available at https://github.com/webd111/sdnet.

IfCMD: A Novel Method for Radar Target Detection under Complex Clutter Backgrounds

Traditional radar target detectors, which are model-driven, often suffer remarkable performance degradation in complex clutter environments due to the weakness in modeling the unpredictable clutter. Deep learning (DL) methods, which are data-driven, have been introduced into the field of radar target detection (RTD) since their intrinsic non-linear feature extraction ability can enhance the separability between targets and the clutter. However, existing DL-based detectors are unattractive since they require a large amount of independent and identically distributed (i.i.d.) training samples of target tasks and fail to be generalized to the other new tasks. Given this issue, incorporating the strategy of meta-learning, we reformulate the RTD task as a few-shot classification problem and develop the Inter-frame Contrastive Learning-Based Meta Detector (IfCMD) to generalize to the new task efficiently with only a few samples. Moreover, to further separate targets from the clutter, we equip our model with Siamese architecture and introduce the supervised contrastive loss into the proposed model to explore hard negative samples, which have the targets overwhelmed by the clutter in the Doppler domain. Experimental results on simulated data demonstrate competitive detection performance for moving targets and superior generalization ability for new tasks of the proposed method.

Metric learning with adversarial hard negative samples for tag recommendation

Metric learning with adversarial hard negative samples for tag recommendation

CoRTEx: contrastive learning for representing terms via explanations with applications on constructing biomedical knowledge graphs.

Biomedical Knowledge Graphs play a pivotal role in various biomedical research domains. Concurrently, term clustering emerges as a crucial step in constructing these knowledge graphs, aiming to identify synonymous terms. Due to a lack of knowledge, previous contrastive learning models trained with Unified Medical Language System (UMLS) synonyms struggle at clustering difficult terms and do not generalize well beyond UMLS terms. In this work, we leverage the world knowledge from large language models (LLMs) and propose Contrastive Learning for Representing Terms via Explanations (CoRTEx) to enhance term representation and significantly improves term clustering. The model training involves generating explanations for a cleaned subset of UMLS terms using ChatGPT. We employ contrastive learning, considering term and explanation embeddings simultaneously, and progressively introduce hard negative samples. Additionally, a ChatGPT-assisted BIRCH algorithm is designed for efficient clustering of a new ontology. We established a clustering test set and a hard negative test set, where our model consistently achieves the highest F1 score. With CoRTEx embeddings and the modified BIRCH algorithm, we grouped 35 580 932 terms from the Biomedical Informatics Ontology System (BIOS) into 22 104 559 clusters with O(N) queries to ChatGPT. Case studies highlight the model's efficacy in handling challenging samples, aided by information from explanations. By aligning terms to their explanations, CoRTEx demonstrates superior accuracy over benchmark models and robustness beyond its training set, and it is suitable for clustering terms for large-scale biomedical ontologies.

Rumor detection based on Attention Graph Adversarial Dual Contrast Learning.

It is becoming harder to tell rumors from non-rumors as social media becomes a key news source, which invites malicious manipulation that could do harm to the public's health or cause financial loss. When faced with situations when the session structure of comment sections is deliberately disrupted, traditional models do not handle them adequately. In order to do this, we provide a novel rumor detection architecture that combines dual comparison learning, adversarial training, and attention filters. We suggest the attention filter module to achieve the filtering of some dangerous comments as well as the filtering of some useless comments, allowing the nodes to enter the GAT graph neural network with greater structural information. The adversarial training module (ADV) simulates the occurrence of malicious comments through perturbation, giving the comments some defense against malicious comments. It also serves as a hard negative sample to aid double contrast learning (DCL), which aims to learn the differences between various comments, and incorporates the final loss in the form of a loss function to strengthen the model. According to experimental findings, our AGAD (Attention Graph Adversarial Dual Contrast Learning) model outperforms other cutting-edge algorithms on a number of rumor detection tasks. The code is available at https://github.com/icezhangGG/AGAD.git.

Medical Question Summarization with Entity-driven Contrastive Learning

By summarizing longer consumer health questions into shorter and essential ones, medical question-answering systems can more accurately understand consumer intentions and retrieve suitable answers. However, medical question summarization is very challenging due to obvious distinctions in health trouble descriptions from patients and doctors. Although deep learning has been applied to successfully address the medical question summarization (MQS) task, two challenges remain: how to correctly capture question focus to model its semantic intention, and how to obtain reliable datasets to fairly evaluate performance. To address these challenges, this article proposes a novel medical question summarization framework based on e ntity-driven c ontrastive l earning (ECL). ECL employs medical entities present in frequently asked questions (FAQs) as focuses and devises an effective mechanism to generate hard negative samples. This approach compels models to focus on essential information and consequently generate more accurate question summaries. Furthermore, we have discovered that some MQS datasets, such as the iCliniq dataset with a 33% duplicate rate, have significant data leakage issues. To ensure an impartial evaluation of the related methods, this article carefully examines leaked samples to reorganize more reasonable datasets. Extensive experiments demonstrate that our ECL method outperforms the existing methods and achieves new state-of-the-art performance, i.e., 52.85, 43.16, 41.31, 43.52 in terms of ROUGE-1 metric on MeQSum, CHQ-Summ, iCliniq, HealthCareMagic dataset, respectively. The code and datasets are available at https://github.com/yrbobo/MQS-ECL

Mining negative samples on contrastive learning via curricular weighting strategy

Contrastive learning, which pulls positive pairs closer and pushes away negative pairs, has remarkably propelled the development of self-supervised representation learning. Previous studies either neglected negative sample selection, resulting in suboptimal performance, or emphasized hard negative samples from the beginning of training, potentially leading to convergence issues. Drawing inspiration from curriculum learning, we find that learning with negative samples ranging from easy to hard improves both model performance and convergence rate. Therefore, we propose a dynamic negative sample weighting strategy for contrastive learning. Specifically, we design a loss function that adaptively adjusts the weights assigned to negative samples based on the model's performance. Initially, the loss prioritizes easy samples, but as training advances, it shifts focus to hard samples, enabling the model to learn more discriminative representations. Furthermore, to prevent an undue emphasis on false negative samples during later stages, which probably results in trivial solutions, we apply L2 regularization on the weights of hard negative samples. Extensive qualitative and quantitative experiments demonstrate the effectiveness of the proposed weighting strategy. The ablation study confirms both the reasonableness of the curriculum and the effectiveness of the regularization.

SETEM: Self-ensemble training with Pre-trained Language Models for Entity Matching

Entity Matching (EM) aims to determine whether records in two datasets refer to the same real-world entity. Existing work often uses Pre-trained Language Models (PLMs) for feature representation, converting EM to a binary classification task. However, due to the dependence of PLMs on large labeled datasets and the overlap between train and test sets in current EM benchmarks, these methods often underperform in real-world scenarios (e.g., small data size, hard negative samples, and unseen entities). To address the limitations of existing methods, we propose SETEM, a self-ensemble training method leveraging the stability and strong generalization of ensemble models to tackle these challenges in real-world scenarios. Additionally, we develop a faster training method for low-resource applications. Experiments on benchmark datasets show that SETEM outperforms Ditto and HierGAT on the F1 score. In particular, SETEM shows the greatest advantage with small datasets and a high proportion of unseen entities in the test set, achieving up to a 9.61% F1 score increment over baselines on the WDC dataset.

Contrastive Continual Learning with Importance Sampling and Prototype-Instance Relation Distillation

Recently, because of the high-quality representations of contrastive learning methods, rehearsal-based contrastive continual learning has been proposed to explore how to continually learn transferable representation embeddings to avoid the catastrophic forgetting issue in traditional continual settings. Based on this framework, we propose Contrastive Continual Learning via Importance Sampling (CCLIS) to preserve knowledge by recovering previous data distributions with a new strategy for Replay Buffer Selection (RBS), which minimize estimated variance to save hard negative samples for representation learning with high quality. Furthermore, we present the Prototype-instance Relation Distillation (PRD) loss, a technique designed to maintain the relationship between prototypes and sample representations using a self-distillation process. Experiments on standard continual learning benchmarks reveal that our method notably outperforms existing baselines in terms of knowledge preservation and thereby effectively counteracts catastrophic forgetting in online contexts. The code is available at https://github.com/lijy373/CCLIS.

On the Effectiveness of Sampled Softmax Loss for Item Recommendation

The learning objective plays a fundamental role to build a recommender system. Most methods routinely adopt either pointwise (e.g., binary cross-entropy) or pairwise (e.g., BPR) loss to train the model parameters, while rarely pay attention to softmax loss, which assumes the probabilities of all classes sum up to 1, due to its computational complexity when scaling up to large datasets or intractability for streaming data where the complete item space is not always available. The sampled softmax (SSM) loss emerges as an efficient substitute for softmax loss. Its special case, InfoNCE loss, has been widely used in self-supervised learning and exhibited remarkable performance for contrastive learning. Nonetheless, limited recommendation work uses the SSM loss as the learning objective. Worse still, none of them explores its properties thoroughly and answers “Does SSM loss suit for item recommendation?” and “What are the conceptual advantages of SSM loss, as compared with the prevalent losses?”, to the best of our knowledge. In this work, we aim at offering a better understanding of SSM for item recommendation. Specifically, we first theoretically reveal three model-agnostic advantages: (1) mitigating popularity bias, which is beneficial to long-tail recommendation; (2) mining hard negative samples, which offers informative gradients to optimize model parameters; and (3) maximizing the ranking metric, which facilitates top- K performance. However, based on our empirical studies, we recognize that the default choice of cosine similarity function in SSM limits its ability in learning the magnitudes of representation vectors. As such, the combinations of SSM with the models that also fall short in adjusting magnitudes (e.g., matrix factorization) may result in poor representations. One step further, we provide mathematical proof that message passing schemes in graph convolution networks can adjust representation magnitude according to node degree, which naturally compensates for the shortcoming of SSM. Extensive experiments on four benchmark datasets justify our analyses, demonstrating the superiority of SSM for item recommendation. Our implementations are available in both TensorFlow 1 and PyTorch. 2

FFLDGA-Net: Image retrieval method based on Feature Fusion Learnable Descriptor Graph Attention Network

Image retrieval aims to retrieve and return the image in the database that is most similar to the query image. However, the performance of image retrieval models is often hindered by the limited dimensionality of images, which lacks depth information about objects. To address this issue, we propose a novel image retrieval model called FFLDGA-Net (Feature Fusion-based Learnable Descriptor Graph Attention Network). This model aims to overcome the absence of depth information in images by fusing feature information from both image data and point cloud data. First, we introduce the LDGA-Net, which effectively improved the model’s ability to mine hard samples and negative samples. Then, we combine the multi-scale route convolution module with a one-dimensional path aggregation network to fuse point clouds and image features at multiple scales, and establish a high-dimensional relationship between features and low-dimensional features. To mitigate training noise, we incorporate a soft label strategy tailored to the dataset’s characteristics. Our experimental results on two benchmark datasets demonstrate the significant improvements achieved by FFLDGA-Net in image retrieval performance.

Synthetic Hard Negative Samples for Contrastive Learning

Contrastive learning has emerged as an essential approach in self-supervised visual representation learning. Its main goal is to maximize the similarities between augmented versions of the same image (positive pairs), while minimizing the similarities between different images (negative pairs). Recent studies have demonstrated that harder negative samples, i.e., those that are more challenging to differentiate from the anchor sample perform a more crucial function in contrastive learning. However, many existing contrastive learning methods ignore the role of hard negative samples. In order to provide harder negative samples for the network model more efficiently. This paper proposes a novel feature-level sample sampling method, namely sampling synthetic hard negative samples for contrastive learning (SSCL). Specifically, we generate more and harder negative samples by mixing them through linear combination and ensure their reliability by debiasing. Finally, we execute weighted sampling of these negative samples. Compared to state-of-the-art methods, our method can provide more high-quality negative samples. Experiments show that SSCL improves the classification performance on different image datasets and can be readily integrated into existing methods.

Rapid: Zero-shot Domain Adaptation for Code Search with Pre-trained Models

Code search, which refers to the process of identifying the most relevant code snippets for a given natural language query, plays a crucial role in software maintenance. However, current approaches heavily rely on labeled data for training, which results in performance decreases when confronted with cross-domain scenarios including domain-specific or project-specific situations. This decline can be attributed to their limited ability to effectively capture the semantics associated with such scenarios. To tackle the aforementioned problem, we propose a ze R o-shot dom A in ada P tion with pre-tra I ned mo D els framework for code search named RAPID. The framework first generates synthetic data by pseudo labeling, then trains the CodeBERT with sampled synthetic data. To avoid the influence of noisy synthetic data and enhance the model performance, we propose a mixture sampling strategy to obtain hard negative samples during training. Specifically, the mixture sampling strategy considers both relevancy and diversity to select the data that are hard to be distinguished by the models. To validate the effectiveness of our approach in zero-shot settings, we conduct extensive experiments and find that RAPID outperforms the CoCoSoDa and UniXcoder model by an average of 15.7% and 10%, respectively, as measured by the MRR metric. When trained on full data, our approach results in an average improvement of 7.5% under the MRR metric using CodeBERT. We observe that as the model’s performance in zero-shot tasks improves, the impact of hard negatives diminishes. Our observation also indicates that fine-tuning CodeT5 for generating pseudo labels can enhance the performance of the code search model, and using only 100-shot samples can yield comparable results to the supervised baseline. Furthermore, we evaluate the effectiveness of RAPID in real-world code search tasks in three GitHub projects through both human and automated assessments. Our findings reveal RAPID exhibits superior performance, e.g., an average improvement of 18% under the MRR metric over the top-performing model.