Negative Sampling Method Research Articles

Does Negative Sampling Matter? a Review With Insights Into its Theory and Applications.

Negative sampling has swiftly risen to prominence as a focal point of research, with wide-ranging applications spanning machine learning, computer vision, natural language processing, data mining, and recommender systems. This surge in interest prompts us to question the fundamental impact of negative sampling: Does negative sampling really matter? Is there a general framework that can incorporate all negative sampling methods? In what fields is it applied? Addressing these questions, we propose a general framework that using negative sampling. Delving into the history of negative sampling, we chart its evolution across five distinct trajectories. We dissect and categorize the strategies used to select negative sample candidates, detailing global, local, mini-batch, hop, and memory-based approaches. Our comprehensive review extends to an analysis of current negative sampling methodologies, systematically grouping them into five classifications: static, hard, GAN-based, Auxiliary-based, and In-batch. Beyond detailed categorization, we explore the practical application of negative sampling across various fields. Finally, we briefly discuss open problems and future directions for negative sampling.

Event representation via contrastive learning with prototype based hard negative sampling

Event representing methods usually use external knowledge to construct potentially more useful positive or negative samples, while neglecting the importance of sampling them during the training process. In this paper, we propose a prototype based negative sampling method to sample useful negatives and better learn the semantic classes. This sampling method has theoretical generalization guarantees and handles false negatives in a natural way. Furthermore, we apply a parametric augmentation strategy to generate positive pairs, which can alleviate BERT embedding bias and decouple the interrelation of augmented positive pairs. Experimental results show that our unsupervised approach has a 0.8%–5.1% improvement over state-of-the-art methods on hard similarity dataset, and a 2.62% improvement on the MCNC dataset. Our proposed method provides a novel insight into negative sampling and augmentation for event representation.

An efficient hybrid deep learning architecture for predicting short antimicrobial peptides.

Short-length antimicrobial peptides (AMPs) have been demonstrated to have intensified antimicrobial activities against a wide spectrum of microbes. Therefore, exploration of novel and promising short AMPs is highly essential in developing various types of antimicrobial drugs or treatments. In addition to experimental approaches, computational methods have been developed to improve screening efficiency. Although existing computational methods have achieved satisfactory performance, there is still much room for model improvement. In this study, we proposed iAMP-DL, an efficient hybrid deep learning architecture, for predicting short AMPs. The model was constructed using two well-known deep learning architectures: the long short-term memory architecture and convolutional neural networks. To fairly assess the performance of the model, we compared our model with existing state-of-the-art methods using the same independent test set. Our comparative analysis shows that iAMP-DL outperformed other methods. Furthermore, to assess the robustness and stability of our model, the experiments were repeated 10 times to observe the variation in prediction efficiency. The results demonstrate that iAMP-DL is an effective, robust, and stable framework for detecting promising short AMPs. Another comparative study of different negative data sampling methods also confirms the effectiveness of our method and demonstrates that it can also be used to develop a robust model for predicting AMPs in general. The proposed framework was also deployed as an online web server with a user-friendly interface to support the research community in identifying short AMPs.

Improved negative sampling method in collaborative filtering recommendation based on Generative adversarial network

ObjectiveThe problem of low model performance caused by the lack of negative samples in the recommendation method based on implicit feedback information can be solved. MethodsThe implicit feedback recommendation model DAEGAN is constructed based on the conditional generative adversarial network framework. The Denoising Auto-Encoder is used as a generator to capture nonlinear potential factors in the interaction and improve the robustness of model. In this paper, a strong and weak negative sampling strategy is proposed, which combines the visibility of user in time points to mine uninteresting items and acquire strong negative samples, and injects these information into the model by modifying the masking mechanism to solve the problem of missing negative samples. ResultsExperiments on MovieLens 100 K, Amazon Movie and TV, MovieLens 1 M datasets show that the recommendation accuracy of CFGAN based on strong and weak negative sampling and DAEGAN proposed in this paper has been improved. LimitationsThe generation of strong negative samples is based on user interaction records, which cannot solve effectively cold start problems in extremely sparse data. ConclusionsAfter DAEGAN application, the strong and weak negative sampling method proposed in this paper has generally higher recommendation accuracy than those mainstream recommendation algorithms. The code is available at https://github.com/nanjingzhuyuxuan/DAEGAN.

Exploring the Performance of Continuous-Time Dynamic Link Prediction Algorithms

Dynamic Link Prediction (DLP) addresses the prediction of future links in evolving networks. However, accurately portraying the performance of DLP algorithms poses challenges that might impede progress in the field. Importantly, common evaluation pipelines usually calculate ranking or binary classification metrics, where the scores of observed interactions (positives) are compared with those of randomly generated ones (negatives). However, a single metric is not sufficient to fully capture the differences between DLP algorithms, and is prone to overly optimistic performance evaluation. Instead, an in-depth evaluation should reflect performance variations across different nodes, edges, and time segments. In this work, we contribute tools to perform such a comprehensive evaluation. (1) We propose Birth–Death diagrams, a simple but powerful visualization technique that illustrates the effect of time-based train–test splitting on the difficulty of DLP on a given dataset. (2) We describe an exhaustive taxonomy of negative sampling methods that can be used at evaluation time. (3) We carry out an empirical study of the effect of the different negative sampling strategies. Our comparison between heuristics and state-of-the-art memory-based methods on various real-world datasets confirms a strong effect of using different negative sampling strategies on the test area under the curve (AUC). Moreover, we conduct a visual exploration of the prediction, with additional insights on which different types of errors are prominent over time.

ABNS: Association-based negative sampling for collaborative filtering

The purpose of the recommendation systems (RS) is to provide personalized information filtering for users by analyzing their interactions with items. However, the sparsity of data in RS, which means that the number of negative samples is much larger than the number of positive samples, causes the necessity of the negative sampling method. Previous work has shown that an appropriate negative sampling method, rather than simple random negative sampling, can significantly improve the performance of recommendation algorithms. Unfortunately, the existing works have significant shortcomings in explainability and computational complexity. To remedy these shortcomings, we conducted a theoretical analysis of the gradient of the Bayesian Personalized Ranking (BPR) loss function and propose an explainable negative sampling method. Moreover, we conducted experiments on real-world datasets and find that the proposed negative sampling method improves the recommendation performance by an average of 10.23% over uniform negative sampling. Finally, the effectiveness of the proposed method is demonstrated through comparison with other methods and hyperparameter sensitivity analysis.

TriSampler: A Better Negative Sampling Principle for Dense Retrieval

Negative sampling stands as a pivotal technique in dense retrieval, essential for training effective retrieval models and significantly impacting retrieval performance. While existing negative sampling methods have made commendable progress by leveraging hard negatives, a comprehensive guiding principle for constructing negative candidates and designing negative sampling distributions is still lacking. To bridge this gap, we embark on a theoretical analysis of negative sampling in dense retrieval. This exploration culminates in the unveiling of the quasi-triangular principle, a novel framework that elucidates the triangular-like interplay between query, positive document, and negative document. Fueled by this guiding principle, we introduce TriSampler, a straightforward yet highly effective negative sampling method. The keypoint of TriSampler lies in its ability to selectively sample more informative negatives within a prescribed constrained region. Experimental evaluation show that TriSampler consistently attains superior retrieval performance across a diverse of representative retrieval models.

Adaptive Hardness Negative Sampling for Collaborative Filtering

Negative sampling is essential for implicit collaborative filtering to provide proper negative training signals so as to achieve desirable performance. We experimentally unveil a common limitation of all existing negative sampling methods that they can only select negative samples of a fixed hardness level, leading to the false positive problem (FPP) and false negative problem (FNP). We then propose a new paradigm called adaptive hardness negative sampling (AHNS) and discuss its three key criteria. By adaptively selecting negative samples with appropriate hardnesses during the training process, AHNS can well mitigate the impacts of FPP and FNP. Next, we present a concrete instantiation of AHNS called AHNS_{p

Towards a better negative sampling strategy for dynamic graphs

As dynamic graphs have become indispensable in numerous fields due to their capacity to represent evolving relationships over time, there has been a concomitant increase in the development of Temporal Graph Neural Networks (TGNNs). When training TGNNs for dynamic graph link prediction, the commonly used negative sampling method often produces starkly contrasting samples, which can lead the model to overfit these pronounced differences and compromise its ability to generalize effectively to new data. To address this challenge, we introduce an innovative negative sampling approach named Enhanced Negative Sampling (ENS). This strategy takes into account two pervasive traits observed in dynamic graphs: (1) Historical dependence, indicating that nodes frequently reestablish connections they held in the past, and (2) Temporal proximity preference, which posits that nodes are more inclined to connect with those they have recently interacted with. Specifically, our technique employs a designed scheduling function to strategically control the progression of difficulty of the negative samples throughout the training. This ensures that the training progresses in a balanced manner, becoming incrementally challenging, and thereby enhancing TGNNs’ proficiency in predicting links within dynamic graphs. In our empirical evaluation across multiple datasets, we discerned that our ENS, when integrated as a modular component, notably augments the performance of four SOTA baselines. Additionally, we further investigated the applicability of ENS in handling dynamic graphs of varied attributes. Our code is available at https://github.com/qqaazxddrr/ENS.

HDGCN: Dual-Channel Graph Convolutional Network With Higher-Order Information for Robust Feature Learning

Graph convolutional network (GCN) algorithms have been employed to learn graph embedding due to its inductive inference property, which is extended to GCN with higher-order information. However, the application of these higher-order information is confusing and not effectively distinguished. In addition, single channel GCN using higher-order information is weak for robust feature learning, and existing dual-channel GCNs rarely take higher-order information into account. To alleviate the above problems, we propose a dual-channel GCN with higher-order information for robust feature learning, denoted as HDGCN. Firstly, features of positive and negative higher-order graphs are extracted that fully exploits the self-contained attributes and higher-order geometric information. Meantime, the features of original graph structure are extracted by a conventional GCN that utilizes the self-contained feature attributes. Then, node features are represented as edge features by a feature fusion function. For the selection of negative samples, a fractional staggered negative sampling method is applied, by which the trainable graph model gains better topological features. Finally, the performance on seven real-world datasets demonstrates that HDGCN obtains the state-of-the-art performance on pairwise link prediction, higher-order structure prediction, and node classification tasks. By changing the attributes of multiple tasks, it can be proved that HDGCN has good robustness.

Community-Enhanced Contrastive Learning for Graph Collaborative Filtering

Graph collaborative filtering can efficiently find the hidden interests of users for recommender systems in recent years. This method can learn complex interactions between nodes in the graph, identify user preferences, and provide satisfactory recommendations. However, recommender systems face the challenge of data sparsity. To address this, recent studies have utilized contrastive learning to make use of unlabeled data structures. However, the existing positive and negative example sampling methods are not reasonable. Random-based or data augmentation-based sampling cannot make use of useful latent information. Clustering-based sampling methods ignore the semantics of node features and the relationship between global and local information. To utilize the latent structures in the data, we introduce a novel Community-Enhanced Contrastive Learning method to help the recommendation main task called CECL which uses a community detection algorithm to sample examples with semantic and global information, using both known and hidden community connections in the bipartite interaction graph. Extensive experiments are conducted on two well-known datasets, the results of which show a 12% and 8% performance improvement compared to that of the existing baseline methods.

Meta-classifier free negative sampling for extreme multilabel classification

Negative sampling is a common approach for making the training of deep models in classification problems with very large output spaces, known as extreme multilabel classification (XMC) problems, tractable. Negative sampling methods aim to find per instance negative labels with higher scores, known as hard negatives, and limit the computations of the negative part of the loss to these labels. Two well-known methods for negative sampling in XMC models are meta-classifier-based and Maximum Inner product Search (MIPS)-based adaptive methods. Owing to their good prediction performance, methods which employ a meta classifier are more common in contemporary XMC research. On the flip side, they need to train and store the meta classifier (apart from the extreme classifier), which can involve millions of additional parameters. In this paper, we focus on the MIPS-based methods for negative sampling. We highlight two issues which may prevent deep models trained by these methods to undergo stable training. First, we argue that using hard negatives excessively from the beginning of training leads to unstable gradient. Second, we show that when all the negative labels in a MIPS-based method are restricted to only those determined by MIPS, training is sensitive to the length of intervals for pre-processing the weights in the MIPS method. To mitigate the aforementioned issues, we propose to limit the labels selected by MIPS to only a few and sample the rest of the needed labels from a uniform distribution. We show that our proposed MIPS-based negative sampling can reach the performance of LightXML, a transformer-based model trained by a meta classifier, while there is no need to train and store any additional classifier. The code for our experiments is available at https://github.com/xmc-aalto/mips-negative-sampling.

An Adaptive Mixup Hard Negative Sampling for Zero-Shot Entity Linking

Recently, the focus of entity linking research has centered on the zero-shot scenario, where the entity purposed to be labeled at the time of testing was never observed during the training phase, or it may belong to a different domain than the source domain. Current studies have used BERT as the base encoder, as it effectively establishes distributional links between source and target domains. The currently available negative sampling methods all use an extractive approach, which makes it difficult for the models to learn diverse and more challenging negative samples. To address this problem, we propose a generative negative sampling method, adaptive_mixup_hard, which generates more difficult negative entities by fusing the features of both positive and negative samples on top of hard negative sampling and introduces a transformable adaptive parameter, W, to increase the diversity of negative samples. Next, we fuse our method with the Biencoder architecture and evaluate its performance under three different score functions. Ultimately, experimental results on the standard benchmark dataset, Zeshel, demonstrate the effectiveness of our method.

Criterion-based Heterogeneous Collaborative Filtering for Multi-behavior Implicit Recommendation

Recent years have witnessed the explosive growth of interaction behaviors in multimedia information systems, where multi-behavior recommender systems have received increasing attention by leveraging data from various auxiliary behaviors such as tip and collect. Among various multi-behavior recommendation methods, non-sampling methods have shown superiority over negative sampling methods. However, two observations are usually ignored in existing state-of-the-art non-sampling methods based on binary regression: (1) users have different preference strengths for different items, so they cannot be measured simply by binary implicit data; (2) the dependency across multiple behaviors varies for different users and items. To tackle the above issue, we propose a novel non-sampling learning framework named C riterion-guided H eterogeneous C ollaborative F iltering (CHCF). CHCF introduces both upper and lower thresholds to indicate selection criteria, which will guide user preference learning. Besides, CHCF integrates criterion learning and user preference learning into a unified framework, which can be trained jointly for the interaction prediction of the target behavior. We further theoretically demonstrate that the optimization of Collaborative Metric Learning can be approximately achieved by the CHCF learning framework in a non-sampling form effectively. Extensive experiments on three real-world datasets show the effectiveness of CHCF in heterogeneous scenarios.

RPIPCM: A deep network model for predicting lncRNA-protein interaction based on sequence feature encoding

LncRNA-protein interactionplays an important regulatory role in biological processes. In this paper, the proposed RPIPCM based on a novel deep network model uses the sequence feature encoding of both RNA and protein to predict lncRNA-protein interactions (LPIs). A negative sampling of sliding window method is proposed for solving the problem of unbalanced between positive and negative samples. The proposed negative sampling method is effective and helpful to solve the problem of data imbalance in the existing LPIs research by comparative experiments. Experimental results also show that the proposed sequence feature encoding method has good performance in predicting LPIs for different datasets of different sizes and types. In the RPI488 dataset related to animal, compared with the direct original sequence encoding model, the accuracy of sequence feature encoding model increased by 1.02%, the recall increased by 4.08%, and the value of MCC increased by 1.67%. In the case of the plant dataset ATH948, the sequence feature-based encoding demonstrated a 1.58% higher accuracy, a 1.53% higher recall, a 1.62% higher specificity, a 1.62% higher precision, and a 3.16% higher value of MCC compared to the direct original sequence-based encoding. Compared with the latest prediction work in the ZEA22133 dataset, RPIPCM is shown to be more effective with the accuracy increased by 2.23%, the recall increased by 1.78%, the specificity increased by 2.67%, the precision increased by 2.52%, and the value of MCC increased by 4.43%, which also proves the effectiveness and robustness of RPIPCM. In conclusion, RPIPCM of deep network model based on sequence feature encoding can automatically mine the hidden feature information of the sequence in the lncRNA-protein interaction without relying on external features or prior biomedical knowledge, and its low cost and high efficiency can provide a reference for biomedical researchers.

Simple Knowledge Graph Completion Model Based on Differential Negative Sampling and Prompt Learning

Knowledge graphs (KGs) serve as a crucial resource for numerous artificial intelligence tasks, significantly contributing to the advancement of the AI field. However, the incompleteness of existing KGs hinders their effectiveness in practical applications. Consequently, researchers have proposed the task of KG completion. Currently, embedding-based techniques dominate the field as they leverage the structural information within KGs to infer and complete missing parts. Nonetheless, these methods exhibit limitations. They are limited by the quality and quantity of structural information and are unable to handle the missing entities in the original KG. To overcome these challenges, researchers have attempted to integrate pretrained language models and textual data to perform KG completion. This approach utilizes the definition statements and description text of entities within KGs. The goal is to compensate for the latent connections that are difficult for traditional methods to obtain. However, text-based methods still lag behind embedding-based models in terms of performance. Our analysis reveals that the critical issue lies in the selection process of negative samples. In order to enhance the performance of the text-based methods, various types of negative sampling methods are employed in this study. We introduced prompt learning to fill the gap between the pre-training language model and the knowledge graph completion task, and to improve the model reasoning level. Simultaneously, a ranking strategy based on KG structural information is proposed to utilize KG structured data to assist reasoning. The experiment results demonstrate that our model exhibits strong competitiveness and outstanding inference speed. By fully exploiting the internal structural information of KGs and external relevant descriptive text resources, we successfully elevate the performance levels of KG completion tasks across various metrics.

TuckerDNCaching: high-quality negative sampling with tucker decomposition

Knowledge Graph Embedding (KGE) translates entities and relations of knowledge graphs (KGs) into a low-dimensional vector space, enabling an efficient way of predicting missing facts. Generally, KGE models are trained with positive and negative examples, discriminating positives against negatives. Nevertheless, KGs contain only positive facts; KGE training requires generating negatives from non-observed ones in KGs, referred to as negative sampling. Since KGE models are sensitive to inputs, negative sampling becomes crucial, and the quality of the negatives becomes critical in KGE training. Generative adversarial networks (GAN) and self-adversarial methods have recently been utilized in negative sampling to address the vanishing gradients observed with early negative sampling methods. However, they introduce the problem of false negatives with high probability. In this paper, we extend the idea of reducing false negatives by adopting a Tucker decomposition representation, i.e., TuckerDNCaching, to enhance the semantic soundness of latent relations among entities by introducing a relation feature space. TuckerDNCaching ensures the quality of generated negative samples, and the experimental results reflect that our proposed negative sampling method outperforms the existing state-of-the-art negative sampling methods.

Learning graph representation by aggregating subgraphs via mutual information maximization

Information theory has shown a notable performance in the field of computer vision (CV) and natural language processing (NLP), therefore, many works start to learn better node-level and graph-level representations in the information theory perspective. Previous works have shown great performance by maximizing the mutual information between graph and node representation to capture graph information. However, a simple mixture of information in a single node representation leads to a lack of information related to the graph structure, which leads to the information gap between model and theoretical optimal solution. To solve this problem, we propose to replace the node representation with subgraph representation to reduce the information gap between the model and the optimal case. And to capture enough information of original graph, three operators (information aggregators): attribute-conv, layer-conv and subgraph-conv, are designed to gather information from different aspects, respectively. Moreover, to generate more expressive subgraphs, we propose a universal framework to generate subgraphs autoregressively, which provides a comprehensive understanding of the graph structure in a learnable way. We proposed a Head–Tail negative sampling method to provide more negative samples for more efficient and effective contrastive learning. Moreover, all these components can be plugged into any existed Graph Neural Networks. Experimentally, we achieve new state-of-the-art results in several benchmarks under the unsupervised case. We also evaluate our model on semi-supervised learning tasks and make a fair comparison to state-of-the-art semi-supervised methods.

Adaptive dual graph contrastive learning based on heterogeneous signed network for predicting adverse drug reaction

Adverse Drug Reactions (ADRs) resulting from drug combinations can endanger patients' health and life. In this paper, we propose ADGCL, a novel Adaptive Dual Graph Contrastive Learning based on heterogeneous signed networks for predicting ADRs. Specifically, ADGCL first explicitly models positive and negative drug-drug interactions (DDIs) using a heterogeneous signed network to learn semantic-rich drug feature representations. Additionally, ADGCL employs a dual graph contrastive learning strategy including self-supervised contrastive learning and micro-supervised learning to learn high-level features, which significantly improves prediction performance. Furthermore, ADGCL incorporates an adaptive negative sampling method to generate negative samples. Lastly, an encoder based on the implicit graph neural network is designed to capture long-range dependencies in underlying networks, thereby improving the quality of drug feature representation. Comprehensive experimental results on real-world datasets demonstrate that ADGCL is a promising ADR prediction model.

Reinforced PU-learning with Hybrid Negative Sampling Strategies for Recommendation

The data of recommendation systems typically only contain the purchased item as positive data and other un-purchased items as unlabeled data. To train a good recommendation model, in addition to the known positive information, we also need high-quality negative information. Capturing negative signals in positive and unlabeled data is challenging for recommendation systems. Most studies have used specific data and proposed negative sampling methods suitable to the data characteristics. Existing negative sampling strategies cannot automatically select suitable approaches for different data. However, this one-size-fits-all strategy often makes potential positive samples considered as negative, or truly negative samples considered as potential positive samples and recommend to users. In this way, it will not only turn down the recommendation result, but even also have an adverse effect. Accordingly, we propose a novel negative sampling model, Reinforced PU-learning with Hybrid Negative Sampling Strategies for Recommendation (RHNSR), which can combine multiple sampling strategies and dynamically adjust the proportions used by different sampling strategies. In addition, ensemble learning, which integrates various model sampling strategies for obtaining an improved solution, was applied to RHNSR. Extensive experiments were conducted on three real-world recommendation datasets, and the experimental results indicated that the proposed model significantly outperformed state-of-the-art baseline models and revealed significant improvements in precision and hit ratio (49.02% and 37.41%, respectively).