Negative Sampling Method Research Articles

Collaborative knowledge-aware recommendation based on neighborhood negative sampling

Knowledge graph and negative sampling, as the sources of auxiliary information, have been playing a vital role since they were included in the recommendation system. Creating an end-to-end model based on a graph neural network is the current technical trend. In order to generate samples with negative signals for the recommended model, negative sampling is also done from the unobserved data. However, the current end-to-end model based on graph neural network is unable to effectively capture the high-order collaboration signals of items, making it unable to learn the high-quality user and item representation. Additionally, existing negative sampling technique is insufficient for producing high-quality negative samples that can both reflect users’ true preferences and offer information for model training. Therefore, this paper proposes a new model, namely, collaborative knowledge-aware network based on neighborhood negative sampling (CKNNS). Users and items are technically modeled based on the collaboration-aware and knowledge-aware. To help the model better capture the users’ preferences and the personality traits, a gated aggregation strategy is used to adaptively capture collaboration-aware signals and knowledge-aware information. Meanwhile, a negative sampling method based on user’s neighborhood has been proposed. Specifically, the similarity of users is judged according to their collaboration information, so as to build a negative sampling domain for a specific user and improve the quality of negative sampling. The experimental outcomes on four benchmark datasets reveal that CKNNS outperforms sophisticated approaches like LightGCN, CKAN, KGIN and KGCL.

Molecular Generative Model via Retrosynthetically Prepared Chemical Building Block Assembly.

Deep generative models are attracting attention as a smart molecular design strategy. However, previous models often render molecules with low synthesizability, hindering their real-world applications. Here, a novel graph-based conditional generative model which makes molecules by tailoring retrosynthetically prepared chemical building blocks until achieving target properties in an auto-regressive fashion is proposed. This strategy improves the synthesizability and property control of the resulting molecules and also helps learn how to select appropriate building blocks and bind them together to achieve target properties. By applying a negative sampling method to the selection process of building blocks, this model overcame a critical limitation of previous fragment-based models, which can only use molecules from the training set during generation. As a result, the model works equally well with unseen building blocks without sacrificing computational efficiency. It is demonstrated that the model can generate potential inhibitors with high docking scores against the 3CL protease ofSARS-COV-2.

SimH: A Novel Representation Learning Model With Activation and Projection Mechanisms for COVID-19 Knowledge Bases.

The emergence of coronavirus disease 2019 (COVID-19) has had a significant impact on healthcare and the economy. With representation learning applied in constructing COVID-19 knowledge graphs, abundant COVID-19-related knowledge collected by clinicians and scientists all over the world can be utilized to deepen their understanding of the mechanism and related biological functions of the disease. However, most existing representation learning models cannot deal well with COVID-19 knowledge graph due to its low-connected star-like structure and various complex nonlinear relationships. Besides, lacking reliable negative triplets is also a difficult problem, yet to be adequately resolved. In this article, we propose a novel representation learning model called translation on hyperplanes with an activation operation and similar semantic sampling (SimH) for COVID-19 knowledge graphs. In our proposed SimH, an activation operation is designed to provide additional interaction features for low-in-degree entities. Then the hyperplane projection technique is introduced to the distance-based scoring function so that those complex nonlinear relationships can be modeled with lower complexity maintained in comparison with other nonlinear models. Moreover, a negative triplet sampling method that adaptively replaces entities with similar semantics is introduced to generate reliable negative triplets. To verify the effectiveness of SimH, extensive experiments are conducted on the COVID-19-Concepts dataset. The experimental results show that our SimH model achieves significant improvements in prediction and classification accuracy over existing knowledge representation learning models.

A semantic relation-aware deep neural network model for end-to-end conversational recommendation

Conversational recommendation system (CRS) aims at recommending appropriate items to the user through a multi-turn conversation. The end-to-end CRS is a type of CRS that models the recommendation task and the conversation task simultaneously which has attracted more and more attention in recent years. At the same time, knowledge graph and Transformer are incorporated into the end-to-end CRS to generate better recommendations and better responses to the user, which makes the CRS have state-of-the-art performance. It is known that there exist semantic relations in a conversation. However, we observe that existing end-to-end CRSs in general ignore the semantic relations in the conversation and therefore would likely hinder the performance of CRSs. Motivated by this, we propose a gated cross- and self-attention based CRS utilizing semantic relation information (ASR) model, which can explicitly model and utilize the semantic relations in a conversation. To the best of our knowledge, we are the first to advocate for modelling and utilizing the semantic relations in the end-to-end CRS, which could help to improve the performance of the CRS. Furthermore, to mitigate the class-imbalance problem that most end-to-end CRSs face, we propose a new negative sampling method which could make the proposed CRS learn better. Moreover, we design a Transformer-based dialogue module integrating the semantic relations in a conversation to generate more diversified and precise responses. Extensive experiments on widely used benchmark datasets demonstrate that the proposed ASR model achieves state-of-the-art results in both recommendation and conversation tasks.

IGNSCDA: Predicting CircRNA-Disease Associations Based on Improved Graph Convolutional Network and Negative Sampling.

Accumulating evidences have shown that circRNA plays an important role in human diseases. It can be used as potential biomarker for diagnose and treatment of disease. Although some computational methods have been proposed to predict circRNA-disease associations, the performance still need to be improved. In this paper, we propose a new computational model based on Improved Graph convolutional network and Negative Sampling to predict CircRNA-Disease Associations. In our method, it constructs the heterogeneous network based on known circRNA-disease associations. Then, an improved graph convolutional network is designed to obtain the feature vectors of circRNA and disease. Further, the multi-layer perceptron is employed to predict circRNA-disease associations based on the feature vectors of circRNA and disease. In addition, the negative sampling method is employed to reduce the effect of the noise samples, which selects negative samples based on circRNA's expression profile similarity and Gaussian Interaction Profile kernel similarity. The 5-fold cross validation is utilized to evaluate the performance of the method. The results show that IGNSCDA outperforms than other state-of-the-art methods in the prediction performance. Moreover, the case study shows that IGNSCDA is an effective tool for predicting potential circRNA-disease associations.

Self-Training Enhanced: Network Embedding and Overlapping Community Detection With Adversarial Learning.

Network embedding (NE) aims to encode the relations of vertices into a low-dimensional space. After NE, we can obtain the learned vectors of vertices that preserve the proximity of network structures for subsequent applications, e.g., vertex classification and link prediction. In existing NE models, they usually exploit the skip-gram with a negative sampling method to optimize their objective functions. Generally, this method learns the vertex representation only from the local connectivity of vertices (i.e., neighbors). However, there is a larger scope of vertex connectivity in real-world scenarios: a vertex may have multifaceted aspects and should belong to overlapping communities. Taking a social network as the overlapping example, a user may subscribe to the channels of politics, economy, and sports simultaneously, but the politics share more common attributes with the economy and less with the sports. In this article, we propose an adversarial learning approach (ACNE) for modeling overlapping communities of vertices. Specifically, we map the association between communities and vertices into an embedding space. Moreover, we take further research on enhancing our ACNE with the following two operations. First, in the initialization stage, we adopt a walking strategy with perception to obtain paths containing more possible boundary vertices to improve overlapping community detection. Then, after representation learning with ACNE, we use soft community assignments from a simple classifier as supervision to update the weights of ACNE. This self-training mechanism referred to as ACNE-ST can help ACNE to achieve better performance. Experimental results demonstrate that the proposed methods, including ACNE and ACNE-ST, can outperform the state-of-the-art models on the subsequent tasks of vertex classification and overlapping community detection.

Benchmarks in antimicrobial peptide prediction are biased due to the selection of negative data.

Antimicrobial peptides (AMPs) are a heterogeneous group of short polypeptides that target not only microorganisms but also viruses and cancer cells. Due to their lower selection for resistance compared with traditional antibiotics, AMPs have been attracting the ever-growing attention from researchers, including bioinformaticians. Machine learning represents the most cost-effective method for novel AMP discovery and consequently many computational tools for AMP prediction have been recently developed. In this article, we investigate the impact of negative data sampling on model performance and benchmarking. We generated 660 predictive models using 12 machine learning architectures, a single positive data set and 11 negative data sampling methods; the architectures and methods were defined on the basis of published AMP prediction software. Our results clearly indicate that similar training and benchmark data set, i.e. produced by the same or a similar negative data sampling method, positively affect model performance. Consequently, all the benchmark analyses that have been performed for AMP prediction models are significantly biased and, moreover, we do not know which model is the most accurate. To provide researchers with reliable information about the performance of AMP predictors, we also created a web server AMPBenchmark for fair model benchmarking. AMPBenchmark is available at http://BioGenies.info/AMPBenchmark.

Leveraging network structure for efficient dynamic negative sampling in network embedding

Unsupervised network embedding learns low-dimensional vector representations of nodes based on the network structure. However, typical graphs only contain positive edges. Hence, most network embedding models take sampled negative edges as input to avoid trivial solutions. However, conventional negative sampling methods follow fixed distributions to generate negative samples. Such a static sampling strategy could dramatically hurt the performance of node embeddings in downstream tasks. Researchers have proposed several adaptive sampling strategies to improve the quality of negative samples. However, existing methods do not sufficiently use the network structure to explore high-quality samples, resulting in complex and inefficient negative sampling. This paper develops a novel dynamic negative sampling scheme, which maintains a candidate sample population for each node. We update the populations dynamically and retain high-quality samples in each iteration. We develop a novel network embedding algorithm based on the proposed model, which selects high-quality negative samples adaptively from the population. We conduct extensive experiments to evaluate its effectiveness based on several benchmark datasets. The experimental results verify the superiority of the proposed methods over state-of-the-art approaches.

Negative sampling and rule mining for explainable link prediction in knowledge graphs

Several KG embedding methods were proposed to learn low dimensional vector representations of entities and relations of a KG. Such representations facilitate the link prediction task, in the service of inference and KG completion. In this context, it is important to achieve both an efficient KG embedding and explainable predictions. During learning of efficient embeddings, sampling negative triples was highlighted as an important step as KGs only have observed positive triples. We propose an efficient simple negative sampling (SNS) method based on the assumption that the entities which are closer in the embedding space to the corrupted entity are able to provide high-quality negative triples. As for explainability, it actually constitutes a thriving research question especially when it comes to analyze KGs with their rich semantics rooted in description logics. Hence, we propose in this paper a new rule mining method on the basis of learned embeddings. We extensively evaluate our proposals through several experiments. We evaluate our SNS sampling method plugged to several KG embedding models through link prediction task performances on well-known datasets. Experimental results show that the SNS improves the prediction performance of KG embedding models, and outperforms the existing sampling methods. To assess the performance of our rule mining method with and without SNS, we mine and evaluate rules on three popular datasets. The extracted rules are evaluated as knowledge nuggets extracted from the KG and also as support for explainable link prediction. The overall results are good and open the way to many improvements and new perspectives.

Computational methods, databases and tools for synthetic lethality prediction.

Synthetic lethality (SL) occurs between two genes when the inactivation of either gene alone has no effect on cell survival but the inactivation of both genes results in cell death. SL-based therapy has become one of the most promising targeted cancer therapies in the last decade as PARP inhibitors achieve great success in the clinic. The key point to exploiting SL-based cancer therapy is the identification of robust SL pairs. Although many wet-lab-based methods have been developed to screen SL pairs, known SL pairs are less than 0.1% of all potential pairs due to large number of human gene combinations. Computational prediction methods complement wet-lab-based methods to effectively reduce the search space of SL pairs. In this paper, we review the recent applications of computational methods and commonly used databases for SL prediction. First, we introduce the concept of SL and its screening methods. Second, various SL-related data resources are summarized. Then, computational methods including statistical-based methods, network-based methods, classical machine learning methods and deep learning methods for SL prediction are summarized. In particular, we elaborate on the negative sampling methods applied in these models. Next, representative tools for SL prediction are introduced. Finally, the challenges and future work for SL prediction are discussed.

DCFGAN: An adversarial deep reinforcement learning framework with improved negative sampling for session-based recommender systems

In recent years, with the development of Internet technology, recommender systems have been widely used by virtue of their ability to meet the personalized needs of users. In order to make full use of users’ interactive behaviors, session-based recommender systems have attracted growing research interest. In previous session-based recommender systems, users' historical interactive behavior is utilized to train and update users' preferences, but users' responses to the current recommendation results (immediate feedback) are not effectively exploited to optimize the recommendation strategy. This leads to the decrease of subsequent recommendation accuracy. Aiming at this problem, based on the Recurrent Neural Network (RNN), this paper combines Reinforcement Learning (RL) and Generative Adversarial Networks (GANs). We fully exploit the users’ immediate feedback with RL, and simultaneously take advantage of GAN to satisfy the requirement of training data brought by RL. Furthermore, we optimize the negative sampling method and propose Deep Generative Adversarial Networks-based Collaborative Filtering (DCFGAN). The experimental results show that this algorithm can effectively improve the recommendation accuracy in session-based recommender systems.

Bytecode Similarity Detection of Smart Contract across Optimization Options and Compiler Versions Based on Triplet Network

In recent years, the number of smart contracts running in the blockchain has increased rapidly, accompanied by many security problems, such as vulnerability propagation caused by code reuse or vicious transaction caused by malicious contract deployment, for example. Most smart contracts do not publish the source code, but only the bytecode. Based on the research of bytecode similarity of smart contract, smart contract upgrade, vulnerability search and malicious contract analysis can be carried out. The difficulty of bytecode similarity research is that different compilation versions and optimization options lead to the diversification of bytecode of the same source code. This paper presents a solution, including a series of methods to measure the similarity of smart contract bytecode. Starting from the opcode of smart contract, a method of pre-training the basic block sequence of smart contract is proposed, which can embed the basic block vector. Positive samples were obtained by basic block marking, and the negative sampling method is improved. After these works, we put the obtained positive samples, negative samples and basic blocks themselves into the triplet network composed of transformers. Our solution can obtain evaluation results with an accuracy of 97.8%, so that the basic block sequence of optimized and unoptimized options can be transformed into each other. At the same time, the instructions are normalized, and the order of compiled version instructions is normalized. Experiments show that our solution can effectively reduce the bytecode difference caused by optimization options and compiler version, and improve the accuracy by 1.4% compared with the existing work. We provide a data set covering 64 currently used Solidity compilers, including one million basic block pairs extracted from them.

Improve Representation for Cross-Language Clone Detection by Pretrain Using Tree Autoencoder

With the rise of deep learning in recent years, many code clone detection (CCD) methods use deep learning techniques and achieve promising results, so is cross-language CCD. However, deep learning techniques require a dataset to train the models. The dataset is typically small and has a gap between real-world clones due to the difficulty of collecting datasets for cross-language CCD. This creates a data bottleneck problem: data scale and quality issues will cause that model with a better design can still not reach its full potential. To mitigate this, we propose a tree autoencoder (TAE) architecture. It uses unsupervised learning to pretrain with abstract syntax trees (ASTs) of a large-scale dataset, then fine-tunes the trained encoder in the downstream CCD task. Our proposed TAE contains a tree Long Short-Term Memory (LSTM) encoder and a tree LSTM decoder. We design a novel embedding method for AST nodes, including type embedding and value embedding. In the training of TAE, we present an “encode and decode by layers” strategy and a node-level batch size design. For the CCD dataset, we propose a negative sampling method based on probability distribution. The experimental results on two datasets verify the effeteness of our embedding method, as well as that TAE and its pretrain enhance the performance of the CCD model. The node context information is well captured, and the reconstruction accuracy of the node-value reaches 95.45%. TAE pretrain improves the performance of CCD with a 4% increase in F1 score, which alleviates the data bottleneck problem.

Region or Global A Principle for Negative Sampling in Graph-based Recommendation

Graph-based recommendation systems are blossoming recently, which models user-item interactions as a user-item graph and utilizes graph neural networks (GNNs) to learn the embeddings for users and items. A fundamental challenge of graph-based recommendation is that there only exists observed positive user-item pairs in the user-item graph. Negative sampling is a vital technique to solve the one-class problem and is widely used in many recommendation methods. However, the previous works only focus on the design of negative sampling distribution but ignore the sampled region for negative sampling. In this work, we propose the Three-Region Principle to guide negative sampling, which suggests that we should negatively sample more items at an intermediate region and less adjacent and distant items. In light of this principle, we present the RecNS method, which is a general negative sampling method designed with two sampling strategies: positive-assisted sampling and exposure-augmented sampling. Instead of sampling existing negative items from graph data, we merge these two strategies in embedding space to generate negative item embeddings. Extensive experiments demonstrate that our proposed RecNS method significantly outperforms all negative sampling baselines, e.g., 10.47% for PinSage, 6.02% for NGCF, and 8.20% for LightGCN in terms of Recall@20 on the Alibaba dataset.

Language mapping functions: Improving softmax estimation and word embedding quality

SummaryOne of the best methods for estimating the softmax layer in neural network language models is the noise‐contrastive estimation (NCE) method. However, this method is not proper for word embedding applications compared with some other robust methods such as the negative sampling (NEG) method. The NEG method implements the pointwise mutual information (PMI) relation between the word‐context space in the neural network, and the NCE method implements conditional probability. Both the NCE and NEG methods use dot‐product‐based mapping to map words and contexts vector to the probabilities. This article presents the parametric objective function, which uses the mapping function as the parameter. Also, we obtained a parametric relation between word‐context space according to the mapping parameter. Using the parametric objective function, we identify conditions for a mapping that make it a proper selection for both softmax estimation and word embedding. The article also presents two specific mapping functions with the required conditions, and we compared their performance with that of the dot‐product mapping function. The performance of the new mapping functions is also reported over common word embedding and language models' benchmarks.

Pair-wise ranking based preference learning for points-of-interest recommendation

Recommending point-of-interest (POI) to users accurately is a hot topic in business. In the past, many researchers proposed recommendation models based on collaborative filtering or matrix factorization from the perspectives of time, geography, and social relationship. However, only a few studies have focused on user preference which is the key factor influencing user decision. This work focuses on studying the representation and mining of user preference from check-in data for POI recommendation. Pair-wise ranking is the common solution for implementing preference learning. However, traditional ways of constructing pair-wise data cut off the connections between multiple options in the decision process, affecting the effectiveness of preference learning. In this work, we change the ratio of negative to positive instance in pair-wise data from 1:1 to k:1 to ensure the data construction in line with the real decision making process. We propose a new negative sampling method taking the geographical distance and POI categorical distance into consideration jointly for enhancing the quality of training data. For our specialized pair-wise data, we propose a new optimization criterion for implementing effective preference learning. Finally, we conduct extensive experiments on two real-world datasets to validate the effectiveness of our proposed approach. The experiment results show that our approach outperforms the state-of-the-art models by at least 19.7% on F1-Score and 24.4% on nDCG. Additionally, our approach can be easily generalized to other domains, such as commodities, news, and movie recommendation.

Simple and automated negative sampling for knowledge graph embedding

Negative sampling, which samples negative triplets from non-observed ones in knowledge graph (KG), is an essential step in KG embedding. Recently, generative adversarial network (GAN) has been introduced in negative sampling. By sampling negative triplets with large gradients, these methods avoid the problem of vanishing gradient and thus obtain better performance. However, they make the original model more complex and harder to train. In this paper, motivated by the observation that negative triplets with large gradients are important but rare, we propose to directly keep track of them with the cache. In this way, our method acts as a “distilled” version of previous GAN-based methods, which does not waste training time on additional parameters to fit the full distribution of negative triplets. However, how to sample from and update the cache are two critical questions. We propose to solve these issues by automated machine learning techniques. The automated version also covers GAN-based methods as special cases. Theoretical explanation of NSCaching is also provided, justifying the superior over fixed sampling scheme. Besides, we further extend NSCaching with skip-gram model for graph embedding. Finally, extensive experiments show that our method can gain significant improvements on various KG embedding models and the skip-gram model and outperforms the state-of-the-art negative sampling methods.

POI Recommendation Method Using Deep Learning in Location‐Based Social Networks

To solve the problems of large data sparsity and lack of negative samples in most point of interest (POI) recommendation methods, a POI recommendation method based on deep learning in location‐based social networks is proposed. Firstly, a bidirectional long‐short‐term memory (Bi‐LSTM) attention mechanism is designed to give different weights to different parts of the current sequence according to users’ long‐term and short‐term preferences. Then, the POI recommendation model is constructed, the sequence state data of the encoder is input into Bi‐LSTM‐Attention to get the attention representation of the current POI check‐in sequence, and the Top‐N recommendation list is generated after the decoder processing. Finally, a negative sampling method is proposed to obtain an effective negative sample set, which is used to improve the calculation of the Bayesian personalized ranking loss function. The proposed method is demonstrated experimentally on Foursquare and Gowalla datasets. The experimental results show that the proposed method has better accuracy, recall, and F1 value than other comparison methods.

A collaborative filtering recommendation framework based on Wasserstein GAN

Compared with the original GAN, Wasserstein GAN minimizes the Wasserstein Distance between the generative distribution and the real distribution, can well capture the potential distribution of data and has achieved excellent results in image generation. However, the exploration of Wasserstein GAN on recommendation systems has received relatively less scrutiny. In this paper, we propose a collaborative filtering recommendation framework based on Wasserstein GAN called CFWGAN to improve recommendation accuracy. By learning the real user distribution, we can mine the potential nonlinear interactions between users and items, and capture users’ preferences for all items. Besides, we combine two positive and negative item sampling methods and add the reconstruction loss to the generator’s loss. This can well handle the problem of discrete data in recommendation (relative to the continuity of image data). By continuously approximating the generative distribution to the real user distribution, we can finally obtain better users’ preference information and provide higher accuracy in recommendation. We evaluate the CFWGAN model on three real-world datasets, and the empirical results show that our method is competitive with or superior to state-of-the-art approaches on the benchmark top-N recommendation task.

HNS: Hierarchical negative sampling for network representation learning

Network representation learning (NRL) aims at modeling network graph by encoding vertices and edges into a low-dimensional space. These learned representations can be used for subsequent applications, such as vertex classification and link prediction. Negative Sampling (NS) is the most widely used method for boosting the performance of NRL. However, most of the existing work only randomly draws negative samples based on vertex frequencies, i.e., the vertices with higher frequency are more likely to be drawn, which ignores the situation that the sampled one may not be a true negative sample, thus, lead to undesirable embeddings. In this paper, we propose a new negative sampling method, called Hierarchical Negative Sampling (HNS), which is able to model the latent structures of vertices and learn the relations among them. During sampling, HNS can draw more appropriate negative samples and thereby obtain better performance on network embeddings. Firstly, we theoretically demonstrate the superiority of HNS over NS. And then we use experimental results to show that our proposed method outperforms the state-of-the-art models on vertex classification tasks at different training scales in real-world networks.