Item Embeddings Research Articles

Using Graph Neural Networks for Social Recommendations

Recommender systems have revolutionized the way users discover and engage with content. Moving beyond the collaborative filtering approach, most modern recommender systems leverage additional sources of information, such as context and social network data. Such data can be modeled using graphs, and the recent advances in Graph Neural Networks have led to the prominence of a new family of graph-based recommender system algorithms. In this work, we propose the RelationalNet algorithm, which not only models user–item, and user–user relationships but also item–item relationships with graphs and uses them as input to the recommendation process. The rationale for utilizing item–item interactions is to enrich the item embeddings by leveraging the similarities between items. By using Graph Neural Networks (GNNs), RelationalNet incorporates social influence and similar item influence into the recommendation process and captures more accurate user interests, especially when traditional methods fall short due to data sparsity. Such models improve the accuracy and effectiveness of recommendation systems by leveraging social connections and item interactions. Results demonstrate that RelationalNet outperforms current state-of-the-art social recommendation algorithms.

Advanced sentence-embedding method considering token importance based on explainable artificial intelligence and text summarization model

Although pretrained language models achieve high performance on various natural language processing tasks, they still require further improvements in the sentence embedding task. Many studies have improved performance in this task using pre-trained language models and contrastive learning, but these approaches are limited because they are based on naive average pooling and CLS tokens. Therefore, we propose an advanced sentence-embedding method based on weighted pooling that considers token importance. Specifically, the token importance is calculated by combining an explainable artificial-intelligence module with a text summarization model, and the final sentence embedding is derived through weighted pooling token embedding and token importance. Thus, we derive a sentence embedding that considers both the local information of the token embedding and the global information of the entire sentence. Experimental results reveal that our proposed sentence embedding outperforms other models on both text similarity tasks and text classification. Moreover, the proposed method’s robustness is verified through the results of an ablation study.

Deep adaptive collaborative graph neural network for social recommendation

Most graph convolutional network (GCN)-based social recommendation frameworks fuse social links with user-item interactions to enrich user representations, which alleviate the cold-start problem and data sparsity problem. However, GCN-based recommender systems still suffer from two limitations. First, Excessive reliance on social graphs to extract user interests for rating predictions is unreliable due to social inconsistency. Second, GCN-based models suffer from over-smoothing problems, node embeddings become more similar when going deeper to enable larger receptive fields. To address the two aforementioned problems simultaneously, we propose a Deep Adaptive Collaborative Graph Neural Network for Social Recommendation (DUI-SoRec). First, the graph generation module decomposes the user-item interaction to generate two subgraphs: an u2u graph and an i2i graph. Secondly, the graph learning module utilizes a deep adaptive graph neural network to learn user and item embeddings on the two subgraphs and the existing social graph, while solving the over-smoothing problem. Finally, we designed a refined fusion module to aggregate the social graph and u2u graph to address the social inconsistency. We conducted extensive experiments on four real-world datasets and the results demonstrate the model’s effectiveness.

Graph gating-mixer for sequential recommendation

Recent Transformer-based architectures have achieved encouraging performance for sequential recommendation, whereas their computational complexity is quadratic to the sequence length. MLP4Rec is a promising solution to settle this issue, which captures item transition patterns by a MLP-Mixer layer. Despite effectiveness, we argue that it still faces two critical limitations. On the one hand, it employs the one-hot ID technique to generate each user/item embedding, which has no specific semantics apart from being an identifier. In this case, given these ID embeddings as the original input of a MLP-Mixer layer, it is non-trivial to distill the useful information for other layers. On the other hand, it fails to explicitly differentiate the significance of different factors of an item, which is unrealistic to capture the user’s true taste in a short context; meanwhile, it also does not discriminate the importance of each item instance given the recent actions of a user. To overcome such two limitations, we propose a new solution for sequential recommendation, namely a graph Gating-Mixer Recommender (GMRec). Our solution decomposes the sequential recommendation workflow into three steps. First, by means of graph neural networks, we embed a linear graph propagation module to produce high-quality user and item embeddings. Afterwards, we replace the MLP-Mixer layer in MLP4Rec with a devised dual gating block, which could dynamically control what features and which items can be passed to the downstream layers. Lastly, we devise a user-specific gating strategy to adaptively integrate two components in GMRec. Extensive experiments are performed on the Beauty, Cellphone, Gowalla, and ML-10M datasets, demonstrating the rationality and effectiveness of our solution. Specifically, when Precision@10, Recall@10, MAP@10, and NDCG@10 are adopted as evaluation metrics, the performance gains of GMRec over recent state-of-the-art methods on four datasets are 11.91%, 19.46%, 9.56%, and 13.01%, respectively. Our implemented codes and datasets are available via https://github.com/wubinzzu/GMRec.

Adversarial Heterogeneous Graph Neural Network for Robust Recommendation

Recommendation systems play a vital role in identifying the hidden interactions between users and items in online social networks. Recently, graph neural networks (GNNs) have exhibited significant performance gains by modeling the information propagation process in graph-structured data for a recommendation. However, existing GNN-based methods do not have broad applicability to heterogeneous graphs that integrate auxiliary data with diverse types. Moreover, graph structures are susceptible to noise and even unnoticed malicious perturbations, as perturbations from connected nodes can create cumulative effects on a target node in the graph. To enhance the robustness and generalization of GNN-based recommendations, we propose a new optimization model named Adversarial Heterogeneous Graph Neural Network for RECommendation (AHGNNRec). First, AHGNNRec learns user and item embeddings by exploring the distinct contributions of various types of interactions between users and items using a hierarchical heterogeneous graph neural network (HGNN). Second, to produce more robust embeddings for recommendations, we employ the adversarial training (AT) method to optimize the HGNN layers. AT is a min-max optimization training process where the generated adversarial fake nodes from normal nodes with intentional perturbations try to maximally deteriorate the recommendation performance. Following this, we learn about these adversarial user or item nodes by minimizing the impact of an additional regularization term for the recommendation. The experimental outcomes on two real-world benchmark datasets demonstrate the effectiveness of AHGNNRec.

Creating synthetic datasets for collaborative filtering recommender systems using generative adversarial networks

Research and education in machine learning requires diverse, representative, and open datasets that contain sufficient samples to handle the necessary training, validation, and testing tasks. Currently, the Recommender Systems area includes a large number of subfields in which accuracy and beyond-accuracy quality measures are continuously being improved. To feed this research variety, it is both necessary and convenient to reinforce the existing datasets with synthetic ones. This paper proposes a Generative Adversarial Network (GAN)-based method to generate collaborative filtering datasets in a parameterized way by selecting their preferred number of users, items, samples, and stochastic variability. This parameterization cannot be performed using regular GANs. Our GAN model is fed with dense, short, and continuous embedding representations of items and users, instead of sparse, large, and discrete vectors, to ensure fast and accurate learning, as compared to the traditional approach based on large and sparse input vectors. The proposed architecture includes a DeepMF model to extract the dense user and item embeddings and a clustering process to convert the dense GAN generated samples to the discrete and sparse samples necessary to create each required synthetic dataset. The results from three different source datasets show adequate distributions and expected quality values and evolutions in the generated datasets compared to the source datasets. Synthetic datasets and source codes are available to researchers.

Reducing noise-triplets via differentiable sampling for knowledge-enhanced recommendation with collaborative signal guidance

Knowledge Graph (KG) is widely used for recommendation tasks due to its rich semantic information and external structure. Current knowledge graph recommendation models are insufficient for the learning of user–item interactive behaviors as well as external knowledge, and ignore the noise information in KG. To alleviate these limitations, we present a Collaborative Relation-aware Attention Network with Differentiable sampling (DCRAN) that can fully learn user–item interactions and knowledge graphs, and accurately select recommendation-related entities in KG. Specifically, DCRAN explicitly learns the embeddings of users and items and encodes them into collaborative interaction signals, which are used as guidance signals to extract knowledge. Furthermore, DCRAN emphasizes the importance of KG relations and constructs a relation-aware attention network to learn the representation of items in KG. Most importantly, DCRAN applies a differentiable sampling strategy on entities, which can reduce triplets that have a negative effect on recommendation. Experimental results on three public datasets manifest the effectiveness of DCRAN.

Combining Graph Contrastive Embedding and Multi-head Cross-Attention Transfer for Cross-Domain Recommendation

Cross-domain recommendation (CDR) has become an important research direction in the field of recommender systems due to the increasing demand for personalized recommendations across different domains. However, CDR faces multiple challenges, including data sparsity, popularity bias, and long-tail problems. To address these challenges, we propose a novel framework that combines graph contrastive embedding and multi-head cross-attention transfer for cross-domain recommendation, called GCE-MCAT. Specifically, in the pre-training process, we generate more uniform user and item embeddings through contrastive learning, effectively solving the problem of inconsistent data embedding space distribution and recommendation popularity bias. Moreover, we propose a multi-head cross-attention transfer mechanism that allows the model to extract user common and specific domain features from multiple perspectives and perform cross-domain bidirectional knowledge transfer. Finally, we propose a cross-domain feature fusion mechanism that dynamically assigns weights to common user features and specific domain features. This enables the model to more effectively learn common user interests. We evaluate the proposed framework on three real-world CDR datasets and show that GCE-MCAT consistently and significantly improves recommendation performance compared to state-of-the-art methods. In particular, the proposed framework has demonstrated remarkable effectiveness in addressing long-tail distribution and enhancing recommendation novelty, providing users with more diversified recommendations and reducing popularity bias.

TIEN: Temporal interest-aware evolution model for “Next Item Recommendation”

The “next item recommendation” is used to predict the next item that a user may be interested in and its main challenges is identifying the user's interests. The current state-of-the-art methods mostly adopt time-series-based recommendation methods but ignore the role of time intervals in the interest evolution process. This motivated us to propose a new Time Interest-aware Evolutionary Network model (TIEN) to predict the next item that a user may be interested in. TIEN highlights the role of time intervals in the interest evolution process and divides interests into long-term interests and short-term interests. In TIEN: (1) We devise a time interval-based attention network TAN to capture the user's interests, which employs item embeddings, behavior sequence embeddings, and time intervals to calculate the user's interest attention; (2) We devise an ATT-GRU network based on candidate activation vectors to capture the interest evolution process, which skillfully leverages the reset gate, hidden layer state, item information, time intervals, etc. to participate in interest evolution; (3) We introduce a dual-network collaborative training mechanism, where the obtained interests of users are treated as weights to guide the interest evolution process. Experimental results on multiple benchmark datasets show that TIEN has higher performance than the current state-of-the-art methods.

Exploring global information for session-based recommendation

Session-based recommendation (SBR) aims to recommend items based on anonymous behavior sequences. However, most existing SBR approaches focus solely on the current session while neglecting the item-transition information from other sessions, which suffer from the inability of modeling the complicated item-transition. To address the limitations, we introduce global item-transition information to augment the modeling of item-transitions. Specifically, we first propose a basic GNN-based framework (BGNN), which solely uses session-level item-transition information. Based on BGNN, we propose a novel approach, called Session-based Recommendation with Global Information (SRGI), which infers the user preferences via fully exploring item-transitions over all sessions from two different perspectives: (i) Fusion-based Model (SRGI-FM), which recursively incorporates the neighbor embeddings of each node on global graph into the learning process of item representation; and (ii) Constrained-based Model (SRGI-CM), which treats the global-level information as a constraint to ensure the learned item embeddings are consistent with the global item-transition. Extensive experiments conducted on three popular benchmark datasets demonstrate that both SRGI-FM and SRGI-CM outperform the state-of-the-art methods.

Understanding Diversity in Session-based Recommendation

Current session-based recommender systems (SBRSs) mainly focus on maximizing recommendation accuracy, while few studies have been devoted to improve diversity beyond accuracy. Meanwhile, it is unclear how the accuracy-oriented SBRSs perform in terms of diversity. In addition, the asserted “tradeoff” relationship between accuracy and diversity has been increasingly questioned in the literature. Toward the aforementioned issues, we conduct a holistic study to particularly examine the recommendation performance of representative SBRSs w.r.t. both accuracy and diversity, striving for better understanding of the diversity-related issues for SBRSs and providing guidance on designing diversified SBRSs. Particularly, for a fair and thorough comparison, we deliberately select state-of-the-art non-neural, deep neural, and diversified SBRSs by covering more scenarios with appropriate experimental setups, e.g., representative datasets, evaluation metrics, and hyper-parameter optimization technique. The source code can be obtained via github.com/qyin863/Understanding-Diversity-in-SBRSs . Our empirical results unveil that (1) non-diversified methods can also obtain satisfying performance on diversity, which can even surpass diversified ones, and (2) the relationship between accuracy and diversity is quite complex. Besides the “tradeoff” relationship, they can be positively correlated with each other, that is, having a same-trend (win–win or lose–lose) relationship, which varies across different methods and datasets. Additionally, we further identify three possible influential factors on diversity in SBRSs (i.e., granularity of item categorization, session diversity of datasets, and length of recommendation lists) and offer an intuitive guideline and a potential solution regarding learned item embeddings for more effective session-based recommendation.

PVE: A log parsing method based on VAE using embedding vectors

Log parsing is a critical task that converts unstructured raw logs into structured data for downstream tasks. Existing methods often rely on manual string-matching rules to extract template tokens, leading to lower adaptability on different log datasets. To address this issue, we propose an automated log parsing method, PVE, which leverages Variational Auto-Encoder (VAE) to build a semi-supervised model for categorizing log tokens. Inspired by the observation that log template tokens often consist of words, we choose common words and their combinations to serve as training data to enhance the diversity of structure features of template tokens. Specifically, PVE constructs two types of embedding vectors, the sum embedding and the n-gram embedding, for each word and word combination. The structure features of template tokens can be learned by training VAE on these embeddings. PVE categorizes a token as a template token if it is similar to the training data when log parsing. To improve efficiency, we use the average similarity between token embedding and VAE samples to determine the token type, rather than the reconstruction error. Evaluations on 16 real-world log datasets demonstrate that our method has an average accuracy of 0.878, which outperforms comparison methods in terms of parsing accuracy and adaptability.

Heterogeneous Hypergraph Neural Network for Social Recommendation using Attention Network

Graph neural networks (GNNs) have been used extensively as a backbone for social recommendation. However, their application to a diverse range of situations is still rather limited. This is because graph structures only leverage pairwise user relationships. They cannot capture the higher-order relationships so common in the real world, and ignoring the interest friends and strangers might have in similar items is severely hampering the expressiveness of the current graph-based recommendation models. Hence, in this paper, we outline a heterogeneous hypergraph neural network for social recommendation called HHGSA that incorporates an attention network to address these issues. The hypergraph is able to represent higher-order relationships through five motifs: friend and stranger item appeal, item similarity, user similarity based on interactions with items, and social relations. Two modules, the attentive vertex aggregation module, and the attentive hyperedge aggregation module, capture user and item attention. In addition, it has been discovered that similar items have identical appeal when displayed to users. A GNN aggregates the user embedding data, including information about the friend and stranger and item embeddings. Finally, information about users and items is aggregated for social recommendations. Extensive experiments on four datasets demonstrate that the HHGSA model outperforms a wide range of baselines and can significantly improve the accuracy of recommendations.

A Lightweight Method of Knowledge Graph Convolution Network for Collaborative Filtering

In recent years, knowledge-aware recommendation systems have gained popularity as a solution to address the challenges of data sparsity and cold start in collaborative filtering. However, traditional knowledge graph convolutional networks impose significant computational burdens during training, demanding substantial resources and increasing the cost of recommendations. To address this issue, this article proposes a lightweight knowledge graph convolutional network for collaborative filtering (LKGCF). LKGCF eliminates the feature transformation and nonlinear activation components, by focusing on essential elements such as neighborhood aggregation and layer combination. LKGCF captures the user's long-distance personalized interests on the knowledge graph by sampling from neighborhood information and constructing a weighted sum of item embeddings. Experimental results demonstrate that the proposed model is easy to train and implement due to its coherence and simplicity. Furthermore, notable improvements in recommendation performance are observed compared to strong baselines.

Predict, pretrained, select and answer: Interpretable and scalable complex question answering over knowledge bases

Complex question answering (CQA) over knowledge bases(KB) is a challenging task that has attracted increasing attention in recent years. Semantic parsing-based methods face challenges such as poor adaptability for incomplete KB, large search spaces, and high costs to label logic forms. The gap between knowledge graph representations and question token embeddings leads to poor generalizability and uninterpretable reasoning of information retrieval-based methods. We propose an interpretable and scalable system called Predict, Pretrained, Select and Answer (PPSA) to solve CQA tasks over KB. Our system first trains a language model to predict the reasoning paths required to answer questions. We select only the entities that the predicted reasoning paths pass through in the knowledge graph as candidate entities to reduce the amount of distracting information. The paths that connect the topic entity and the selected candidate entity along with the question are then fed into another language model for answer prediction. The answer prediction module loads the parameters of the trained path prediction module before training to improve accuracy. The system reduces the search space by predicting the path and does not need expensive logic forms annotation. The textual path is the input to the language model, which bridges the gap between the graph representations and token embeddings. We analyse the system reasoning ability over knowledge graphs with different degrees of sparseness, and evaluate the system generalizability. The results of experiments performed with the Complex WebQuestions and WebQuestionsSP datasets demonstrate the effectiveness of our approach for CQA task.

Enhancing interaction representation for joint entity and relation extraction

Jointly extracting entities and their relations from texts is an important task in information extraction. Despite the great success, traditional models suffer from two problems. First, the same token embeddings are shared in two subtasks. It ignores the difference between semantic granularities, in which named entities are more dependent on local features and relations are semantic expressions relevant a whole sentence. Second, the interaction between two subtasks is rarely considered, which is important to encode semantic dependencies relevant to two named entities. To address the above problems, we presents a novel joint entity and relation extraction model. It constructs two independent token embedding modules for encoding features about entities and relations respectively. It enables to encode semantic representations with different granularities for named entities and entity relations. Then, a cross-attention is used to capture the interaction between two subtasks for learning semantic dependencies in a relation instance. The experimental results demonstrate that our model outperforms previous state-of-art models on several public datasets. Extensive additional experiments further confirm the effectiveness of our model.

PLASMe: a tool to identify PLASMid contigs from short-read assemblies using transformer.

Plasmids are mobile genetic elements that carry important accessory genes. Cataloging plasmids is a fundamental step to elucidate their roles in promoting horizontal gene transfer between bacteria. Next generation sequencing (NGS) is the main source for discovering new plasmids today. However, NGS assembly programs tend to return contigs, making plasmid detection difficult. This problem is particularly grave for metagenomic assemblies, which contain short contigs of heterogeneous origins. Available tools for plasmid contig detection still suffer from some limitations. In particular, alignment-based tools tend to miss diverged plasmids while learning-based tools often have lower precision. In this work, we develop a plasmid detection tool PLASMe that capitalizes on the strength of alignment and learning-based methods. Closely related plasmids can be easily identified using the alignment component in PLASMe while diverged plasmids can be predicted using order-specific Transformer models. By encoding plasmid sequences as a language defined on the protein cluster-based token set, Transformer can learn the importance of proteins and their correlation through positionally token embedding and the attention mechanism. We compared PLASMe and other tools on detecting complete plasmids, plasmid contigs, and contigs assembled from CAMI2 simulated data. PLASMe achieved the highest F1-score. After validating PLASMe on data with known labels, we also tested it on real metagenomic and plasmidome data. The examination of some commonly used marker genes shows that PLASMe exhibits more reliable performance than other tools.

Cold-start item recommendation for representation learning based on heterogeneous information networks with fusion side information

In recommendation systems, the cold-start problem refers to the situation that users or items have too few interaction events with the system to be recommended, such as clicks or ratings. The cold-start problem leads to the loss of new users and difficulty in recommending new items and weakens the performance of recommendations. In this paper, we focus on alleviating the cold-start problem of items. Previous work has used the structure of item homogeneous graphs to establish connections between items and generate item embeddings. However, the homogeneous graph of a single node cannot fully explore the potential structural features between users and goods. Due to the flexibility of heterogeneous data, recommendation systems can also employ the Heterogeneous Information Network(HIN) to characterize complex, heterogeneous data. In this paper, a novel Heterogeneous Graph Embedding Learning based Skip-Gram (HSG) model is proposed to mitigate the cold-start problem. First, we introduce three different types of subnetworks and combine them to construct heterogeneous graphs. Then meaningful sequences of nodes are generated via meta-path-based random walk for network embedding. Finally, a Skip-Gram model incorporating side information is used to learn item representations. The learned node embeddings are used as pre-training for the Matrix Factorization (MF) model. The HSG model is proven to be effective by extensive experiments on four publicly available datasets.

Cold-Start Next-Item Recommendation by User-Item Matching and Auto-Encoders

Recommendation systems provide personalized service to users and aim at suggesting to them items that they may prefer. There is an increasing requirement of next-item recommendation systems to infer a user's next favor item based on his/her historical selection of items. In this paper, we study the next-item recommendation under the cold-start situation, where the users in the system share no interaction with the new items. Specifically, we seek to address the problem from the perspective of zero-shot learning (ZSL), which classifies samples whose classes are unseen during training. To this end, we crystallize the relationship and setting from ZSL to cold-start next-item recommendation, and further propose a novel model called User-Item Matching and Auto-encoders (UIMA) which learns the latent embeddings for both users and items by exploiting user historical preferences and item attributes. Concretely, UIMA consists of three components, i.e., two auto-encoders for learning user and item embeddings and a matching network to explore the relationship between the learned user and item embeddings. We perform experiments on several cold-start next-item recommendation datasets, including movies, music, and bookmarks. Promising results demonstrate the effectiveness of the proposed method for cold-start next-item recommendation.

Untargeted Attack against Federated Recommendation Systems via Poisonous Item Embeddings and the Defense

Federated recommendation (FedRec) can train personalized recommenders without collecting user data, but the decentralized nature makes it susceptible to poisoning attacks. Most previous studies focus on the targeted attack to promote certain items, while the untargeted attack that aims to degrade the overall performance of the FedRec system remains less explored. In fact, untargeted attacks can disrupt the user experience and bring severe financial loss to the service provider. However, existing untargeted attack methods are either inapplicable or ineffective against FedRec systems. In this paper, we delve into the untargeted attack and its defense for FedRec systems. (i) We propose ClusterAttack, a novel untargeted attack method. It uploads poisonous gradients that converge the item embeddings into several dense clusters, which make the recommender generate similar scores for these items in the same cluster and perturb the ranking order. (ii) We propose a uniformity-based defense mechanism (UNION) to protect FedRec systems from such attacks. We design a contrastive learning task that regularizes the item embeddings toward a uniform distribution. Then the server filters out these malicious gradients by estimating the uniformity of updated item embeddings. Experiments on two public datasets show that ClusterAttack can effectively degrade the performance of FedRec systems while circumventing many defense methods, and UNION can improve the resistance of the system against various untargeted attacks, including our ClusterAttack.