Item Embeddings Research Articles

Dynamic intent-aware iterative denoising network for session-based recommendation

Session-based recommendation aims to predict items that a user will interact with based on historical behaviors in anonymous sessions. It has long faced two challenges: (1) the dynamic change of user intents which makes user preferences towards items change over time; (2) the uncertainty of user behaviors which adds noise to hinder precise preference learning. They jointly preclude recommender system from capturing real intents of users. Existing methods have not properly solved these problems since they either ignore many useful factors like the temporal information when building item embeddings, or do not explicitly filter out noisy clicks in sessions. To tackle above issues, we propose a novel Dynamic Intent-aware Iterative Denoising Network (DIDN) for session-based recommendation. Specifically, to model the dynamic intents of users, we present a dynamic intent-aware module that incorporates item-aware, user-aware and temporal-aware information to learn dynamic item embeddings. A novel iterative denoising module is then devised to explicitly filter out noisy clicks within a session. In addition, we mine collaborative information to further enrich the session semantics. Extensive experimental results on three real-world datasets demonstrate the effectiveness of the proposed DIDN. Specifically, DIDN obtains improvements over the best baselines by 1.66%, 1.75%, and 7.76% in terms of P@20 and 1.70%, 2.20%, and 10.48% in terms of MRR@20 on all datasets.

Logit Averaging: Capturing Global Relation for Session-Based Recommendation

Session-based recommendation predicts an anonymous user’s next action, whether she or he is likely to purchase based on the user’s behavior in the current session as sequences. Most recent research on session-based recommendations makes predictions based on a single-session without incorporating global relationships between sessions. It does not guarantee a better performance because item embeddings learned by solely utilizing a single session (inter-session) have less item transition information than utilizing both intra- and inter-session ones. Some existing methods tried to enhance recommendation performance by adopting memory modules and global transition graphs; however, those need more computation cost and time. We propose a novel algorithm called Logit Averaging (LA), utilizing both (i) local-level logits, which come from intra-sessions item transitions and (ii) global-level logits, which come from gathered logits of related sessions. The proposed method shows an improvement in recommendation performance in respect of accuracy and diversity through extensive experiments.

Collaborative Graph Learning for Session-based Recommendation

Session-based recommendation (SBR) , which mainly relies on a user’s limited interactions with items to generate recommendations, is a widely investigated task. Existing methods often apply RNNs or GNNs to model user’s sequential behavior or transition relationship between items to capture her current preference. For training such models, the supervision signals are merely generated from the sequential interactions inside a session, neglecting the correlations of different sessions, which we argue can provide additional supervisions for learning the item representations. Moreover, previous methods mainly adopt the cross-entropy loss for training, where the user’s ground truth preference distribution towards items is regarded as a one-hot vector of the target item, easily making the network over-confident and leading to a serious overfitting problem. Thus, in this article, we propose a Collaborative Graph Learning (CGL) approach for session-based recommendation. CGL first applies the Gated Graph Neural Networks (GGNNs) to learn item embeddings and then is trained by considering both the main supervision as well as the self-supervision signals simultaneously. The main supervisions are produced by the sequential order while the self-supervisions are derived from the global graph constructed by all sessions. In addition, to prevent overfitting, we propose a Target-aware Label Confusion (TLC) learning method in the main supervised component. Extensive experiments are conducted on three publicly available datasets, i.e., Retailrocket, Diginetica, and Gowalla. The experimental results show that CGL can outperform the state-of-the-art baselines in terms of Recall and MRR.

Contrastive Graph Learning for Social Recommendation

Owing to the strength in learning representation of the high-order connectivity of graph neural networks (GNN), GNN-based collaborative filtering has been widely adopted in recommender systems. Furthermore, to overcome the data sparsity problem, some recent GNN-based models attempt to incorporate social information and to design contrastive learning as an auxiliary task to assist the primary recommendation task. Existing GNN and contrastive-learning-based recommendation models learn user and item representations in a symmetrical way and utilize social information and contrastive learning in a complex manner. The above two strategies lead to these models being either ineffective for datasets with a serious imbalance between users and items or inefficient for datasets with too many users and items. In this work, we propose a contrastive graph learning (CGL) model, which combines social information and contrastive learning in a simple and powerful way. CGL consists of three modules: diffusion, readout, and prediction. The diffusion module recursively aggregates and integrates social information and interest information to learn representations of users and items. The readout module takes the average value of user embeddings from all diffusion layers and item embeddings at the last diffusion layer as readouts of users and items, respectively. The prediction module calculates prediction rating scores with an interest graph to emphasize interest information. Three different losses are designed to ensure the function of each module. Extensive experiments on three benchmark datasets are implemented to validate the effectiveness of our model.

A hierarchical recommendation system for E-commerce using online user reviews

• We propose a hierarchical recommendation system for e-commerce domain that uses online user reviews. • Our recommendation system uses not only item title and description, but also item’s reviews when generating item embedding vectors. • We provide an analysis of using all item reviews and randomly selected item reviews when making a recommendation. • We provide a transparent recommendation system that capable of explaining all recommendations. Recommendation systems are considered as one of the important components of e-commerce platforms due to their direct impact on profitability. In this study, we propose a hierarchical recommendation system to increase the performance of the e-commerce recommendation system. Our DeepIDRS approach has a two-level hierarchical structure: (1) The first level uses bidirectional encoder representations to represent textual information of an item (title, description, and a subset of item reviews), efficiently and accurately; (2) The second level is an attention-based sequential recommendation model that uses item embeddings derived from the first level of the hierarchical structure. Furthermore, we compare our approach DeepIDRS with various approaches from different perspectives. Our results in the real-world dataset show that DeepIDRS provides at least 10% better HR@10 and NCCG@10 performance than other review-based models. With this study, for e-commerce, we clearly show that a hierarchical, explainable recommendation system that accurately represents the item title, description, and a subset of item reviews, improves performance.

Item Concept Network: Towards Concept-Based Item Representation Learning

Item concept modeling is commonly achieved by leveraging textual information. However, many existing models do not leverage the inferential property of concepts to capture word meanings, which therefore ignores the relatedness between correlated concepts, a phenomenon which we term conceptual “correlation sparsity.” In this paper, we distinguish between word modeling and concept modeling and propose an item concept modeling framework centering around the item concept network (ICN). ICN models and further enriches item concepts by leveraging the inferential property of concepts and thus addresses the correlation sparsity issue. Specifically, there are two stages in the proposed framework: ICN construction and embedding learning. In the first stage, we propose a generalized network construction method to build ICN, a structured network which infers expanded concepts for items via matrix operations. The second stage leverages neighborhood proximity to learn item and concept embeddings. With the proposed ICN, the resulting embedding facilitates both homogeneous and heterogeneous tasks, such as item-to-item and concept-to-item retrieval, and delivers related results which are more diverse than traditional keyword-matching-based approaches. As our experiments on two real-world datasets show, the framework encodes useful conceptual information and thus outperforms traditional methods in various item classification and retrieval tasks.

Predicting Dynamic User–Item Interaction with Meta-Path Guided Recursive RNN

Accurately predicting user–item interactions is critically important in many real applications, including recommender systems and user behavior analysis in social networks. One major drawback of existing studies is that they generally directly analyze the sparse user–item interaction data without considering their semantic correlations and the structural information hidden in the data. Another limitation is that existing approaches usually embed the users and items into the different embedding spaces in a static way, but ignore the dynamic characteristics of both users and items. In this paper, we propose to learn the dynamic embedding vector trajectories rather than the static embedding vectors for users and items simultaneously. A Metapath-guided Recursive RNN based Shift embedding method named MRRNN-S is proposed to learn the continuously evolving embeddings of users and items for more accurately predicting their future interactions. The proposed MRRNN-S is extended from our previous model RRNN-S which was proposed in the earlier work. Comparedwith RRNN-S, we add the word2vec module and the skip-gram-based meta-path module to better capture the rich auxiliary information from the user–item interaction data. Specifically, we first regard the interaction data of each user with items as sentence data to model their semantic and sequential information and construct the user–item interaction graph. Then we sample the instances of meta-paths to capture the heterogeneity and structural information from the user–item interaction graph. A recursive RNN is proposed to iteratively and mutually learn the dynamic user and item embeddings in the same latent space based on their historical interactions. Next, a shift embedding module is proposed to predict the future user embeddings. To predict which item a user will interact with, we output the item embedding instead of the pairwise interaction probability between users and items, which is much more efficient. Through extensive experiments on three real-world datasets, we demonstrate that MRRNN-S achieves superior performance by extensive comparison with state-of-the-art baseline models.

SGCF: Inductive Movie Recommendation System with Strongly Connected Neighborhood Sampling

User and item embeddings are key resources for the development of recommender systems. Recent works has exploited connectivity between users and items in graphs to incorporate the preferences of local neighborhoods into embeddings. Information inferred from graph connections is very useful, especially when interaction between user and item is sparse. In this paper, we propose graphSAGE Collaborative Filtering (SGCF), an inductive graph-based recommendation system with local sampling weight. We conducted an experiment to investigate recommendation performance for SGCF by comparing its performance with baseline and several SGCF variants in Movielens dataset, which are commonly used as recommendation system benchmark data. Our experiment shows that weighted SGCF perform 0.5% higher than benchmark in NDCG@5 and NDCG@10, and 0.8% in NDCG@100. Weighted SGCF perform 0.79% higher than benchmark in recall@5, 0.4% increase for recall@10 and 1.85% increase for recall@100. All the improvements are statistically significant with p-value 0.05.

CKGAT: Collaborative Knowledge-Aware Graph Attention Network for Top-N Recommendation

Knowledge graph-based recommendation methods are a hot research topic in the field of recommender systems in recent years. As a mainstream knowledge graph-based recommendation method, the propagation-based recommendation method captures users’ potential interests in items by integrating the representations of entities and relations in the knowledge graph and the high-order connection patterns between entities to provide personalized recommendations. For example, the collaborative knowledge-aware attentive network (CKAN) is a typical state-of-the-art propagation-based recommendation method that combines user-item interactions and knowledge associations in the knowledge graph, and performs heterogeneous propagation in the knowledge graph to generate multi-hop ripple sets, thereby capturing users’ potential interests. However, existing propagation-based recommendation methods, including CKAN, usually ignore the complex relations between entities in the multi-hop ripple sets and do not distinguish the importance of different ripple sets, resulting in inaccurate user potential interests being captured. Therefore, this paper proposes a top-N recommendation method named collaborative knowledge-aware graph attention network (CKGAT). Based on the heterogeneous propagation strategy, CKGAT uses the knowledge-aware graph attention network to extract the topological proximity structures of entities in the multi-hop ripple sets and then learn high-order entity representations, thereby generating refined ripple set embeddings. CKGAT further uses an attention aggregator to perform weighted aggregation on the ripple set embeddings, the user/item initial entity set embeddings, and the original representations of items to generate accurate user embeddings and item embeddings for the top-N recommendations. Experimental results show that CKGAT, overall, outperforms three baseline methods and six state-of-the-art propagation-based recommendation methods in terms of recommendation accuracy, and outperforms four representative propagation-based recommendation methods in terms of recommendation diversity.

Augment BERT with average pooling layer for Chinese summary generation

The BERT pre-trained language model has achieved good results in various subtasks of natural language processing, but its performance in generating Chinese summaries is not ideal. The most intuitive reason is that the BERT model is based on character-level composition, while the Chinese language is mostly in the form of phrases. Directly fine-tuning the BERT model cannot achieve the expected effect. This paper proposes a novel summary generation model with BERT augmented by the pooling layer. In our model, we perform an average pooling operation on token embedding to improve the model’s ability to capture phrase-level semantic information. We use LCSTS and NLPCC2017 to verify our proposed method. Experimental data shows that the average pooling model’s introduction can effectively improve the generated summary quality. Furthermore, different data needs to be set with varying pooling kernel sizes to achieve the best results through comparative analysis. In addition, our proposed method has strong generalizability. It can be applied not only to the task of generating summaries, but also to other natural language processing tasks.

KLGCN: Knowledge graph-aware Light Graph Convolutional Network for recommender systems

Most popular recommender systems learn the embedding of users and items through capturing valuable information from user–item interactions or item knowledge graph (KG) with Graph Convolutional Network. However, only a few methods capture information from both source data, and they introduce many trainable parameters that increase training difficulty. In this work, we aim to aggregate information from both the user–item interaction graph and the item KG in a light and effective manner. We first experimentally verify the effectiveness of removing feature transformation and nonlinear activation in KG-aware recommendation, which has been proven to greatly reduce parameters while improving performance in the collaborative filtering-based recommendation. Then we propose a new Knowledge graph-aware Light Graph Convolutional Network (KLGCN), which can learn partial embeddings of users and items by aggregating features on the source graphs for recommendation and introduces no extra parameters. Extensive experiments on three public datasets demonstrate that KLGCN achieves substantial improvement over several state-of-the-art models and maintains satisfactory performance on cold-start scenarios.

Research on Recommendation Algorithm of Joint Light Graph Convolution Network and DropEdge

Overfitting in a deep neural network leads to low recommendation precision and high loss. To mitigate these issues in a deep neural network-based recommendation algorithm, we propose a recommendation algorithm, LG-DropEdge, joint light graph convolutional network, and the DropEdge. First, to reduce the cost of data storage and calculation, we initialize user and item embedding in the embedding layer of the algorithm. Then, to obtain high-order interaction relationships to optimize the embedding representation, we enrich the embedding by injecting high-order connectivity relationships in the convolutional layer. In the training phase, DropEdge is used to randomly discard connected relationships (interaction edges) to prevent overfitting. Finally, to reasonably aggregate the embedding results learned on all layers, the weighted average is expressed as the final embedding, so that users can make preferences in the item. We conduct experiments on three public datasets, using two performance indicators; namely, recall and NDCG, are used for evaluation. For the Gowalla dataset, compared with the optimal baseline method, recall@20 and ndcg@20 increased by 2.53% and 2.39%, respectively. For the Yelp2018 dataset, recall@20 and ndcg@20 increased by 6.17% and 5.58%, respectively. For the Amazon-book dataset, recall@20 and ndcg@20 increased by 4.82% and 4.67%, respectively. The results show that LG-DropEdge can not only reduce the degree of neural network overfitting but also improve the recommended results’ precision.

Graph convolution machine for context-aware recommender system

The latest advance in recommendation shows that better user and item representations can be learned via performing graph convolutions on the user-item interaction graph. However, such finding is mostly restricted to the collaborative filtering (CF) scenario, where the interaction contexts are not available. In this work, we extend the advantages of graph convolutions to context-aware recommender system (CARS, which represents a generic type of models that can handle various side information). We propose \textit{Graph Convolution Machine} (GCM), an end-to-end framework that consists of three components: an encoder, graph convolution (GC) layers, and a decoder. The encoder projects users, items, and contexts into embedding vectors, which are passed to the GC layers that refine user and item embeddings with context-aware graph convolutions on user-item graph. The decoder digests the refined embeddings to output the prediction score by considering the interactions among user, item, and context embeddings. We conduct experiments on three real-world datasets from Yelp and Amazon, validating the effectiveness of GCM and the benefits of performing graph convolutions for CARS. Our implementations are available at \url{https://github.com/wujcan/GCM}.

Transition Information Enhanced Disentangled Graph Neural Networks for Session-Based Recommendation

Session-based recommendation is a practical recommendation task that predicts the next item based on an anonymous behavior sequence, and its performance relies heavily on the transition information between items in the sequence. The SOTA methods in SBR employ GNN to model neighboring item transitions from global (i.e, other sessions) and local (i.e, current session) contexts. However, most existing methods treat neighbors from different sessions equally without considering that the neighbor items from different sessions may share similar features with the target item on different aspects and may have different contributions. In other words, they have not explored finer-granularity transition information between items in the global context, leading to sub-optimal performance. In this paper, we fill this gap by proposing a novel Transition Information Enhanced Disentangled Graph Neural Network (TIE-DGNN) model to capture finer-granular transition information between items and try to interpret the reason of the transition by modeling the various factors of the item. Specifically, we propose a position-aware global graph, which utilizes the relative position information to model the neighboring item transition. Then, we slice item embeddings into blocks, each of which represents a factor, and use disentangling module to separately learn the factor embeddings over the global graph. For local context, we train item embeddings by using attention mechanisms to capture transition information from the current session. To this end, our model considers two levels of transition information. Especially in global text, we not only consider finer-granularity transition information between items but also take user intents at factor-level into account to interpret the key reason for the transition. Extensive experiments on three datasets demonstrate the superiority of our method over the SOTA methods.

An E2E-ASR-Based Iteratively-Trained Timestamp Estimator

Text-to-speech alignment, also known as time alignment, is essential for automatic speech recognition (ASR) systems used for speech retrieval tasks, such as keyword search and speech segment extraction. Previous works have used the Gaussian mixture model-hidden Markov model (GMM-HMM) forced alignment to improve the alignment performance. However, when used with end-to-end (E2E) ASR, GMM-HMM forced alignment causes extra reliance on expertise such as pronunciation lexica. It also increases the system complexity because GMM-HMMs are very dissimilar to E2E models. To tackle these two problems, we propose an E2E-ASR-based iteratively-trained timestamp estimator (ITSE), which performs alignment between token-level transcription and speech. We train ITSE first with coarse initial alignment targets generated using connectionist temporal classification (CTC) posteriors. During training, we iteratively perform realignment to update the targets. We attribute the effectiveness of the iterative training to ITSE’s two vital features. First, ITSE performs alignment using similarities between token and speech embeddings instead of frame-wise token classification posteriors. Second, ITSE uses speech embeddings that are aware of left context rather than global context. ITSE significantly outperforms CTC-based baselines in word alignment accuracy and is comparable to a GMM-HMM forced aligner. In short, ITSE is an accurate, lightweight text-to-speech alignment module implemented without expertise such as pronunciation lexica.

LESSFormer: Local-Enhanced Spectral-Spatial Transformer for Hyperspectral Image Classification

Currently, the convolutional neural networks (CNNs) have become the mainstream methods for hyperspectral image (HSI) classification, due to their powerful ability to extract local features. However, CNNs fail to effectively and efficiently capture the long-range contextual information and diagnostic spectral information of HSI. In contrast, the leading-edge vision transformers are capable of capturing long-range dependencies and processing sequential data such as spectral signatures. Nevertheless, pre-existing transformer-based classification methods generally generate inaccurate token embeddings from a single spectral or spatial dimension of raw HSIs and encounter difficulty modeling locality with insufficient training data. To mitigate these limitations, we propose a novel local-enhanced spectral-spatial transformer method (i.e., LESSFormer) specifically devised for HSI classification. Two effective and efficient modules are designed in LESSFormer, i.e., the HSI2Token module and the local-enhanced transformer encoder. The former is devised to transform HSI into the adaptive spectral-spatial tokens, and the latter is built to further enhance the representation ability of tokens by reinforcing the local information explicitly with a simple attention mask as well as retaining the long-range information in the meantime. Extensive experimental results on the new Xiong’an dataset and the widely used Pavia University and Houston University datasets have shown the superiority of LESSFormer over other state-of-the-art networks.

Effective Convolutional Transformer for Highly Accurate Planetary Gearbox Fault Diagnosis

To extract the global temporal correlations and local features together to enhance the accuracy for fault diagnosis, this paper proposes an effective convolutional Transformer (ECT), which can learn the global temporal correlations using Transformer and local features with convolution at the same time. The proposed method designs a multi-stage hierarchical structure of Transformer, which utilizes convolutional tokenization to distill dominating sequence features from raw vibration signals while increasing the dimension of token embedding across stages at the same time as that in CNNs. The spatial-reduction attention (SRA) and the linear dimension reduction projections are introduced respectively to Transformer at different stages to reduce the resource consumption of the model. Finally, the proposed method utilizes a sequence pooling strategy on the output of Transformer to eliminate the requirement of the class token and make the model accurate for classification. The specially designed structure makes the model flexible and effective for planetary gearbox fault diagnosis. Experiments performed on planetary gearbox fault simulators indicate that the ECT method has significant effectiveness and high accuracy compared with the state-of-the-art methods for planetary gearbox fault diagnosis.

Model-based clinical note entity recognition for rheumatoid arthritis using bidirectional encoder representation from transformers.

Rheumatoid arthritis (RA) is a disease of the immune system with a high rate of disability and there are a large amount of valuable disease diagnosis and treatment information in the clinical note of the electronic medical record. Artificial intelligence methods can be used to mine useful information in clinical notes effectively. This study aimed to develop an effective method to identify and classify medical entities in the clinical notes relating to RA and use the entity identification results in subsequent studies. In this paper, we introduced the bidirectional encoder representation from transformers (BERT) pre-training model to enhance the semantic representation of word vectors. The generated word vectors were then inputted into the model, which is composed of traditional bidirectional long short-term memory neural networks and conditional random field machine learning algorithms for the named entity recognition of clinical notes to improve the model's effectiveness. The BERT method takes the combination of token embeddings, segment embeddings, and position embeddings as the model input and fine-tunes the model during training. Compared with the traditional Word2vec word vector model, the performance of the BERT pre-training model to obtain a word vector as model input was significantly improved. The best F1-score of the named entity recognition task after training using many rheumatoid arthritis clinical notes was 0.936. This paper confirms the effectiveness of using an advanced artificial intelligence method to carry out named entity recognition tasks on a corpus of a large number of clinical notes; this application is promising in the medical setting. Moreover, the extraction of results in this study provides a lot of basic data for subsequent tasks, including relation extraction, medical knowledge graph construction, and disease reasoning.

Refining Graph Representation for Cross-Domain Recommendation Based on Edge Pruning in Latent Space

In this paper, we present refining graph representation for cross-domain recommendation (CDR) based on edge pruning considering feature distribution in a latent space. Conventional graph-based CDR methods have utilized all ratings and purchase histories of user’s products. However, some items purchased by users are not related to the domain for recommendation, and this information becomes noise when making CDR. So, the proposed method introduces edge pruning into the latest graph-based CDR method to refine graph representation. To compare the item embedding features calculated in different domains, we construct a latent space and perform edge pruning through their correlations. Additionally, we introduce a state-of-the-art graph neural network into the graph construction of the proposed method that considers the interactions between users and items thereby obtaining effective embedding features in a domain. This makes it possible to consider domain-specific user preferences and estimate embedding features with high-expressive power. Furthermore, to compare the embedding features of items in the two domains, we construct their latent spaces and project them. Edge pruning is performed using the correlation of items between the two domains on the latent space. We obtain cross domain specific graph representation through edge pruning, which improves the performance by considering the relationship between both items across domains. To the best of our knowledge, no study in the CDR field focuses on eliminating unnecessary node information. We have demonstrated the effectiveness of the proposed method by comparing several graph-based state-of-the-art methods.

Region or Global A Principle for Negative Sampling in Graph-based Recommendation

Graph-based recommendation systems are blossoming recently, which models user-item interactions as a user-item graph and utilizes graph neural networks (GNNs) to learn the embeddings for users and items. A fundamental challenge of graph-based recommendation is that there only exists observed positive user-item pairs in the user-item graph. Negative sampling is a vital technique to solve the one-class problem and is widely used in many recommendation methods. However, the previous works only focus on the design of negative sampling distribution but ignore the sampled region for negative sampling. In this work, we propose the Three-Region Principle to guide negative sampling, which suggests that we should negatively sample more items at an intermediate region and less adjacent and distant items. In light of this principle, we present the RecNS method, which is a general negative sampling method designed with two sampling strategies: positive-assisted sampling and exposure-augmented sampling. Instead of sampling existing negative items from graph data, we merge these two strategies in embedding space to generate negative item embeddings. Extensive experiments demonstrate that our proposed RecNS method significantly outperforms all negative sampling baselines, e.g., 10.47% for PinSage, 6.02% for NGCF, and 8.20% for LightGCN in terms of Recall@20 on the Alibaba dataset.