Item Embeddings Research Articles

Knowledge graph enhanced neural collaborative recommendation

Existing neural collaborative filtering (NCF) recommendation methods suffer from severe sparsity problem. Knowledge Graph (KG), which commonly consists of fruitful connected facts about items, presents an unprecedented opportunity to alleviate the sparsity problem. However, pure NCF models can hardly model the high-order connectivity in KG, and ignores complex pairwise correlations between user/item embedding dimensions.To address these problems, we propose a novel Knowledge graph enhanced Neural Collaborative Recommendation (K-NCR) framework, which effectively combines user–item interaction information and auxiliary knowledge information for recommendation task into three parts: (1) For items, the proposed propagating model learns the representation of item entity. It recursively aggregates information from its multi-hop neighbours in KG, and employs an attention mechanism to discriminate the importance of the relation type to mine users’ potential preferences. (2) For users, another heterogeneous attention weights are leveraged to strengthen the embedding learning of users. (3) The user and item embeddings are then fed into a newly designed two-dimensional interaction map with convolutional hidden layers to model the complex pairwise correlations between their embedding dimensions explicitly. Extensive experimental results on three benchmark datasets demonstrate the effectiveness of our K-NCR framework.

Deep Bi-LSTM Networks for Sequential Recommendation

Recent years have seen a surge in approaches that combine deep learning and recommendation systems to capture user preference or item interaction evolution over time. However, the most related work only consider the sequential similarity between the items and neglects the item content feature information and the impact difference of interacted items on the next items. This paper introduces the deep bidirectional long short-term memory (LSTM) and self-attention mechanism into the sequential recommender while fusing the information of item sequences and contents. Specifically, we deal with the issues in a three-pronged attack: the improved item embedding, weight update, and the deep bidirectional LSTM preference learning. First, the user-item sequences are embedded into a low-dimensional item vector space representation via Item2vec, and the class label vectors are concatenated for each embedded item vector. Second, the embedded item vectors learn different impact weights of each item to achieve item awareness via self-attention mechanism; the embedded item vectors and corresponding weights are then fed into the bidirectional LSTM model to learn the user preference vectors. Finally, the top similar items in the preference vector space are evaluated to generate the recommendation list for users. By conducting comprehensive experiments, we demonstrate that our model outperforms the traditional recommendation algorithms on Recall@20 and Mean Reciprocal Rank (MRR@20).

Simple and effective neural-free soft-cluster embeddings for item cold-start recommendations

Recommender systems are widely used in online platforms for easy exploration of personalized content. The best available recommendation algorithms are based on using the observed preference information among collaborating entities. A significant challenge in recommender system continues to be item cold-start recommendation: how to effectively recommend items with no observed or past preference information. Here we propose a two-stage algorithm based on soft clustering to provide an efficient solution to this problem. The crux of our approach lies in representing the items as soft-cluster embeddings in the space spanned by the side-information associated with the items. Though many item embedding approaches have been proposed for item cold-start recommendations in the past—and simple as they might appear—to the best of our knowledge, the approach based on soft-cluster embeddings has not been proposed in the research literature. Our experimental results on four benchmark datasets conclusively demonstrate that the proposed algorithm makes accurate recommendations in item cold-start settings compared to the state-of-the-art algorithms according to commonly used ranking metrics like Normalized Discounted Cumulative Gain (NDCG) and Mean Average Precision (MAP). The performance of our proposed algorithm on the MovieLens 20M dataset clearly demonstrates the scalability aspect of our algorithm compared to other popular algorithms. We also propose the metric Cold Items Precision (CIP) to quantify the ability of a system to recommend cold-start items. CIP can be used in conjunction with relevance ranking metrics like NDCG and MAP to measure the effectiveness of the cold-start recommendation algorithm.

A graph embedding based model for fine-grained POI recommendation

Point-of-interest (POI) recommendation is an important technique widely used in self-driving services. While POI recommendation aims to recommend unvisited POIs to self-driving users, users always expect their intended items can be suggested together with these POIs, e.g. what activities to perform at the recommended places. However, existing methods cannot well support such POI recommendation in a finer granularity. In this paper, we investigate this new problem and propose a novel POI-based item recommendation model via graph embedding. The model accurately captures the joint effect of geographical and temporal influences on both POI-level and item-level recommendation in a shared space, which can address data sparsity and cold start problems effectively. To optimize the model efficiently and accurately, a novel weighted negative sampling strategy is designed. Besides, we propose a novel fine-grained user dynamic preference modeling method, which can accurately capture dynamic user preferences in a finer granularity based on the embeddings of both POIs and items. Comprehensive experimental studies have been conducted on three datasets. Results show that our model achieves significant improvement over state-of-the-art baselines.

A recommendation algorithm based on fine-grained feature analysis

Most existing recommender methods have insufficient power to capture or recover fine-grained features of products and user preferences. Knowledge graphs contain considerable information about products and mutual relationships in the world. To analyze the fine-grained features during the process of recommendation, we propose a knowledge-aware collaborative learning framework (KACL), which is the first work to combine users’ historical reviews with knowledge graphs. First, we use a named-entity recognition (NER) system to recognize the named entities that correspond to item features in the unstructured reviews. After that, we use an entity-linking (EL) system to map entities which identified in the first step to the corresponding entity in Wikipedia. Next, we constructed a sub-graph that depends on the extracted entity and the related ones and embedded the sub-graph into a low latent vector space as pretrain item embedding through the knowledge graph embedding model TransR. In the next step, we will fine-tune item embedding through deep learning built by 4 levels of relu and combine collaborative information with pretrain item embedding. And in accordance with the user review behavior, the item embedding will be averagely poled to express the user embedding, we will put the item embedding and the user embedding into the classification model. Finally, we use the KACL to integrate collaborative filtering with entity representation and make recommendations. Experiments are performed on real-world datasets to evaluate the effectiveness of our new method. The results show that the analysis of fine-grained features based on a knowledge graph helps our KACL improve recommender accuracy compared with the selected state-of-the-art models.

Towards syntax-aware token embeddings

AbstractDistributional semantic word representations are at the basis of most modern NLP systems. Their usefulness has been proven across various tasks, particularly as inputs to deep learning models. Beyond that, much work investigated fine-tuning the generic word embeddings to leverage linguistic knowledge from large lexical resources. Some work investigated context-dependent word token embeddings motivated by word sense disambiguation, using sequential context and large lexical resources. More recently, acknowledging the need for an in-context representation of words, some work leveraged information derived from language modelling and large amounts of data to induce contextualised representations. In this paper, we investigate Syntax-Aware word Token Embeddings (SATokE) as a way to explicitly encode specific information derived from the linguistic analysis of a sentence in vectors which are input to a deep learning model. We propose an efficient unsupervised learning algorithm based on tensor factorisation for computing these token embeddings given an arbitrary graph of linguistic structure. Applying this method to syntactic dependency structures, we investigate the usefulness of such token representations as part of deep learning models of text understanding. We encode a sentence either by learning embeddings for its tokens and the relations between them from scratch or by leveraging pre-trained relation embeddings to infer token representations. Given sufficient data, the former is slightly more accurate than the latter, yet both provide more informative token embeddings than standard word representations, even when the word representations have been learned on the same type of context from larger corpora (namely pre-trained dependency-based word embeddings). We use a large set of supervised tasks and two major deep learning families of models for sentence understanding to evaluate our proposal. We empirically demonstrate the superiority of the token representations compared to popular distributional representations of words for various sentence and sentence pair classification tasks.

Attention-Based Multimodal Neural Network for Automatic Evaluation of Press Conferences

In the study, a multimodal neural network is proposed to automatically predict the evaluation of a professional consultant team for press conferences using text and audio data. Seven publicly available press conference videos were collected, and all the Q&A pairs between speakers and journalists were annotated by the consultant team. The proposed multimodal neural network consists of a language model, an audio model, and a feature fusion network. The word representation is made up by a token embedding using ELMo and a type embedding. The language model is an LSTM with an attention layer. The audio model is based on a six-layer CNN to extract segmental feature as well as an attention network to measure the importance of each segment. Two approaches of feature fusion are proposed: a shared attention network and the production of text features and audio features. The former can explain the importance between speech content and speaking style. The latter achieved the best performance with the average accuracy of 60.1% for all evaluation criteria.

A multi-attention deep neural network model base on embedding and matrix factorization for recommendation

Matrix factorization is a popular method in recommendation system. However, the quality of recommendation algorithm based on matrix decomposition is greatly affected by the sparsity of rating data. This paper presents a multi-attention deep neural network model base on Embedding and matrix factorization for recommendation. By integrating user / item embedding representation and matrix factorization representation, data sparsity and cold start problems can be effectively alleviated. The scalability of model can be solved by using the deep neural network via GPU technology. The experimental results on real data sets show that, with other classical matrix factorization recommendation algorithms, the proposed algorithm can produce higher prediction results and effectively improve the recommendation quality and performance.

Hierarchical Attentive Transaction Embedding With Intra- and Inter-Transaction Dependencies for Next-Item Recommendation

A transaction-based recommender system (TBRS) aims to predict the next item by modeling dependencies in transactional data. Generally, two kinds of dependencies considered are intra-transaction dependence and inter-transaction dependence. Most existing TBRSs recommend next item by only modeling the intra-transaction dependence within the current transaction while ignoring inter-transaction dependence with recent transactions that may also affect the next item. However, as not all recent transactions are relevant to the current and next items, the relevant ones should be identified and prioritized. In this article, we propose a novel hierarchical attentive transaction embedding (HATE) model to tackle these issues. Specifically, a two-level attention mechanism integrates both item embedding and transaction embedding to build an attentive context representation that incorporates both intra- and inter-transaction dependencies. With the learned context representation, HATE then recommends the next item. Experimental evaluations on two real-world transaction datasets show that HATE significantly outperforms the state-of-the-art methods in terms of recommendation accuracy.

GraphER: Token-Centric Entity Resolution with Graph Convolutional Neural Networks

Entity resolution (ER) aims to identify entity records that refer to the same real-world entity, which is a critical problem in data cleaning and integration. Most of the existing models are attribute-centric, that is, matching entity pairs by comparing similarities of pre-aligned attributes, which require the schemas of records to be identical and are too coarse-grained to capture subtle key information within a single attribute. In this paper, we propose a novel graph-based ER model GraphER. Our model is token-centric: the final matching results are generated by directly aggregating token-level comparison features, in which both the semantic and structural information has been softly embedded into token embeddings by training an Entity Record Graph Convolutional Network (ER-GCN). To the best of our knowledge, our work is the first effort to do token-centric entity resolution with the help of GCN in entity resolution task. Extensive experiments on two real-world datasets demonstrate that our model stably outperforms state-of-the-art models.

Hierarchical Contextualized Representation for Named Entity Recognition

Named entity recognition (NER) models are typically based on the architecture of Bi-directional LSTM (BiLSTM). The constraints of sequential nature and the modeling of single input prevent the full utilization of global information from larger scope, not only in the entire sentence, but also in the entire document (dataset). In this paper, we address these two deficiencies and propose a model augmented with hierarchical contextualized representation: sentence-level representation and document-level representation. In sentence-level, we take different contributions of words in a single sentence into consideration to enhance the sentence representation learned from an independent BiLSTM via label embedding attention mechanism. In document-level, the key-value memory network is adopted to record the document-aware information for each unique word which is sensitive to similarity of context information. Our two-level hierarchical contextualized representations are fused with each input token embedding and corresponding hidden state of BiLSTM, respectively. The experimental results on three benchmark NER datasets (CoNLL-2003 and Ontonotes 5.0 English datasets, CoNLL-2002 Spanish dataset) show that we establish new state-of-the-art results.

CD-VulD: Cross-Domain Vulnerability Discovery Based on Deep Domain Adaptation

A major cause of security incidents such as cyber attacks is rooted in software vulnerabilities. These vulnerabilities should ideally be found and fixed before the code gets deployed. Machine learning-based approaches achieve state-of-the-art performance in capturing vulnerabilities. These methods are predominantly supervised. Their prediction models are trained on a set of ground truth data where the training data and test data are assumed to be drawn from the same probability distribution. However, in practice, the test data often differs from the training data in terms of distribution because they are from different projects or they differ in the types of vulnerability. In this article, we present a new system for <u>C</u>ross <u>D</u>omain Software <u>Vul</u>nerability <u>D</u>iscovery (<i>CD-VulD</i>) using deep learning (DL) and domain adaptation (DA). We employ DL because it has the capacity of automatically constructing high-level abstract feature representations of programs, which are likely of more cross-domain useful than the handcrafted features driven by domain knowledge. The divergence between distributions is reduced by learning cross-domain representations. First, given software program representations, CD-VulD converts them into token sequences and learns the token embeddings for generalization across tokens. Next, CD-VulD employs a deep feature model to build abstract high-level presentations based on those sequences. Then, the metric transfer learning framework (MTLF) technique is employed to learn cross-domain representations by minimizing the distribution divergence between the source domain and the target domain. Finally, the cross-domain representations are used to build a classifier for vulnerability detection. Experimental results show that CD-VulD outperforms the state-of-the-art vulnerability detection approaches by a wide margin. We make the new datasets publicly available so that our work is replicable and can be further improved.

Evaluating and improving the interpretability of item embeddings using item-tag relevance information

Matrix factorization (MF) methods have superior recommendation performance and are flexible to incorporate other side information, but it is hard for humans to interpret the derived latent factors. Recently, the item-item cooccurrence information is exploited to learn item embeddings and enhance the recommendation performance. However, the item-item co-occurrence information, constructed from the sparse and long-tail distributed user-item interaction matrix, is over-estimated for rare items, which could lead to bias in learned item embeddings. In this paper, we seek to evaluate and improve the interpretability of item embeddings by leveraging a dense item-tag relevance matrix. Specifically, we design two metrics to quantitatively evaluate the interpretability of item embeddings from different viewpoints: interpretability of individual dimensions of item embeddings and semantic coherence of local neighborhoods in the latent space. We also propose a tag-informed item embedding (TIE) model that jointly factorizes the user-item interaction matrix, the item-item co-occurrence matrix and the item-tag relevance matrix with shared item embeddings so that different forms of information can co-operate with each other to learn better item embeddings. Experiments on the MovieLens20M dataset demonstrate that compared with other state-of-the-art MF methods, TIE achieves better top-N recommendations, and the relative improvement is larger when the user-item interaction matrix becomes sparser. By leveraging the itemtag relevance information, individual dimensions of item embeddings are more interpretable and local neighborhoods in the latent space are more semantically coherent; the bias in learned item embeddings are also mitigated to some extent.

Major-Minor Long Short-Term Memory for Word-Level Language Model.

Language model (LM) plays an important role in natural language processing (NLP) systems, such as machine translation, speech recognition, learning token embeddings, natural language generation, and text classification. Recently, the multilayer long short-term memory (LSTM) models have been demonstrated to achieve promising performance on word-level language modeling. For each LSTM layer, larger hidden size usually means more diverse semantic features, which enables the LM to perform better. However, we have observed that when a certain LSTM layer reaches a sufficiently large scale, the promotion of overall effect will slow down, as its hidden size increases. In this article, we analyze that an important factor leading to this phenomenon is the high correlation between the newly extended hidden states and the original hidden states, which hinders diverse feature expression of the LSTM. As a result, when the scale is large enough, simply lengthening the LSTM hidden states will cost tremendous extra parameters but has little effect. We propose a simple yet effective improvement on each LSTM layer consisting of a large-scale Major LSTM and a small-scale Minor LSTM to break the high correlation between the two parts of hidden states, which we call Major-Minor LSTMs (MMLSTMs). In experiments, we demonstrate the LM with MMLSTMs surpasses the existing state-of-the-art model on Penn Treebank (PTB) and WikiText-2 (WT2) data sets and outperforms the baseline by 3.3 points in perplexity on WikiText-103 data set without increasing model parameter counts.

Modeling Embedding Dimension Correlations via Convolutional Neural Collaborative Filtering

As the core of recommender systems, collaborative filtering (CF) models the affinity between a user and an item from historical user-item interactions, such as clicks, purchases, and so on. Benefiting from the strong representation power, neural networks have recently revolutionized the recommendation research, setting up a new standard for CF. However, existing neural recommender models do not explicitly consider the correlations among embedding dimensions, making them less effective in modeling the interaction function between users and items. In this work, we emphasize on modeling the correlations among embedding dimensions in neural networks to pursue higher effectiveness for CF. We propose a novel and general neural collaborative filtering framework—namely, ConvNCF, which is featured with two designs: (1) applying outer product on user embedding and item embedding to explicitly model the pairwise correlations between embedding dimensions, and (2) employing convolutional neural network above the outer product to learn the high-order correlations among embedding dimensions. To justify our proposal, we present three instantiations of ConvNCF by using different inputs to represent a user and conduct experiments on two real-world datasets. Extensive results verify the utility of modeling embedding dimension correlations with ConvNCF, which outperforms several competitive CF methods.

TransNFCM: Translation-Based Neural Fashion Compatibility Modeling

Identifying mix-and-match relationships between fashion items is an urgent task in a fashion e-commerce recommender system. It will significantly enhance user experience and satisfaction. However, due to the challenges of inferring the rich yet complicated set of compatibility patterns in a large e-commerce corpus of fashion items, this task is still underexplored. Inspired by the recent advances in multirelational knowledge representation learning and deep neural networks, this paper proposes a novel Translation-based Neural Fashion Compatibility Modeling (TransNFCM) framework, which jointly optimizes fashion item embeddings and category-specific complementary relations in a unified space via an end-to-end learning manner. TransNFCM places items in a unified embedding space where a category-specific relation (category-comp-category) is modeled as a vector translation operating on the embeddings of compatible items from the corresponding categories. By this way, we not only capture the specific notion of compatibility conditioned on a specific pair of complementary categories, but also preserve the global notion of compatibility. We also design a deep fashion item encoder which exploits the complementary characteristic of visual and textual features to represent the fashion products. To the best of our knowledge, this is the first work that uses category-specific complementary relations to model the category-aware compatibility between items in a translation-based embedding space. Extensive experiments demonstrate the effectiveness of TransNFCM over the state-of-the-arts on two real-world datasets.

Session-Based Recommendation with Graph Neural Networks

The problem of session-based recommendation aims to predict user actions based on anonymous sessions. Previous methods model a session as a sequence and estimate user representations besides item representations to make recommendations. Though achieved promising results, they are insufficient to obtain accurate user vectors in sessions and neglect complex transitions of items. To obtain accurate item embedding and take complex transitions of items into account, we propose a novel method, i.e. Session-based Recommendation with Graph Neural Networks, SR-GNN for brevity. In the proposed method, session sequences are modeled as graphstructured data. Based on the session graph, GNN can capture complex transitions of items, which are difficult to be revealed by previous conventional sequential methods. Each session is then represented as the composition of the global preference and the current interest of that session using an attention network. Extensive experiments conducted on two real datasets show that SR-GNN evidently outperforms the state-of-the-art session-based recommendation methods consistently.

Gated and attentive neural collaborative filtering for user generated list recommendation

Recommending user generated lists (e.g., playlists) has become an emerging task in many online systems. Many existing list recommendation methods predict user preferences on lists by aggregating their preferences on individual items, which neglects the list-level information (e.g., list attributes) and thus results in suboptimal performance. This paper proposes a neural network-based solution for user generated list recommendation, which can leverage both item-level information and list-level information to improve performance. Firstly, a representation learning network with attention and gate mechanism is proposed to learn the user embeddings, item embeddings and list embeddings simultaneously. Then, an interaction network is proposed to learn user–item interactions and user–list interactions, in which the two kinds of interactions can share the convolution layers to further improve performance. Experimental studies on two real-world datasets demonstrate that (1) the proposed representation learning network can learn more representative user/item/list embedding than existing methods and (2) the proposed solution can outperform state-of-the-art methods in both item recommendation and list recommendation in terms of accuracy.

DeepRec: A deep neural network approach to recommendation with item embedding and weighted loss function

Traditional collaborative filtering techniques suffer from the data sparsity problem in practice. That is, only a small proportion of all items in the recommender system occur in a user's rated item list. However, in order to retrieve items meeting a user's interest, all possible candidate items should be investigated. To address this problem, this paper proposes a recommendation approach called DeepRec, based on feedforward deep neural network learning with item embedding and weighted loss function. Specifically, item embedding learns numerical vectors for item representation, and weighted loss function balances popularity and novelty of recommended items. Moreover, it introduces two strategies, i.e. sampling by random (Ran-Strategy) and sampling by distribution (Pro-Strategy), to leave one item as output and the remaining as input from each user's historically rated item list. Max-pooling and average-pooling are employed to combine individual item vectors to derive users’ input vectors for feedforward deep neural network learning. Experiments on the App dataset and the Last.fm dataset demonstrate that the proposed DeepRec approach is superior to state-of-the-art techniques in recommending Apps and songs in terms of accuracy and diversity as well as complexity.

Bilateral neural embedding for collaborative filtering-based multimedia recommendation

As one of the most popular and successfully applied recommendation methods, collaborative filtering aims to extract low-dimensional user and item representation from historic user-item interaction matrix. The similarity between the user and item representation vectors in the same space well measures the degree of interest and thus can be directly used for recommendation. This paper proposes to leverage the emerging deep neural language model to solve the collaborative filtering-based multimedia recommendation problem. By applying the standard Word2Vec model on the user-item interaction data, we can obtain the item embedding representation. Based on this, three strategies are introduced to derive the user embedded representation in the same space, which exploit both the user-item interaction and the correlation among items. Experiments on article recommendation application demonstrate that the proposed deep neural language models achieve superior performance than the traditional collaborative filtering methods based on matrix factorization and topic model.