User Representation Learning Research Articles

Keywords-enhanced Contrastive Learning Model for travel recommendation

Travel recommendation aims to infer travel intentions of users by analyzing their historical behaviors on Online Travel Agencies (OTAs). However, crucial keywords in clicked travel product titles, such as destination and itinerary duration, indicating tourists’ intentions, are often overlooked. Additionally, most previous studies only consider stable long-term user interests or temporary short-term user preferences, making the recommendation performance unreliable. To mitigate these constraints, this paper proposes a novel Keywords-enhanced Contrastive Learning Model (KCLM). KCLM simultaneously implements personalized travel recommendation and keywords generation tasks, integrating long-term and short-term user preferences within both tasks. Furthermore, we design two kinds of contrastive learning tasks for better user and travel product representation learning. The preference contrastive learning aims to bridge the gap between long-term and short-term user preferences. The multi-view contrastive learning focuses on modeling the coarse-grained commonality between clicked products and their keywords. Extensive experiments are conducted on two tourism datasets and a large-scale e-commerce dataset. The experimental results demonstrate that KCLM achieves substantial gains in both metrics compared to the best-performing baseline methods. Specifically, HR@20 improved by 5.79%–14.13%, MRR@20 improved by 6.57%–18.50%. Furthermore, to have an intuitive understanding of the keyword generation by the KCLM model, we provide a case study for several randomized examples.

Explicitly Exploiting Implicit User and Item Relations in Graph Convolutional Network (GCN) for Recommendation

Most existing collaborative filtering-based recommender systems rely solely on available user–item interactions for user and item representation learning. Their performance often suffers significantly when interactions are sparse, as limited user and item interactions are insufficient for learning robust representations. To address this issue, recent research has explored additional information between users and items by leveraging the user–item bipartite graph. However, these methods have not fully exploited high-order neighborhood information, primarily using sampled interactions to enrich training data rather than integrating this information directly into representation learning. In this paper, we propose a novel model, EIR-GCN (Embedding Integration with Relational Graph Convolutional Network), which directly incorporates various types of collaborative relations, such as user–user and item–item interactions, into the embedding function for user preference modeling. Specifically, our model employs advanced graph convolutional network (GCN) techniques to integrate user–item, user–user, and item–item relations for comprehensive representation learning. EIR-GCN initially selects the most influential second-order neighbors from the user–item bipartite graph to form user–user and item–item connections. With these enriched connections, a message-passing method is adopted to learn node representations by aggregating messages from directly linked nodes, including first-order item neighbors and selected second-order user neighbors. Extensive experiments on several public datasets demonstrate that EIR-GCN outperforms strong baselines, including recent GCN-based models and those exploiting high-order information. Our results show that EIR-GCN achieves state-of-the-art performance and effectively addresses the sparsity issue, highlighting its robustness and efficacy in recommendation tasks.

NRMG: News Recommendation With Multiview Graph Convolutional Networks

The emergence of a news recommendation system can effectively improve the news reading experience of users. The most important task of the system is how to learn news and user representations accurately. In the process of learning news representations, most of the current research works do not fully utilize the news features, which makes it difficult to learn more comprehensive news representations. Most research work only learns user representations from a single perspective, which may not be sufficient to learn diverse and dynamic user representations. Therefore, we propose a news recommendation system with a multiview graph convolutional network (NRMG). It contains two parts: news representation and user representation. The knowledge–content collaboration network is adapted to learn news representations from news content and entities, while the multiview graph convolutional network (GCN) is utilized to learn user representations from the user’s click history. The advantage of the NRMG system is that we not only expand the available features by constructing a subclass knowledge graph (KG), but also effectively improve the ability of the news recommendation system to accurately learn news and user representations. Experimental results on the real dataset MIcrosoft News Dataset (MIND) show that the NRMG achieves a 2.25% improvement in area under the receiver operating characteristic (ROC) curve (AUC) value compared with state-of-the-art methods.

Tail-STEAK: Improve Friend Recommendation for Tail Users via Self-Training Enhanced Knowledge Distillation

Graph neural networks (GNNs) are commonly employed in collaborative friend recommendation systems. Nevertheless, recent studies reveal a notable performance gap, particularly for users with limited connections, commonly known as tail users, in contrast to their counterparts with abundant connections (head users). Uniformly treating head and tail users poses two challenges for tail user preference learning: (C1) Label Sparsity, as tail users typically possess limited labels; and (C2) Neighborhood Sparsity, where tail users exhibit sparse observable friendships, leading to distinct preference distributions and performance degradation compared to head users. In response to these challenges, we introduce Tail-STEAK, an innovative framework that combines self-training with enhanced knowledge distillation for tail user representation learning. To address(C1), we present Tail-STEAK-base, a two-stage self-training framework. In the first stage, only head users and their accurate connections are utilized for training, while pseudo links are generated for tail users in the second stage. To tackle (C2), we propose two data augmentation-based self-knowledge distillation pretext tasks. These tasks are seamlessly integrated into different stages of Tail-STEAK-base, culminating in the comprehensive Tail-STEAK framework. Extensive experiments, conducted on state-of-the-art GNN-based friend recommendation models, substantiate the efficacy of Tail-STEAK in significantly improving tail user performance. Our code and data are publicly available at https://github.com/antman9914/Tail-STEAK.

Geography-aware Heterogeneous Graph Contrastive Learning for Travel Recommendation

Recommendation system concentrates on quickly matching products to consumer’s needs which plays a major role in improving user experiences and increase conversion rate. Travel recommendation has become a hot topic in both industry and academia with the development of the tourism industry. Nevertheless, the selection of travel products entails careful consideration of various geographical factors, such as departure and destination. Meanwhile, due to the limitation of finance and time, users browse and purchase travel products less frequently than they do for traditional products, which leads to data sparsity problem in representation learning. To solve these challenges, a novel model named GHGCL (short for G eography-aware H eterogeneous G raph C ontrastive L earning) is proposed for recommending travel products. Concretely, we model the travel recommender system as an heterogeneous information network with geographical information, and capture diverse user preferences from local and high-order structures. Especially, we design two kinds of contrastive learning tasks for better user and travel product representation learning. The multi-view contrastive learning aims to bridge the gap between network schema and meta-path view representations. The meta-path contrastive learning focuses on modeling the coarse-grained commonality between different meta-paths from the perspective of different geographical factors, i.e., departure and destination. We assess the performance of GHGCL by performing a series of experiments on a real-world dataset and the results clearly verify its superiority as compared to the baseline methods.

On Mining User-Item Interactions via Knowledge Graph for Recommendation

Introducing Knowledge Graph (KG) to facilitate recommender system has become a tendency in recent years. Many existing methods leverage KG to obtain side information of items to promote item representation learning for enhancing recommendation performance. However, they ignore that KG also may contribute to better user representation learning. To solve the above issue, we propose a novel algorithm, KIGR ( K nowledge-aware I nteraction G raph for R ecommendation), to mine user-item interactions via Knowledge Graph for assisting user representation learning. Specifically, a user-item interaction is encoded by attentively summing up the relation embedding about the item in KG. Then an unsupervised learning method is used to group the user-item interactions into different latent types. Further, a user-item interaction graph is divided into several subgraphs, which is referred to as Knowledge-aware Interaction Graph, making each subgraph only contains one latent type of interactions. Finally, user representation is the fusion of user interest embedding, which is learned on knowledge-aware interaction graph; While item representation is learned on KG. Experimental results on MovieLens, LastFM and Amazon-Book validate that the proposed KIGR has a superior performance compared with the SOTA algorithms.

Dynamic Multi-view Group Preference Learning for group behavior prediction in social networks

Group behavior modeling is an important research topic in the field of social network analysis. Existing methods regarding this topic can only learn the static group preference, ignoring the multiple characteristics of the group behavior, so they cannot model the group behavior in a complete way. In this paper, we propose a Dynamic Multi-view Group Preference Learning (DMGPL) model for group behavior prediction in social networks. Firstly, an area-aware user dynamic preference extracting module is developed concerning user representation learning, which can integrate the dynamic user behavior preference and the corresponding group structural information into group members’ representations. Then, to simultaneously obtain stability, propensity, potentiality, and heterogeneity of the group behavior, Multi-view Group Preference Aggregation (MGA) is designed for group behavior modeling. In MGA, various aggregation strategies are used to capture different kinds of group behavior properties, which can achieve the effect of highlighting different properties in different scenarios. Moreover, considering the influence of the member’s size on group behavior, we define the group scaling degree and perform group representation scaling to make MGA more flexible. Finally, extensive experiments are performed on two real-world datasets, and the results show that the accuracy of group behavior prediction by DMGPL improved by about 10% compared to the baseline models.

MM-FRec: Multi-Modal Enhanced Fashion Item Recommendation

Existing studies on fashion item recommendation mainly focused on incorporating the visual signals of items to boost the user preference learning, while overlooking the semantic attributes (e.g., material and brand) of fashion items that also contain important cues about items' properties and users' preference. To bridge this gap, we aim to comprehensively explore the attribute and vision modalities of items to improve the fashion item recommendation performance. However, this is non-trivial due to the latent visual-semantic consistency, various relation types, and unique attributes with insufficient samples. To address these challenges, we propose a Multi-Modal enhanced Fashion item Recommendation scheme (MM-FRec). Specifically, to cope with the multi-modal data, we introduce a relation-oriented graph as well as a vision-oriented graph, and design MM-FRec with three key components: attribute-enhanced latent representation learning, visual representation learning, and multi-modal enhanced preference modeling. To deal with the various relation types, we present a new relation-aware propagation method for adaptively aggregating the information from neighbor nodes to promote the user and item representation learning. To cope with the unique attributes, we introduce the deep multi-task learning strategy in the relation-aware confidence assignment. Extensive experiments on a real-world dataset demonstrate the superiority of our model over state-of-the-art methods.

DDHCN: Dual decoder Hyperformer convolutional network for Downstream-Adaptable user representation learning on app usage

In mobile scenarios, there is a need for general user representations to solve multiple target tasks. However, there are some challenges in the related research (e.g., difficulty in learning a representation that satisfies both great generalization and performance). To address these problems, we proposed a network for downstream-adaptable mobile user modeling, which employed a novel fine-tuning strategy for optimizing the performance of several downstream tasks. Additionally, we designed a time-difference module to eliminate the impact of low-frequency and non-uniform app usage behavior over time. A parallel decoder structure was developed to incorporate multi-type features by minimizing information loss. We evaluated our method on a real-world dataset of 100,000 mobile users and three downstream tasks (i.e., age prediction, gender prediction, and app recommendation). The experimental results showed that our method could outperform existing methods significantly. It achieved 96.5% ACC on gender prediction, 68.1% ACC on age prediction, and 64.2% Recall@5 on app recommendation. These results imply that our method performs well on both generalization and performance. It could be anticipated promising to the unseen tasks inference.

Group-Based Personalized News Recommendation with Long- and Short-Term Fine-Grained Matching

Personalized news recommendation aims to help users find news content they prefer, which has attracted increasing attention recently. There are two core issues in news recommendation: learning news representation and matching candidate news with user interests. In this context, “candidate” indicates potential for interest. Due to the superior ability to understand natural language demonstrated by Pretrained Language Models (PLMs), recent works utilize PLMs (e.g., BERT) to strengthen news modeling, obtaining more accurate user interest matching and achieving notable improvement in news recommendation. However, the existing PLM-based methods are usually incapable of fully exploring the fine-grained (i.e., word-level) relatedness between user behaviors and candidate news due to the heavy computational cost brought by PLMs. In this article, we propose a group-based personalized news recommendation method with long- and short-term matching mechanisms between users and candidate news based on PLMs to learn fine-grained matching efficiently and effectively. In our approach, we design to group user historical clicked news into chunks with quite shorter news sequences according to their clicked timestamps, which could alleviate the computation issues of PLMs. PLMs are applied in each group jointly with the candidate news to capture their word-level interaction, and global group-level matching is learned across different groups. In addition, the group-based mechanism could be naturally adapted for long- and short-term user representation learning, in which we build users’ long preferences from the representations of all groups and treat the last group as short interests, respectively. Finally, we employ a gate network to dynamically unify the group-level, long- and short-term representations, yielding comprehensive user-news matching effectively. Extensive experiments are conducted on two real-world datasets. The results show that our proposed method achieves superior performance in news recommendations.

Scaling Law for Recommendation Models: Towards General-Purpose User Representations

Recent advancement of large-scale pretrained models such as BERT, GPT-3, CLIP, and Gopher, has shown astonishing achievements across various task domains. Unlike vision recognition and language models, studies on general-purpose user representation at scale still remain underexplored. Here we explore the possibility of general-purpose user representation learning by training a universal user encoder at large scales. We demonstrate that the scaling law is present in user representation learning areas, where the training error scales as a power-law with the amount of computation. Our Contrastive Learning User Encoder (CLUE), optimizes task-agnostic objectives, and the resulting user embeddings stretch our expectation of what is possible to do in various downstream tasks. CLUE also shows great transferability to other domains and companies, as performances on an online experiment shows significant improvements in Click-Through-Rate (CTR). Furthermore, we also investigate how the model performance is influenced by the scale factors, such as training data size, model capacity, sequence length, and batch size. Finally, we discuss the broader impacts of CLUE in general.

LightGCAN: A lightweight graph convolutional attention network for user preference modeling and personalized recommendation

Graph Neural Network (GNN) is a promising technique in representation learning on graph data. Graph Convolution Network (GCN) and Graph Attention Network (GAT) are two main representative techniques in GNN, which can learn node embedding on the graph by aggregating the embedding of the neighborhoods. However, the complex computation process of existing GCN and GAT makes them less effective for tasks with complex graph structures. Moreover, traditional GCN and GAT ignore the time factor in user representation learning, so the obtained user preference model is static and cannot reflect the dynamics of user preferences. In this work, we propose a time-aware Lightweight Graph Convolutional Attention Network (LightGCAN), which can capture static and dynamic user preferences efficiently by using different GNN strategies. Specifically, static user preferences are modeled by a GCN with only neighborhood aggregation, and dynamic user preferences are extracted by a time-aware GAT based on recently interacted items. Static and dynamic user preferences are then combined and fed into a dual-channel Deep Neural Network (DNN) model for feature interaction learning and matching score prediction. Extensive experiments on different datasets show that LightGCAN is superior to the SOTA recommendation method.

MoCo4SRec: A momentum contrastive learning framework for sequential recommendation

Sequential recommendation (SR) is an essential component of modern recommender systems. It models the dynamic interest of users based on their sequential interactions. Recently, several studies have utilized sequential deep learning models such as Recurrent Neural Networks and Transformers to facilitate sequential recommendation, which shows promising results. Inspired by the rise of contrastive learning techniques, some methods are devoted to enhance sequential deep learning models by designing contrastive learning loss on self-supervised signals. However, there are still a number of obstacles that make it challenging to efficiently learn user representations by contrastive learning. These issues include, but are not limited to, data sparsity, noisy data, and sampling bias (e.g., false negative), particularly in complex, parameter-intensive models.In light of these challenges, we examine how to deal with data sparsity and noisy data by implementing contrastive Self-Supervised Learning (SSL) and Momentum Contrast (MoCo) to the sequential recommendation. Except typical in-batch negatives, our basic idea is to maintain a dynamic queue to expand negative samples with a moving-averaged encoder. After being augmented by sequence-level and embedding-level methods, the representations from all historical encoder output are pushed into the dynamic queue, which usually leads to sampling bias when potential positives in the queue are used as expanded negative samples in contrastive learning. To tackle this issue, we integrate momentum updating mechanism with a novel instance weighting mechanism to penalize false negatives and guarantee the model’s efficacy. To this end, we introduce a fresh framework called Momentum Contrastive Learning Framework for Sequential Recommendation (MoCo4SRec). Experiments on eight real-world datasets demonstrate how the suggested strategy might boost model performance over current benchmarks by adopting improved user representations.

Enhancing review-based user representation on learned social graph for recommendation

In recent years, review-based methods have been widely used to learn user representations because reviews contain abundant information. However, few users would like to write reviews for items. The data sparsity problem in reviews has become an emerging challenge in the field. Meanwhile, other approaches resort to social information based on graph neural networks (GNNs) to augment user representation. However, the efficacy of these approaches is always jeopardized because social graphs are not available in most real-world scenarios. Therefore, we propose a new Enhancing Review-based User Representation Model on Learned Social Graph for Recommendation, named ERUR. Specifically, we first introduce a review encoder to model review-based user/item representations. Second, we design a graph learning network to learn social relations between users according to the review-based user representation. Third, a graph neural network is developed to augment the final user representation under the supervision of a generative adversarial network. It integrates user reviews and social relations to enrich the final user representation for recommendation and further alleviate the data sparsity problem in reviews. Finally, we conduct experiments on seven datasets to demonstrate the effectiveness of the ERUR model in user representation learning compared to the SOTA recommendation models.

RESUS: Warm-up Cold Users via Meta-learning Residual User Preferences in CTR Prediction

Click-through Rate (CTR) prediction on cold users is a challenging task in recommender systems. Recent researches have resorted to meta-learning to tackle the cold-user challenge, which either perform few-shot user representation learning or adopt optimization-based meta-learning. However, existing methods suffer from information loss or inefficient optimization process, and they fail to explicitly model global user preference knowledge, which is crucial to complement the sparse and insufficient preference information of cold users. In this article, we propose a novel and efficient approach named RESUS, which decouples the learning of global preference knowledge contributed by collective users from the learning of residual preferences for individual users. Specifically, we employ a shared predictor to infer basis user preferences, which acquires global preference knowledge from the interactions of different users. Meanwhile, we develop two efficient algorithms based on the nearest neighbor and ridge regression predictors, which infer residual user preferences via learning quickly from a few user-specific interactions. Extensive experiments on three public datasets demonstrate that our RESUS approach is efficient and effective in improving CTR prediction accuracy on cold users, compared with various state-of-the-art methods.

Graph Neural News Recommendation with User Existing and Potential Interest Modeling

Personalized news recommendations can alleviate the information overload problem. To enable personalized recommendation, one critical step is to learn a comprehensive user representation to model her/his interests. Many existing works learn user representations from the historical clicked news articles, which reflect their existing interests. However, these approaches ignore users’ potential interests and pay less attention to news that may interest the users in the future. To address this problem, we propose a novel G raph neural news R ecommendation model with user E xisting and P otential interest modeling, named GREP. Different from existing works, GREP introduces three modules to jointly model users’ existing and potential interests: (1) Existing Interest Encoding module mines user historical clicked news and applies the multi-head self-attention mechanism to capture the relatedness among the news; (2) Potential Interest Encoding module leverages the graph neural network to explore the user potential interests on the knowledge graph; and (3) Bi-directional Interaction module dynamically builds a news-entity bipartite graph to further enrich two interest representations. Finally, GREP combines the existing and potential interest representations to represent the user and leverages a prediction layer to estimate the clicking probability of the candidate news. Experiments on two real-world large-scale datasets demonstrate the state-of-the-art performance of GREP.

Joint Learning of User Representation With Diffusion Sequence and Network Structure

Information sharing behavior and social link building behavior have shown strong correlation on social media. The aim of this paper is to explore this correlation for simultaneously modeling and predicting sharing behavior in information diffusion sequences and linking behavior in social network, which correspond to information diffusion prediction and social link prediction problems. To achieve this goal, we propose a joint user representation learning model to characterize the two correlated behaviors in a shared latent space. The proposed model learns user representations via two maximum likelihood estimation objectives defined on observed information diffusion sequences and social network structure respectively and incorporates them in a unified framework. A multi-task learning algorithm is designed for efficient model optimization. Based on the learned representations, the model can be directly applied to predicting diffusion processes and inferring unobserved social links at the same time. We evaluate the proposed model on two real social media datasets with extensive experiments. The model consistently achieves significant improvements over the state-of-the-art approaches on diffusion prediction and link prediction tasks. The better robustness of our model in further ablation studies demonstrates that capturing the behavior correlation in the shared representation space is beneficial.

Catch me if you can: A participant-level rumor detection framework via fine-grained user representation learning

Researchers have exerted tremendous effort in designing ways to detect and identify rumors automatically. Traditional approaches focus on feature engineering. They require lots of human actions and are difficult to generalize. Deep learning solutions come to help. However, they usually fail to capture the underlying structure of the rumor propagation and the influence of all participants involved in the spreading chain. In this study, we propose a novel participant-level rumor detection framework. It explicitly models and integrates various fine-grained user representations (i.e., user influence, susceptibility, and temporal information) of all participants from the propagation threads via deep representation learning. Experiments conducted on real-world datasets demonstrate a significant accuracy improvement of our approach. Theoretically, we contribute to the effective usage of data science and analytics for social information diffusion design, particularly rumor detection. Practically, our results can be used to improve the quality of rumor detection services for social platforms.

User Representation Learning for Social Networks: An Empirical Study

Gathering useful insights from social media data has gained great interest over the recent years. User representation can be a key task in mining publicly available user-generated rich content offered by the social media platforms. The way to automatically create meaningful observations about users of a social network is to obtain real-valued vectors for the users with user embedding representation learning models. In this study, we presented one of the most comprehensive studies in the literature in terms of learning high-quality social media user representations by leveraging state-of-the-art text representation approaches. We proposed a novel doc2vec-based representation method, which can encode both textual and non-textual information of a social media user into a low dimensional vector. In addition, various experiments were performed for investigating the performance of text representation techniques and concepts including word2vec, doc2vec, Glove, NumberBatch, FastText, BERT, ELMO, and TF-IDF. We also shared a new social media dataset comprising data from 500 manually selected Twitter users of five predefined groups. The dataset contains different activity data such as comment, retweet, like, location, as well as the actual tweets composed by the users.

Exploiting Behavioral Consistence for Universal User Representation

User modeling is critical for developing personalized services in industry. A common way for user modeling is to learn user representations that can be distinguished by their interests or preferences. In this work, we focus on developing universal user representation model. The obtained universal representations are expected to contain rich information, and be applicable to various downstream applications without further modifications (e.g., user preference prediction and user profiling). Accordingly, we can be free from the heavy work of training task-specific models for every downstream task as in previous works. In specific, we propose Self-supervised User Modeling Network (SUMN) to encode behavior data into the universal representation. It includes two key components. The first one is a new learning objective, which guides the model to fully identify and preserve valuable user information under a self-supervised learning framework. The other one is a multi-hop aggregation layer, which benefits the model capacity in aggregating diverse behaviors. Extensive experiments on benchmark datasets show that our approach can outperform state-of-the-art unsupervised representation methods, and even compete with supervised ones.