Dynamic Representation Learning Research Articles

FELRec: efficient handling of item cold-start with dynamic representation learning in recommender systems

AbstractRecommender systems suffer from the cold-start problem whenever a new user joins the platform or a new item is added to the catalog. To address item cold-start, we propose to replace the embedding layer in sequential recommenders with a dynamic storage that has no learnable weights and can keep an arbitrary number of representations. In this paper, we present , a large embedding network that refines the existing representations of users and items in a recursive manner, as new information becomes available. In contrast to similar approaches, our model represents new users and items without side information and time-consuming finetuning, instead it runs a single forward pass over a sequence of existing representations. During item cold-start, our method outperforms similar method by 29.50–47.45%. Further, our proposed model generalizes well to previously unseen datasets in zero-shot settings. The source code is publicly available at https://github.com/kweimann/FELRec.

Leveraging Brain Modularity Prior for Interpretable Representation Learning of fMRI.

Resting-state functional magnetic resonance imaging (rs-fMRI) can reflect spontaneous neural activities in the brain and is widely used for brain disorder analysis. Previous studies focus on extracting fMRI representations using machine/deep learning methods, but these features typically lack biological interpretability. The human brain exhibits a remarkable modular structure in spontaneous brain functional networks, with each module comprised of functionally interconnected brain regions-of-interest (ROIs). However, existing learning-based methods cannot adequately utilize such brain modularity prior. In this paper, we propose a brain modularity-constrained dynamic representation learning framework for interpretable fMRI analysis, consisting of dynamic graph construction, dynamic graph learning via a novel modularity-constrained graph neural network (MGNN), and prediction and biomarker detection. The designed MGNN is constrained by three core neurocognitive modules (i.e., salience network, central executive network, and default mode network), encouraging ROIs within the same module to share similar representations. To further enhance discriminative ability of learned features, we encourage the MGNN to preserve network topology of input graphs via a graph topology reconstruction constraint. Experimental results on 534 subjects with rs-fMRI scans from two datasets validate the effectiveness of the proposed method. The identified discriminative brain ROIs and functional connectivities can be regarded as potential fMRI biomarkers to aid in clinical diagnosis.

Entropy-Aware Time-Varying Graph Neural Networks with Generalized Temporal Hawkes Process: Dynamic Link Prediction in the Presence of Node Addition and Deletion

This paper addresses the problem of learning temporal graph representations, which capture the changing nature of complex evolving networks. Existing approaches mainly focus on adding new nodes and edges to capture dynamic graph structures. However, to achieve more accurate representation of graph evolution, we consider both the addition and deletion of nodes and edges as events. These events occur at irregular time scales and are modeled using temporal point processes. Our goal is to learn the conditional intensity function of the temporal point process to investigate the influence of deletion events on node representation learning for link-level prediction. We incorporate network entropy, a measure of node and edge significance, to capture the effect of node deletion and edge removal in our framework. Additionally, we leveraged the characteristics of a generalized temporal Hawkes process, which considers the inhibitory effects of events where past occurrences can reduce future intensity. This framework enables dynamic representation learning by effectively modeling both addition and deletion events in the temporal graph. To evaluate our approach, we utilize autonomous system graphs, a family of inhomogeneous sparse graphs with instances of node and edge additions and deletions, in a link prediction task. By integrating these enhancements into our framework, we improve the accuracy of dynamic link prediction and enable better understanding of the dynamic evolution of complex networks.

Generic and Dynamic Graph Representation Learning for Crowd Flow Modeling

Many deep spatio-temporal learning methods have been proposed for crowd flow modeling in recent years. However, most of them focus on designing a spatial and temporal convolution mechanism to aggregate information from nearby nodes and historical observations for a pre-defined prediction task. Different from the existing research, this paper aims to provide a generic and dynamic representation learning method for crowd flow modeling. The main idea of our method is to maintain a continuous-time representation for each node, and update the representations of all nodes continuously according to the streaming observed data. Along this line, a particular encoder-decoder architecture is proposed, where the encoder converts the newly happened transactions into a timestamped message, and then the representations of related nodes are updated according to the generated message. The role of the decoder is to guide the representation learning process by reconstructing the observed transactions based on the most recent node representations. Moreover, a number of virtual nodes are added to discover macro-level spatial patterns and also share the representations among spatially-interacted stations. Experiments have been conducted on two real-world datasets for four popular prediction tasks in crowd flow modeling. The result demonstrates that our method could achieve better prediction performance for all the tasks than baseline methods.

Dynamic Representation Learning with Temporal Point Processes for Higher-Order Interaction Forecasting

The explosion of digital information and the growing involvement of people in social networks led to enormous research activity to develop methods that can extract meaningful information from interaction data. Commonly, interactions are represented by edges in a network or a graph, which implicitly assumes that the interactions are pairwise and static. However, real-world interactions deviate from these assumptions: (i) interactions can be multi-way, involving more than two nodes or individuals (e.g., family relationships, protein interactions), and (ii) interactions can change over a period of time (e.g., change of opinions and friendship status). While pairwise interactions have been studied in a dynamic network setting and multi-way interactions have been studied using hypergraphs in static networks, there exists no method, at present, that can predict multi-way interactions or hyperedges in dynamic settings. Existing related methods cannot answer temporal queries like what type of interaction will occur next and when it will occur. This paper proposes a temporal point process model for hyperedge prediction to address these problems. Our proposed model uses dynamic representation learning techniques for nodes in a neural point process framework to forecast hyperedges. We present several experimental results and set benchmark results. As far as our knowledge, this is the first work that uses the temporal point process to forecast hyperedges in dynamic networks.

Live streaming recommendations based on dynamic representation learning

As an emerging form of social media, live streaming services (e.g., Twitch and Clubhouse) allow users to interact with hosts and peers in real time while enjoying shows or participating in discussions. These platforms are also dynamic, with shows or discussions changing quickly inside a room and users frequently switching between rooms. To improve user engagement and experience on such platforms, we design a new recommendation model named Dynamic Representations for Live Streaming Rooms (DRIVER) to provide room recommendations. Guided by the Integrated Framework for Consumer Path Modeling and the social affordance theory, DRIVER infers dynamic representations of live streaming rooms by leveraging users’ behavior paths in entering, staying in, and leaving rooms. One contribution of our model is a new and efficient dynamic learning framework to model instantaneous and ever-changing inter-room relationships by considering individual users’ behavior paths after leaving a room. Also supported by social affordance theory, another methodological novelty of our model is to capture dynamic characteristics of a room by incorporating features of the current audience inside the room. Experiments on real-world datasets from two different types of live streaming platforms demonstrate that DRIVER outperforms state-of-the-art representation learning methods and sequential recommender systems. The proposed method also has implications for recommender system design in other contexts, in which items are characterized by users’ dynamic behavior paths and ongoing social interactions.

Dynamic Representation Learning via Recurrent Graph Neural Networks

A large number of real-world systems generate graphs that are structured data aligned with nodes and edges. Graphs are usually dynamic in many scenarios, where nodes or edges keep evolving over time. Recently, graph representation learning (GRL) has received great success in network analysis, which aims to produce informative and representative features or low-dimensional embeddings by exploring node attributes and network topology. Most state-of-the-art models for dynamic GRL are composed of a static representation learning model and a recurrent neural network (RNN). The former generates the representations of a graph or nodes from one static graph at a discrete time step, while the latter captures the temporal correlation between adjacent graphs. However, the two-stage design ignores the temporal dynamics between contiguous graphs during the learning processing of graph representations. To alleviate this problem, this article proposes a representation learning model for dynamic graphs, called DynGNN. Differently, it is a single-stage model that embeds an RNN into a graph neural network to produce better representations in a compact form. This takes the fusion of temporal and topology correlations into account from low-level to high-level feature learning, enabling the model to capture more fine-grained evolving patterns. From the experimental results on both synthetic and real-world networks, the proposed DynGNN yields significant improvements in multiple tasks compared to the state-of-the-art counterparts.

Modularity-Constrained Dynamic Representation Learning for Interpretable Brain Disorder Analysis with Functional MRI.

Resting-state functional MRI (rs-fMRI) is increasingly used to detect altered functional connectivity patterns caused by brain disorders, thereby facilitating objective quantification of brain pathology. Existing studies typically extract fMRI features using various machine/deep learning methods, but the generated imaging biomarkers are often challenging to interpret. Besides, the brain operates as a modular system with many cognitive/topological modules, where each module contains subsets of densely inter-connected regions-of-interest (ROIs) that are sparsely connected to ROIs in other modules. However, current methods cannot effectively characterize brain modularity. This paper proposes a modularity-constrained dynamic representation learning (MDRL) framework for interpretable brain disorder analysis with rs-fMRI. The MDRL consists of 3 parts: (1) dynamic graph construction, (2) modularity-constrained spatiotemporal graph neural network (MSGNN) for dynamic feature learning, and (3) prediction and biomarker detection. In particular, the MSGNN is designed to learn spatiotemporal dynamic representations of fMRI, constrained by 3 functional modules (i.e., central executive network, salience network, and default mode network). To enhance discriminative ability of learned features, we encourage the MSGNN to reconstruct network topology of input graphs. Experimental results on two public and one private datasets with a total of 1,155 subjects validate that our MDRL outperforms several state-of-the-art methods in fMRI-based brain disorder analysis. The detected fMRI biomarkers have good explainability and can be potentially used to improve clinical diagnosis.

Sparse‐Dyn: Sparse dynamic graph multirepresentation learning via event‐based sparse temporal attention network

Dynamic graph neural networks (DGNNs) have been widely used in modeling and representation learning of graph structure data. Current dynamic representation learning focuses on either discrete learning which results in temporal information loss, or continuous learning which involves heavy computation. In this study, we proposed a novel DGNN, sparse dynamic (Sparse-Dyn). It adaptively encodes temporal information into a sequence of patches with an equal amount of temporal-topological structure. Therefore, while avoiding using snapshots which cause information loss, it also achieves a finer time granularity, which is close to what continuous networks could provide. In addition, we also designed a lightweight module, Sparse Temporal Transformer, to compute node representations through structural neighborhoods and temporal dynamics. Since the fully connected attention conjunction is simplified, the computation cost is far lower than the current state-of-the-art. Link prediction experiments are conducted on both continuous and discrete graph data sets. By comparing several state-of-the-art graph embedding baselines, the experimental results demonstrate that Sparse-Dyn has a faster inference speed while having competitive performance.

Multi-Response Awareness for Retrieval-Based Conversations: Respond with Diversity via Dynamic Representation Learning

Conversational systems now attract great attention due to their promising potential and commercial values. To build a conversational system with moderate intelligence is challenging and requires big (conversational) data, as well as interdisciplinary techniques. Thanks to the prosperity of the Web, the massive data available greatly facilitate data-driven methods such as deep learning for human-computer conversational systems. In general, retrieval-based conversational systems apply various matching schema between query utterances and responses, but the classic retrieval paradigm suffers from prominent weakness for conversations: the system finds similar responses given a particular query. For real human-to-human conversations, on the contrary, responses can be greatly different yet all are possibly appropriate. The observation reveals the diversity phenomenon in conversations. In this article, we ascribe the lack of conversational diversity to the reason that the query utterances are statically modeled regardless of candidate responses through traditional methods. To this end, we propose a dynamic representation learning strategy that models the query utterances and different response candidates in an interactive way. To be more specific, we propose a Respond-with-Diversity model augmented by the memory module interacting with both the query utterances and multiple candidate responses. Hence, we obtain dynamic representations for the input queries conditioned on different response candidates. We frame the model as an end-to-end learnable neural network. In the experiments, we demonstrate the effectiveness of the proposed model by achieving a good appropriateness score and much better diversity in retrieval-based conversations between humans and computers.

Vehicle Trajectory Clustering Based on Dynamic Representation Learning of Internet of Vehicles

With the widely used Internet of Things, 5G, and smart city technologies, we are able to acquire a variety of vehicle trajectory data. These trajectory data are of great significance which can be used to extract relevant information in order to, for instance, calculate the optimal path from one position to another, detect abnormal behavior, monitor the traffic flow in a city, and predict the next position of an object. One of the key technology is to cluster vehicle trajectory. However, existing methods mainly rely on manually designed metrics which may lead to biased results. Meanwhile, the large scale of vehicle trajectory data has become a challenge because calculating these manually designed metrics will cost more time and space. To address these challenges, we propose to employ network representation learning to achieve accurate vehicle trajectory clustering. Specifically, we first construct the k-nearest neighbor-based internet of vehicles in a dynamic manner. Then we learn the low-dimensional representations of vehicles by performing dynamic network representation learning on the constructed network. Finally, using the learned vehicle vectors, vehicle trajectories are clustered with machine learning methods. Experimental results on the real-word dataset show that our method achieves the best performance compared against baseline methods.

HIN_DRL: A random walk based dynamic network representation learning method for heterogeneous information networks

Learning the low-dimensional vector representation of networks can effectively reduce the complexity of various network analysis tasks, such as link prediction, clustering and classification. However, most of the existing network representation learning (NRL) methods are aimed at homogeneous or static networks, while the real-world networks are usually heterogeneous and tend to change dynamically over time, therefore providing an intelligent insight into the evolution of heterogeneous networks is more practical and significant. Based on this consideration, we focus on the dynamic representation learning problem for heterogeneous information networks, and propose a random walk based Dynamic Representation Learning method for Heterogeneous Information Networks (HIN_DRL), which can learn the representation of network nodes at different timestamps. Specifically, we improve the first step of the existing random walk based NRL methods, which generally include two steps: constructing node sequences through random walk process, and then learning node representations by throwing the node sequences into a homogeneous or heterogeneous Skip-Gram model. In order to construct optimized node sequences for evolving heterogeneous networks, we propose a method for automatically extracting and extending meta-paths, and propose a new method for generating node sequences via dynamic random walk based on meta-path and timestamp information of networks. We also propose two strategies for adjusting the quantity and length of node sequences during each random walk process, which makes it more effective to construct the node sequences for heterogeneous information networks at a specific timestamp, thus improving the effect of dynamic representation learning. Extensive experimental results show that compared with the state-of-art algorithms, HIN_DRL achieves better results in Macro-F1, Micro-F1 and NMI for multi-label node classification, multi-class node classification and node clustering on several real-world network datasets. Furthermore, case studies of visualization and dynamic on Microsoft Academic dataset demonstrate that HIN_DRL can learn network representation dynamically and more effectively.

Entity Thematic Similarity Measurement for Personal Explainable Searching Services in the Edge Environment

Currently, search engines are widely used to address the information overload problem. Different from the existing client-and-sever-based frameworks, edge computing (EC) technology can provide a new architecture for personalized searching services. The issue of how to measure the similarities among entities by using the context information generated by user behavior in the edge environment is vital in the task of entity-related personal searching. To analyze and measure the similarities among entities, existing methods are mainly based on either the textual content or relationships unilaterally, and the results usually have a fixed degree of similarity. However, the similarities among entities depend on the set of properties that belong to the entities. This approach should be used in determining the similarity or dissimilarity associated with the surrounding context. To address this limitation, we propose a novel semantic augmentation method with a double attention mechanism. The method refers to a dynamic representation learning process that maps an entity to a real number vector in semantic space. In this article, different from the existing similarity measurement methods, we propose a thematic similarity measure approach to analyze the connotation and denotation similarities among entities. The experimental results show that the double attention mechanism leads to a significant improvement in the entity thematic similarity measurement tasks. The model can make a separation among the entities from different domains effectively. In addition, it can take similar entities that are closer in the same domain. It also shows excellent performance on the task of entity thematic similarity, which makes the recommendation results more explainable.

Modeling Temporal Dynamics of Users’ Purchase Behaviors for Next Basket Prediction

Next basket prediction attempts to provide sequential recommendations to users based on a sequence of the user’s previous purchases. Ideally, a good prediction model should be able to explore the personalized preference of the users, as well as the sequential relations of the items. This goal of modeling becomes even more challenging when both factors are time-dependent. However, existing methods either take these two aspects as static, or only consider temporal dynamics for one of the two aspects. In this work, we propose the dynamic representation learning approach for time-dependent next basket recommendation, which jointly models the dynamic nature of user preferences and item relations. To do so, we explicitly model the transaction timestamps, as well as the dynamic representations of both users and items, so as to capture the personalized user preference on each individual item dynamically. Experiments on three real-world retail datasets show that our method significantly outperforms several state-of-the-art methods for next basket recommendation.

Dynamic reconstruction based representation learning for multivariable process monitoring

Representation learning is a key step for fault detection in the process monitoring. With the consideration of time-correlated, this paper focus on developing a dynamic representation learning method yields high-order correlations in process monitoring. First, a general interpretation of AE net is presented based on Taylor expansion, which motivates the representation ability by the composition of multiple non-linear transformations. Due to the fact that the nearest neighbors over time are not necessarily the nearest spatial neighbors in dynamic process, a dynamic reconstruction is developed based on its k nearest neighbors. The reconstruction can not only maintain the separability, but also increase the distinguishable distance between categories. Finally, numerical results shown that a multi-layer Taylor network is efficiently to approximate the smooth sigmoid functions of AE net, and experiments on Tennessee Eastman process (TEP) illustrated the proposed method’s superior detectability, specifically for incipient faults.