Embedding Learning Research Articles

Multi-view graph contrastive representation learning for bundle recommendation

Bundle recommendation can recommend a collection of associated items that can be consumed together to a user rather than recommending these items separately, making it extremely suitable for some scenarios such as product bundle recommendation and game bundle recommendation. Recent bundle recommendation approaches consider auxiliary data to mitigate sparse user-bundle interactions. However, these approaches obtain the node embeddings directly from the established user-bundle graph and do not explicitly exploit the relationships between users (bundles) when constructing recommendation models. Moreover, bundle recommendation approaches based on graph contrastive learning usually construct contrastive views by randomly discarding nodes (edges) in the graph, while discarding some essential nodes or edges will destroy the structure of the original graph, thereby deteriorating the quality of the learned node embeddings. Aiming at these limitations, we propose a bundle recommendation approach based on multi-view graph contrastive representation learning. First, we present a multi-view modeling method to model the relations between entities as several views from different perspectives. These views serve as inputs of graph neural networks for graph representation learning and provide contrastive views for the contrastive learning tasks. Second, we propose a novel framework for bundle recommendation. This framework obtains the user (bundle) embeddings from different views by performing multi-view graph representation learning and enhances the learned user and bundle embeddings through a two-level contrastive learning strategy. On this basis, the enhanced user (bundle) embeddings are fused for prediction. Finally, we design a joint optimization objective to optimize the model parameters, combining the prediction loss that supports multiple negative samples and the contrastive losses. Experiments on the Netease and Youshu datasets reveal that our approach outperforms the state-of-the-art (SOTA) baselines. Furthermore, the average improvements of Recall@K and NDCG@K of our approach over the SOTA baselines are approximately 3.38% and 2.80% on Netease and 3.94% and 4.84% on Youshu.

Cat2Vec with position encoding: A new approach for handling ordinal features using learned embeddings with positional encoding

Cat2Vec with position encoding: A new approach for handling ordinal features using learned embeddings with positional encoding

Machine learning and rule-based embedding techniques for classifying text documents

Machine learning and rule-based embedding techniques for classifying text documents

Graph attention automatic encoder based on contrastive learning for domain recognition of spatial transcriptomics

Spatial transcriptomics is an emerging technology that enables the profiling of gene expression in tissues while preserving spatial location information. This innovative approach is anticipated to provide a comprehensive understanding of the spatial distribution of different cells within tissues and facilitate in-depth analysis of tissue structure. To accurately recognize spatial domains from spatial transcriptomics, we have introduced a generalized deep learning method called GAAEST (Graph Attention-based Autoencoder for Spatial Transcriptomics). Our proposed approach effectively integrates both spatial location information and gene expression data from spatial transcriptomics. Specifically, it leverages spatial location details to construct a neighborhood graph and employs a graph attention network-based encoder to embed gene expression information into a spatially informed space. At the same time, to further optimize the learned potential embedding, self-supervised contrastive learning is introduced to capture spatial information at three levels: local, global and contextual feature of spots. Finally, the decoder reconstructs gene expressions, which are then clustered to identify spatial domains with similar expression patterns and spatial proximity. Based on our experiments conducted on multiple datasets, GAAEST consistently outperforms existing state-of-the-art methods. The proposed GAAEST demonstrates excellent capabilities in spatial domain recognition, positioning it as an ideal tool for advancing spatial transcriptomics research.

Distortion corrected kernel density estimator on Riemannian manifolds

Manifold learning obtains a low-dimensional representation of an underlying Riemannian manifold supporting high-dimensional data. Kernel density estimates of the low-dimensional embedding with a fixed bandwidth fail to account for the way manifold learning algorithms distort the geometry of the Riemannian manifold. We propose a novel distortion-corrected kernel density estimator (DC-KDE) for any manifold learning embedding, with a bandwidth that depends on the estimated Riemannian metric at each data point. Exploiting the geometric information of the manifold leads to more accurate density estimation, which subsequently could be used for anomaly detection. To compare our proposed estimator with a fixed-bandwidth kernel density estimator, we run two simulations including one with data lying in a 100 dimensional ambient space. We demonstrate that the proposed DC-KDE improves the density estimates as long as the manifold learning embedding is of sufficient quality, and has higher rank correlations with the true manifold density. Further simulation results are provided via a supplementary R shiny app. The proposed method is applied to density estimation in statistical manifolds of electricity usage with the Irish smart meter data.

FGTL: Federated Graph Transfer Learning for Node Classification

Unsupervised multi-source domain transfer in federated scenario has become an emerging research direction, which can help unlabeled target domain to obtain the adapted model through source domains under privacy-preserving. However, when local data is graph, the difference of domains (or data heterogeneity) mainly originate from the difference in node attributes and subgraph structures, leading to serious model drift, which is not considered by the existing related algorithms. Currently, there are two challenges in this scenario: (1) The node representations extracted directly through conventional GNNs lack inter-domain generalized and consistent information, making it difficult to apply existing federated learning algorithms. (2) The knowledge of source domains has quality differences, which may lead to negative transfer. To address these issues, we propose a novel two-phase Federated Graph Transfer Learning (FGTL) framework. In the generalization phase, FGTL utilizes local contrastive learning and global context embedding to force node representations to capture the inter-domain generalized and consistent information, lightly alleviating model drift. In the transfer phase, FGTL utilizes consensus knowledge to force the decision bound of classifier to adapt to the target client. In addition, FGTL+ exploits model grouping to make consensus knowledge generation more efficient, further enhancing the scalability of FGTL. Extensive experiments show that FGTL significantly outperforms state-of-the-art related methods, while FGTL+ further enhances privacy protection and reduces both communication and computation overhead.

A Machine Learning and Deep Learning-Based Account Code Classification Model for Sustainable Accounting Practices

Accounting account codes are created within a specific logic framework to systematically and accurately record a company’s financial transactions. Currently, accounting reports are processed manually, which increases the likelihood of errors and slows down the process. This study aims to use image processing techniques to predict cash codes in accounting reports, automate accounting processes, improve accuracy, and save time. Deep learning embeddings from Inception V3, SqueezeNet, VGG-19, VGG-16, Painters, and DeepLoc networks were utilized in the feature extraction phase. A total of six learning algorithms, namely Logistic Regression, Gradient Boosting, Neural Network, kNN, Naive Bayes, and Stochastic Gradient Descent were employed to classify the images. The highest accuracy rate of 99.2% was achieved with the combination of the Inception V3 feature extractor and the Neural Network classifier. The results demonstrate that image processing methods significantly reduce error rates in accounting records, accelerate processes, and support sustainable accounting practices. This indicates that image processing techniques have substantial potential to contribute to digital transformation in accounting, helping businesses achieve their sustainability goals.

Dual-perspective multi-instance embedding learning with adaptive density distribution mining

Multi-instance learning (MIL) is a potent framework for solving weakly supervised problems, with bags containing multiple instances. Various embedding methods convert each bag into a vector in the new feature space based on a representative bag or instance, aiming to extract useful information from the bag. However, since the distribution of instances is related to labels, these methods rely solely on the overall perspective embedding without considering the different distribution characteristics, which will conflate the varied distributions of instances and thus lead to poor classification performance. In this paper, we propose the dual-perspective multi-instance embedding learning with adaptive density distribution mining (DPMIL) algorithm with three new techniques. First, the mutual instance selection technique consists of adaptive density distribution mining and discriminative evaluation. The distribution characteristics of negative instances and heterogeneous instance dissimilarity are effectively exploited to obtain instances with strong representativeness. Second, the embedding technique mines two crucial information of the bag simultaneously. Bags are converted into sequence invariant vectors according to the dual-perspective such that the distinguishability is maintained. Finally, the ensemble technique trains a batch of classifiers. The final model is obtained by weighted voting with the contribution of the dual-perspective embedding information. The experimental results demonstrate that the DPMIL algorithm has higher average accuracy than other compared algorithms, especially on web datasets.

Research on Heterogeneous Information Network Link Prediction Based on Representation Learning

A heterogeneous information network, which is composed of various types of nodes and edges, has a complex structure and rich information content, and is widely used in social networks, academic networks, e-commerce, and other fields. Link prediction, as a key task to reveal the unobserved relationships in the network, is of great significance in heterogeneous information networks. This paper reviews the application of presentation-based learning methods in link prediction of heterogeneous information networks. This paper introduces the basic concepts of heterogeneous information networks, and the theoretical basis of representation learning, and discusses the specific application of the deep learning model in node embedding learning and link prediction in detail. The effectiveness and superiority of these methods on multiple real data sets are demonstrated by experimental verification.

Plant lncRNA-miRNA Interaction Prediction Based on Counterfactual Heterogeneous Graph Attention Network.

Identifying interactions between long non-coding RNAs (lncRNAs) and microRNAs (miRNAs) provides a new perspective for understanding regulatory relationships in plant life processes. Recently, computational methods based on graph neural networks (GNNs) have been widely employed to predict lncRNA-miRNA interactions (LMIs), which compensate for the inadequacy of biological experiments. However, the low-semantic and noise of graph limit the performance of existing GNN-based methods. In this paper, we develop a novel Counterfactual Heterogeneous Graph Attention Network (CFHAN) to improve the robustness to against the noise and the prediction of plant LMIs. Firstly, we construct a real-world based lncRNA-miRNA (L-M) heterogeneous network. Secondly, CFHAN utilizes the node-level attention, the semantic-level attention, and the counterfactual links to enhance the node embeddings learning. Finally, these embeddings are used as inputs for Multilayer Perceptron (MLP) to predict the interactions between lncRNAs and miRNAs. Evaluating our method on a benchmark dataset of plant LMIs, CFHAN outperforms five state-of-the-art methods, and achieves an average AUC and average ACC of 0.9953 and 0.9733, respectively. This demonstrates CFHAN's ability to predict plant LMIs and exhibits promising cross-species prediction ability, offering valuable insights for experimental LMI researches.

Exploring Recent NLP Advances for Tamil: Word Vectors and Hybrid Deep Learning Architectures

The advancements of deep learning methods and the availability of large corpora and data sets have led to an exponential increase in the performance of Natural Language Processing (NLP) methods resulting in successful NLP applications for various day-to-day tasks such as Language translation, Voice to text, Grammar checking, Sentiment analysis, etc. These advancements enabled the well-resourced languages to adapt themselves to the digital era while the gap for the low-resource languages widened. This research work explores the suitability of the recent advancements in NLP for Tamil, a low-resource language spoken mainly in South India, Sri Lanka, and Malaysia. From the literature survey, it has been found that there is a lack of comprehensive study on the effect of the recent advancements of NLP for the Tamil text. To fill this gap, this research work analysed the performance of deep learning based text representation and classification approaches namely word embedding, Convolutional Neural Network (CNN), and Recurrent Neural Network (RNN) for Tamil text classification tasks. Different dimensional pretrained word2Vec and FastText word vectors were built for Tamil and their effectiveness on Text classification was evaluated. The study found that the pre-trained 300- dimensional FastText word vector showed better performance than other pre-trained word vectors for Tamil text classification. Further, in this study, four simple hybrid CNN and Bi-GRU models were proposed for Tamil text classification and their performances were evaluated. The study found that hybrid CNN and Bi-GRU models perform better compared to the classical machine learning models, individual CNN and RNN models, and the Multilingual BERT model. These results confirm that the jointly learned embeddings with different deep learning architectures like CNN and RNN can achieve remarkable results for Tamil text classification, thus ensuring that the deep learning approaches can be successful for NLP on Tamil text.

Heterogeneous propagation graph convolution network for a recommendation system based on a knowledge graph

Recently, there has been a surge of interest in recommendation systems that leverage knowledge graphs, primarily because of their effectiveness in addressing sparsity and cold-start challenges inherent in collaborative filtering approaches. In most previous studies, researchers have focused on the way knowledge associations are encoded in knowledge graphs, but have not sufficiently highlighted the signals of collaboration that are implicit in the interaction between users and items. As a result, the learned embeddings do not provide a complete representation of the semantic information. In this paper, we describe a new model called a heterogeneous propagation graph convolution network for a recommendation system combined with a knowledge graph (HP-GCN). It adopts a heterogeneous propagation to generate user embedding representations, thereby combining encoded collaborative signals and auxiliary knowledge in knowledge graphs. Furthermore, we incorporate an attention mechanism to differentiate the contributions made by diverse neighbors as opposed to those made by users. Since most graph convolutions tend to suffer from over-smoothing when the number of convolutional layers increases, leading to insufficient utilization of high-order information, this paper uses an improved graph convolution strategy to generate item embeddings. This strategy has two different aggregation mechanisms embedding into different subgraphs, which can more fully utilize high-order information and mitigate the over-smoothing problem. Thus, we are able to efficiently prevent negative information originating from higher-order neighbors into the process of embedding learning. In extensive experiments, we applied HP-GCN to four large-scale real datasets for music, books, movies, and restaurants. The experimental outcomes revealed that HP-GCN generally surpassed the baseline methods in both recommendation accuracy and diversity, showing superior recommendation performance overall.

Adaptive indefinite kernels in hyperbolic spaces

Learning embeddings in hyperbolic space has gained increasing interest in the community, due to its property of negative curvature, as a way of encoding data hierarchy. Recent works investigate the improvement of the representation power of hyperbolic embeddings through kernelization. However, existing developments focus on defining positive definite (pd) kernels, which may affect the intriguing property of hyperbolic spaces. This is due to the structures of hyperbolic spaces being modeled in indefinite spaces (e.g., Kreĭn space). This paper addresses this issue by developing adaptive indefinite kernels, which can better utilize the structures in the Kreĭn space. To this end, we first propose an adaptive embedding function in the Lorentz model and define indefinite Lorentz kernels (iLks) via the embedding function. Due to the isometric relationship between the Lorentz model and the Poincaré ball, these iLks are further extended to the Poincaré ball, resulting in the development of what are termed indefinite Poincaré kernels (iPKs). We evaluate the proposed indefinite kernels on a diversity of learning scenarios, including image classification, few-shot learning, zero-shot learning, person re-identification, knowledge distillation, etc. We show that the proposed indefinite kernels can bring significant performance gains over the baselines and enjoy better representation power from RKKSs than pd kernels.

Bilingual phrase induction with local hard negative sampling

AbstractBilingual lexicon induction focuses on learning word translation pairs, also known as bitexts, from monolingual corpora by establishing a mapping between the source and target embedding spaces. Despite recent advancements, bilingual lexicon induction is limited to inducing bitexts consisting of individual words, lacking the ability to handle semantics‐rich phrases. To bridge this gap and support downstream cross‐lingual tasks, it is practical to develop a method for bilingual phrase induction that extracts bilingual phrase pairs from monolingual corpora without relying on cross‐lingual knowledge. In this paper, the authors propose a novel phrase embedding training method based on the skip‐gram structure. Specifically, a local hard negative sampling strategy that utilises negative samples of central tokens in sliding windows to enhance phrase embedding learning is introduced. The proposed method achieves competitive or superior performance compared to baseline approaches, with exceptional results recorded for distant languages. Additionally, we develop a phrase representation learning method that leverages multilingual pre‐trained language models. These mPLMs‐based representations can be combined with the above‐mentioned static phrase embeddings to further improve the accuracy of the bilingual phrase induction task. We manually construct a dataset of bilingual phrase pairs and integrate it with MUSE to facilitate the bilingual phrase induction task.

Double-Structured Sparsity Guided Flexible Embedding Learning for Unsupervised Feature Selection.

In this article, we propose a novel unsupervised feature selection model combined with clustering, named double-structured sparsity guided flexible embedding learning (DSFEL) for unsupervised feature selection. DSFEL includes a module for learning a block-diagonal structural sparse graph that represents the clustering structure and another module for learning a completely row-sparse projection matrix using the l2,0 -norm constraint to select distinctive features. Compared with the commonly used l2,1 -norm regularization term, the l2,0 -norm constraint can avoid the drawbacks of sparsity limitation and parameter tuning. The optimization of the l2,0 -norm constraint problem, which is a nonconvex and nonsmooth problem, is a formidable challenge, and previous optimization algorithms have only been able to provide approximate solutions. In order to address this issue, this article proposes an efficient optimization strategy that yields a closed-form solution. Eventually, through comprehensive experimentation on nine real-world datasets, it is demonstrated that the proposed method outperforms existing state-of-the-art unsupervised feature selection methods.

GABoost: Graph Alignment Boosting via Local Optimum Escape

Heterogeneous graphs provide a universal data structure for representing various kinds of structured data in numerous domains. The graph alignment problem aims to find the correspondences of vertices in different graphs, playing a fundamental role in many downstream tasks of heterogeneous graph mining. In recent years, many graph alignment methods have been proposed, ranging from classical optimization methods , spectral methods , to embedding learning based-methods . Due to the problem's complexity, the result found by most existing methods is either a heuristic solution or a critical point in the solution space. In this paper, we propose GABoost, a graph alignment boosting algorithm that takes as input an initial alignment between two heterogeneous graphs and outputs a boosted alignment via an iterative local-optimum-escape process. One of the distinctive features of GABoost is that it can be sequentially composed with any graph alignment methods to improve the output of upstream methods. To examine the effectiveness of GABoost, we select 7 upstream methods of graph alignment as well as 6 real-world datasets, and quantitatively investigate the degree to which GABoost boosts these methods. The results show that GABoost improves the alignment accuracy of the 7 upstream methods by 25.25% on average with acceptable time overhead.

Pairwise Intent Graph Embedding Learning for Context-Aware Recommendation with Knowledge Graph

Different from the data sparsity that traditional recommendations suffer from, context-aware recommender systems (CARS) face specific sparsity challenges related to contextual features, i.e., feature sparsity and interaction sparsity. How knowledge graphs address these challenges remains under-discussed. To bridge this gap, in this paper, we first propose a novel pairwise intent graph (PIG) containing nodes of users, items, entities, and enhanced intents to integrate knowledge graphs into CARS efficiently. Enhanced intent nodes are generated through the specific fusion of relational sub-intent and contextual sub-intent, and they are derived from semantic information and contextual information, respectively. We develop a pairwise intent graph embedding learning (PING) framework based on it. Specifically, our PING uses a pairwise intent joint graph convolution module to obtain refined embedding of all the features, where each enhanced intent node acts as a hub to effectively propagate information among different features and between all the features and knowledge graphs. Then, a recommendation module with refined embeddings is used to replace the randomly initialized embeddings of downstream recommendation models to improve model performance. Extensive experiments on three public datasets and some real-world scenarios verify the effectiveness and compatibility of our PING.

An implicit aspect-based sentiment analysis method using supervised contrastive learning and knowledge embedding

Aspect-based sentiment analysis aims to analyze and understand people’s opinions from different aspects. Some comments do not contain explicit opinion words but still convey a clear human-perceived emotional orientation, which is known as implicit sentiment. Most previous research relies on contextual information from a text for implicit aspect-based sentiment analysis. However, little work has integrated external knowledge with contextual information. This paper proposes an implicit aspect-based sentiment analysis model combining supervised contrastive learning with knowledge-enhanced fine-tuning on BERT (BERT-SCL+KEFT). In the pre-training phase, the model utilizes supervised contrastive learning (SCL) on large-scale sentiment-annotated corpora to acquire sentiment knowledge. In the fine-tuning phase, the model uses a knowledge-enhanced fine-tuning (KEFT) method to capture explicit and implicit aspect-based sentiments. Specifically, the model utilizes knowledge embedding to embed external general knowledge information into textual entities by using knowledge graphs, enriching textual information. Finally, the model combines external knowledge and contextual features to predict the implicit sentiment in a text. The experimental results demonstrate that the proposed BERT-SCL+KEFT model outperforms other baselines on the general implicit sentiment analysis and implicit aspect-based sentiment analysis tasks. In addition, ablation experimental results show that the proposed BERT-SCL+KEFT model without the knowledge embedding module or supervised contrastive learning module significantly decreases performance, indicating the importance of these modules. All experiments validate that the proposed BERT-SCL+KEFT model effectively achieves implicit aspect-based sentiment classification.

Distinguishing word identity and sequence context in DNA language models

Transformer-based large language models (LLMs) are very suited for biological sequence data, because of analogies to natural language. Complex relationships can be learned, because a concept of "words" can be generated through tokenization. Training the models with masked token prediction, they learn both token sequence identity and larger sequence context. We developed methodology to interrogate model learning, which is both relevant for the interpretability of the model and to evaluate its potential for specific tasks. We used DNABERT, a DNA language model trained on the human genome with overlapping k-mers as tokens. To gain insight into the model′s learning, we interrogated how the model performs predictions, extracted token embeddings, and defined a fine-tuning benchmarking task to predict the next tokens of different sizes without overlaps. This task evaluates foundation models without interrogating specific genome biology, it does not depend on tokenization strategies, vocabulary size, the dictionary, or the number of training parameters. Lastly, there is no leakage of information from token identity into the prediction task, which makes it particularly useful to evaluate the learning of sequence context. We discovered that the model with overlapping k-mers struggles to learn larger sequence context. Instead, the learned embeddings largely represent token sequence. Still, good performance is achieved for genome-biology-inspired fine-tuning tasks. Models with overlapping tokens may be used for tasks where a larger sequence context is of less relevance, but the token sequence directly represents the desired learning features. This emphasizes the need to interrogate knowledge representation in biological LLMs.

An exploration into CTEPH medications: Combining natural language processing, embedding learning, in vitro models, and real-world evidence for drug repurposing.

In the modern era, the growth of scientific literature presents a daunting challenge for researchers to keep informed of advancements across multiple disciplines. We apply natural language processing (NLP) and embedding learning concepts to design PubDigest, a tool that combs PubMed literature, aiming to pinpoint potential drugs that could be repurposed. Using NLP, especially term associations through word embeddings, we explored unrecognized relationships between drugs and diseases. To illustrate the utility of PubDigest, we focused on chronic thromboembolic pulmonary hypertension (CTEPH), a rare disease with an overall limited number of scientific publications. Our literature analysis identified key clinical features linked to CTEPH by applying term frequency-inverse document frequency (TF-IDF) scoring, a technique measuring a term's significance in a text corpus. This allowed us to map related diseases. One standout was venous thrombosis (VT), which showed strong semantic links with CTEPH. Looking deeper, we discovered potential repurposing candidates for CTEPH through large-scale neural network-based contextualization of literature and predictive modeling on both the CTEPH and the VT literature corpora to find novel, yet unrecognized associations between the two diseases. Alongside the anti-thrombotic agent caplacizumab, benzofuran derivatives were an intriguing find. In particular, the benzofuran derivative amiodarone displayed potential anti-thrombotic properties in the literature. Our in vitro tests confirmed amiodarone's ability to reduce platelet aggregation significantly by 68% (p = 0.02). However, real-world clinical data indicated that CTEPH patients receiving amiodarone treatment faced a significant 15.9% higher mortality risk (p<0.001). While NLP offers an innovative approach to interpreting scientific literature, especially for drug repurposing, it is crucial to combine it with complementary methods like in vitro testing and real-world evidence. Our exploration with benzofuran derivatives and CTEPH underscores this point. Thus, blending NLP with hands-on experiments and real-world clinical data can pave the way for faster and safer drug repurposing approaches, especially for rare diseases like CTEPH.