Embedding Learning Research Articles

Fast, Accurate, and Lightweight Memory-Enhanced Embedding Learning Framework for Image-Text Retrieval

Fast, Accurate, and Lightweight Memory-Enhanced Embedding Learning Framework for Image-Text Retrieval

Matching Patients to Clinical Trials using LLaMA 2 Embeddings and Siamese Neural Network.

Patient recruitment is a key desideratum for the success of a clinical trial that entails identifying eligible patients that match the selection criteria for the trial. However, the complexity of criteria information and heterogeneity of patient data render manual analysis a burdensome and time-consuming task. In an attempt to automate patient recruitment, this work proposes a Siamese Neural Network-based model, namely Siamese-PTM. Siamese-PTM employs the pretrained LLaMA 2 model to derive contextual representations of the EHR and criteria inputs and jointly encodes them using two weight-sharing identical subnetworks. We evaluate Siamese-PTM on structured and unstructured EHR to analyze their predictive informativeness as standalone and collective feature sets. We explore a variety of deep models for Siamese-PTM's encoders and compare their performance against the Single-encoder counterparts. We develop a baseline rule-based classifier, compared to which Siamese-PTM improved performance by 40%. Furthermore, visualization of Siamese-PTM's learned embedding space reinforces its predictive robustness.

Bijective upsampling and learned embedding for point clouds correspondences

In this paper, we present a novel pipeline to compute and refine a data-driven solution for estimating the correspondence between 3D point clouds. Our method is compatible with the functional map framework, so it relies on a functional representation of the correspondence, but, differently from other similar approaches, this method is specifically designed to exploit this functional scenario for point cloud matching. Our new method merges a data-driven approach to compute functional basis and descriptors on the shape’s surface and a new refinement method designed for the learned basis. This refinement algorithm arises from a different way of converting functional operators into point-to-point correspondence, which we prove to promote bijectivity between maps, exploiting a theoretical result. Iterating this procedure and performing basis upsampling in the same way as other similar methods, ours increases the accuracy of the correspondence, leading to more bijective correspondences. Different from other approaches, our method allows us to train a functional basis, considering the refinement stage. Combining our new pipeline with an improved feature extractor, our solution outperforms previous methods in various evaluations and settings. We test our method over different datasets, comprising near-isometric and non-isometric pairs.

Class-incremental learning with Balanced Embedding Discrimination Maximization

Class incremental learning is committed to solving representation learning and classification assignments while avoiding catastrophic forgetting in scenarios where categories are increasing. In this work, a unified method named Balanced Embedding Discrimination Maximization (BEDM) is developed to make the intermediate embedding more distinctive. Specifically, we utilize an orthogonality constraint based on doubly-blocked Toeplitz matrix to minimize the correlation of convolution kernels, and an algorithm for similarity visualization is introduced. Furthermore, uneven samples and distribution shift among old and new tasks eventuate strongly biased classifiers. To mitigate the imbalance, we propose an adaptive balance weighting in softmax to compensate insufficient categories dynamically. In addition, hybrid embedding learning is introduced to preserve knowledge from old models, which involves less hyper-parameters than conventional knowledge distillation. Our proposed method outperforms the existing approaches on three mainstream benchmark datasets. Moreover, we technically visualize that our method can produce a more uniform similarity histogram and more stable spectrum. Grad-CAM and t-SNE visualizations further confirm its effectiveness. Code is available at https://github.com/wqzh/BEDM.

TODEAR: Promoting explainable TKG reasoning through temporal offset enhanced dynamic embedding and adaptive reinforcement learning

Recently, Reinforcement Learning (RL) is utilized in Temporal Knowledge Graph (TKG) reasoning to generate analyzable reasoning paths, which achieves explainable reasoning on queries. The process of generating high-quality reasoning paths is facing two major challenges. The first is to construct an efficient entity embedding model from the complex temporal dependencies among entities. The second is to design a fine-grained reward mechanism for the reasoner based on the deep semantics of exploration paths. Motivated by the two challenges, a TKG reasoning framework based on Temporal Offset Enhanced Dynamic Embedding and Adaptive Reinforcement Learning (TODEAR) is proposed in this paper. Firstly, a temporal offset enhanced dynamic embedding model with a distance scoring function is designed to fully exploit the complex temporal dependencies among entities. In order to capture the evolutionary patterns of historical facts, it encodes both the relational structures of the TKG and the temporal offsets between events and queries. Then, a fine-grained adaptive reward mechanism is designed to optimize the reasoner. It generates real-time rewards by analyzing the logic and semantics of exploration paths to mitigate the adverse effects of sparse rewards. Extensive experiments on four benchmark datasets show that TODEAR significantly outperforms the state-of-the-art methods.

Explainable Histopathology Image Classification with Self-organizing Maps: A Granular Computing Perspective

The automatic analysis of histology images is an open research field where machine learning techniques and neural networks, especially deep architectures, are considered successful tools due to their abilities in image classification. This paper proposes a granular computing methodology for histopathological image classification. It is based on embedding tiles of histopathology images using deep metric learning, where a self-organizing map is adopted to generate the granular structure in this learned embedding space. The SOM enables the implementation of an explainable mechanism by visualizing a knowledge space that the experts can use to analyze and classify the new images. Additionally, it provides confidence in the classification results while highlighting each important image fragment, with the benefit of reducing the number of false negatives. An exemplary case is when an image detail is indicated, with small confidence, as malignant in an image globally classified as benign. Another implemented feature is the proposal of additional labelled image tiles sharing the same characteristics to specify the context of the output decision. The proposed system was tested using three histopathology image datasets, obtaining the accuracy of the state-of-the-art black-box methods based on deep learning neural networks. Differently from the methodologies proposed so far for the same purpose, this paper introduces a novel explainable method for medical image analysis where the advantages of the deep learning neural networks used to build the embedding space for the image tiles are combined with the intrinsic explainability of the granular process obtained using the clustering property of a self-organizing map.

Meta-path structured graph pre-training for improving knowledge tracing in intelligent tutoring

Knowledge tracing (KT) aims to predict students’ future performance by tracking their learning behaviors in intelligent tutoring systems (ITS). In KT, three main types of entities are involved: students, exercises, and knowledge concepts. Graph structures provide an effective framework for establishing relationships between different entities. However, existing KT methods have neglected complex connections and implicit relational features, thus facing challenges in capturing high-order information in educational data. To this end, this paper proposes MPSG, a Meta-Path Structured Graph method designed to harness high-order information between entities to pre-train informative exercise embeddings for improving KT. Technically, structured by different meta-paths, four relation graphs are derived to establish implicit cross-entity relationships. On each graph, the surrounding neighbors of each node are then obtained through an intimacy-based sampling strategy. Subsequently, during the representation learning stage, node features under different meta-path views are generated and then aggregated to obtain the final exercise embeddings. The learned embeddings are optimized through three auxiliary tasks, anchored within a self-supervised learning paradigm. Extensive experiments across four public datasets demonstrate that our method can significantly enhance the predictive performance of downstream KT models.

Graphormer-IR: Graph Transformers Predict Experimental IR Spectra Using Highly Specialized Attention.

Infrared (IR) spectroscopy is an important analytical tool in various chemical and forensic domains and a great deal of effort has gone into developing in silico methods for predicting experimental spectra. A key challenge in this regard is generating highly accurate spectra quickly to enable real-time feedback between computation and experiment. Here, we employ Graphormer, a graph neural network (GNN) transformer, to predict IR spectra using only simplified molecular-input line-entry system (SMILES) strings. Our data set includes 53,528 high-quality spectra, measured in five different experimental media (i.e., phases), for molecules containing the elements H, C, N, O, F, Si, S, P, Cl, Br, and I. When using only atomic numbers for node encodings, Graphormer-IR achieved a mean test spectral information similarity (SISμ) value of 0.8449 ± 0.0012 (n = 5), which surpasses that the current state-of-the-art model Chemprop-IR (SISμ = 0.8409 ± 0.0014, n = 5) with only 36% of the encoded information. Augmenting node embeddings with additional node-level descriptors in learned embeddings generated through a multilayer perceptron improves scores to SISμ = 0.8523 ± 0.0006, a total improvement of 19.7σ (t = 19). These improved scores show how Graphormer-IR excels in capturing long-range interactions like hydrogen bonding, anharmonic peak positions in experimental spectra, and stretching frequencies of uncommon functional groups. Scaling our architecture to 210 attention heads demonstrates specialist-like behavior for distinct IR frequencies that improves model performance. Our model utilizes novel architectures, including a global node for phase encoding, learned node feature embeddings, and a one-dimensional (1D) smoothing convolutional neural network (CNN). Graphormer-IR's innovations underscore its value over traditional message-passing neural networks (MPNNs) due to its expressive embeddings and ability to capture long-range intramolecular relationships.

High‐degree penalty based global statistical network embedding for name disambiguation in anonymized graph

SummaryIn person‐centric applications, the prevalence of shared names significantly hampers document retrieval, web search, and database integration, highlighting the critical need for name disambiguation. Network embedding based unsupervised name disambiguation methods have received much attention due to their wide applicability and superior disambiguation performance. However, existing methods require complex feature engineering, such as extracting biographical and source content features or constructing supplementary features from external knowledge bases, which are unavailable in privacy‐preserving scenarios. Moreover, they may face challenges such as imbalanced node training or overlooking global statistical information during node embedding learning. In this article, we propose a method to tackle the name disambiguation problem based on high‐degree penalty and global statistical information using only relational data. First, we construct a weighted source similarity network based on multi‐hop collaborator relationships. Second, we employ a high‐degree penalty based global statistical network embedding model to learn low‐dimensional node embeddings and preserve the structural features of the network. Finally, we cluster the same sources using a joint clustering algorithm that does not require prior knowledge of the number of clusters. The experiment validates the effectiveness of the proposed method on two real‐life datasets.

Unsupervised Feature Selection via Collaborative Embedding Learning

Unsupervised Feature Selection via Collaborative Embedding Learning

Autobalanced Multitask Node Embedding Framework for Intelligent Education.

Recently, online education has become popular. Many e-learning platforms have been launched with various intelligent services aimed at improving the learning efficiency and effectiveness of learners. Graphs are used to describe the pairwise relations between entities, and the node embedding technique is the foundation of many intelligent services, which have received increasing attention from researchers. However, the graph in the intelligent education scenario has three noteworthy properties, namely, heterogeneity, evolution, and lopsidedness, which makes it challenging to implement ecumenical node embedding methods on it. In this article, an autobalanced multitask node embedding model is proposed, named MNE, and applied to the interaction graph, settling a few actual tasks in intelligent education. More specifically, MNE builds two purpose-built self-supervised node embedding learning tasks for heterogeneous evolutive graphs. Edge-specific reconstruction tasks are built according to the semantic information and properties of the heterogeneous edges, and an evolutive weight regression task is designed, aiding the model to perceive the evolution of learners' implicit cognitive states. Then, both aleatoric and epistemic uncertainty quantification techniques are introduced, achieving both task- and node-level weight estimation and instructing subtask autobalancing. Experimental results on real-world datasets indicate that the proposed model outperforms the state-of-the-art graph embedding methods on two assessment tasks and demonstrates the validity of the proposed multitask framework and subtask balancing mechanism. Our implementations are available at https://github.com/ccnu-mathits/MNE4HEN.

View-unaligned clustering with graph regularization

In current multi-view clustering modeling scenarios, the cross-view correspondence of the data is generally presumed in advance. However, this assumption is inevitably violated in practical applications as each view is independently processed during data collection and transmission, thus resulting in the view-unaligned problem (VuP). The absence of cross-view correspondence between the data renders most existing multi-view clustering methods ineffective in addressing the VuP. To address this problem, we propose a novel view-unaligned clustering method, termed View-unaligned Clustering with Graph regularization (VuCG), which performs latent embedding learning on manifold, latent embedding alignment and partition generation in a one-stage manner. Specifically, we implement the latent embedding learning on manifold by seeking a matrix factorization respecting the graph structure, and perform latent embedding alignment by identifying the counterpart from a selected template of the shuffled embedding using global and local information of the data in the latent space. Additionally, we obtain the partition of the aligned data by applying the relaxed k-means algorithm to the aligned embeddings. Extensive experimental results on four practical datasets demonstrate the effectiveness of the proposed method.

Multi-View Graph Embedding Learning for Image Co-Segmentation and Co-Localization

Multi-View Graph Embedding Learning for Image Co-Segmentation and Co-Localization

GE 2 : A General and Efficient Knowledge Graph Embedding Learning System

Graph embedding learning computes an embedding vector for each node in a graph and finds many applications in areas such as social networks, e-commerce, and medicine. We observe that existing graph embedding systems (e.g., PBG, DGL-KE, and Marius) have long CPU time and high CPU-GPU communication overhead, especially when using multiple GPUs. Moreover, it is cumbersome to implement negative sampling algorithms on them, which have many variants and are crucial for model quality. We propose a new system called GE 2 , which achieves both <u>g</u>enerality and <u>e</u>fficiency for <u>g</u>raph <u>e</u>mbedding learning. In particular, we propose a general execution model that encompasses various negative sampling algorithms. Based on the execution model, we design a user-friendly API that allows users to easily express negative sampling algorithms. To support efficient training, we offload operations from CPU to GPU to enjoy high parallelism and reduce CPU time. We also design COVER, which, to our knowledge, is the first algorithm to manage data swap between CPU and multiple GPUs for small communication costs. Extensive experimental results show that, comparing with the state-of-the-art graph embedding systems, GE 2 trains consistently faster across different models and datasets, where the speedup is usually over 2x and can be up to 7.5x.

Learning hierarchical embedding space for image-text matching

There are two mainstream strategies for image-text matching at present. The one, termed as joint embedding learning, aims to model the semantic information of both image and sentence in a shared feature subspace, which facilitates the measurement of semantic similarity but only focuses on global alignment relationship. To explore the local semantic relationship more fully, the other one, termed as metric learning, aims to learn a complex similarity function to directly output score of each image-text pair. However, it significantly suffers from more computation burden at retrieval stage. In this paper, we propose a hierarchically joint embedding model to incorporate the local semantic relationship into a joint embedding learning framework. The proposed method learns the shared local and global embedding spaces simultaneously, and models the joint local embedding space with respect to specific local similarity labels which are easy to access from the lexical information of corpus. Unlike the methods based on metric learning, we can prepare the fixed representations of both images and sentences by concatenating the normalized local and global representations, which makes it feasible to perform the efficient retrieval. And experiments show that the proposed model can achieve competitive performance when compared to the existing joint embedding learning models on two publicly available datasets Flickr30k and MS-COCO.

Emle-engine: A Flexible Electrostatic Machine Learning Embedding Package for Multiscale Molecular Dynamics Simulations.

We present in this work the emle-engine package (https://github.com/chemle/emle-engine)─the implementation of a new machine learning embedding scheme for hybrid machine learning potential/molecular-mechanics (ML/MM) dynamics simulations. The package is based on an embedding scheme that uses a physics-based model of the electronic density and induction with a handful of tunable parameters derived from in vacuo properties of the subsystem to be embedded. This scheme is completely independent of the in vacuo potential and requires only the positions of the atoms of the machine learning subsystem and the positions and partial charges of the molecular mechanics environment. These characteristics allow emle-engine to be employed in existing QM/MM software. We demonstrate that the implemented electrostatic machine learning embedding scheme (named EMLE) is stable in enhanced sampling molecular dynamics simulations. Through the calculation of free energy surfaces of alanine dipeptide in water with two different ML options for the in vacuo potential and three embedding models, we test the performance of EMLE. When compared to the reference DFT/MM surface, the EMLE embedding is clearly superior to the MM one based on fixed partial charges. The configurational dependence of the electronic density and the inclusion of the induction energy introduced by the EMLE model leads to a systematic reduction in the average error of the free energy surface when compared to MM embedding. By enabling the usage of EMLE embedding in practical ML/MM simulations, emle-engine will make it possible to accurately model systems and processes that feature significant variations in the charge distribution of the ML subsystem and/or the interacting environment.

Learning Personalizable Clustered Embedding for Recommender Systems

In recent years, recommender systems have advanced rapidly, where embedding learning for users and items plays a critical role. A standard method learns a unique embedding vector for each user and item. However, such a method has two important limitations in real-world applications: 1) it is hard to learn embeddings that generalize well for users and items with rare interactions; and 2) it may incur unbearably high memory costs when the number of users and items scales up. Existing approaches either can only address one of the limitations or have flawed overall performances. In this paper, we propose Clustered Embedding Learning (CEL) as an integrated solution to these two problems. CEL is a plug-and-play embedding learning framework that can be combined with any differentiable feature interaction model. It is capable of achieving improved performance, especially for cold users and items, with reduced memory cost. CEL enables automatic and dynamic clustering of users and items in a top-down fashion, where clustered entities could jointly learn a shared embedding. The accelerated version of CEL has an optimal time complexity, which supports efficient online updates. Theoretically, we prove the identifiability and the existence of a unique optimal number of clusters for CEL in the context of nonnegative matrix factorization. Empirically, we validate the effectiveness of CEL on three public datasets and one business dataset, showing its consistently superior performance against current state-of-the-art methods. In particular, when incorporating CEL into the business model, it brings an improvement of \(+0.6\% \) in AUC, which translates into a significant revenue gain; meanwhile, the size of the embedding table gets 2650 times smaller. Additionally, we demonstrate that, if there is enough memory, learning a personalized embedding for each user and item around their clustering centers is feasible and can further boost the performance.

Embedding Learning as a Practice of Value: Learning from the Experiences of Early Career Social Workers in Scotland

Abstract Across the Global North, professional learning for social workers has become a matter of concern. Efforts to ‘fix’ the problem have focused narrowly on formal methods of learning and show limited connect with recent research. In this article, we report on findings from a mixed-method longitudinal cohort study, which examined early career social workers experiences of learning over the first five years of professional practice. Drawing on a repeat-measure annual online survey, our findings provide an inside-view of how early career social workers in Scotland experience work-based learning over time and how professional learning for social workers can be enhanced. Quantitative and qualitative data were analysed using descriptive statistical analysis and reflexive thematic analysis respectively and integrated using a convergence coding matrix to identify meta-themes. Our findings support an integrative, developmental and ecological approach to professional learning, embedded in a value-led understanding of social work as practice. We invite the profession to embrace and embed learning as a ‘practice of value’, both as an antidote to managerial approaches to practice and learning and as a way of valuing the extraordinary work that social workers do.

A heterogeneous graph-based semi-supervised learning framework for access control decision-making

For modern information systems, robust access control mechanisms are vital in safeguarding data integrity and ensuring the entire system’s security. This paper proposes a novel semi-supervised learning framework that leverages heterogeneous graph neural network-based embedding to encapsulate both the intricate relationships within the organizational structure and interactions between users and resources. Unlike existing methods focusing solely on individual user and resource attributes, our approach embeds organizational and operational interrelationships into the hidden layer node embeddings. These embeddings are learned from a self-supervised link prediction task based on a constructed access control heterogeneous graph via a heterogeneous graph neural network. Subsequently, the learned node embeddings, along with the original node features, serve as inputs for a supervised access control decision-making task, facilitating the construction of a machine-learning access control model. Experimental results on the open-sourced Amazon access control dataset demonstrate that our proposed framework outperforms models using original or manually extracted graph-based features from previous works. The prepossessed data and codes are available on GitHub,facilitating reproducibility and further research endeavors.

Multi-perspective learning for enhanced user preferences for session-based recommendation

Due to the sparsity and anonymity of interactions within sessions, capturing user preferences by modeling item relationships poses a significant challenge. Previous methods have attempted to utilize collaborative information from adjacent sessions or item attribute knowledge as supplements. However, these methods are prone to introducing interfering information through their session similarity measurement strategies and attribute embedding learning processes, both requiring optimization. Moreover, they often fail to fully model global item relationships. To address these issues, we propose a novel method called Multi-perspective Learning for Enhanced User Preferences (MLEUP) for session-based recommendations. Specifically, we first construct reliable session groups by measuring the overlap and incompatibility of items across sessions to calculate session similarity. Then, by modeling attribute–item and attribute–attribute relationships, along with employing a neighbor-based contrastive self-supervised attribute learning task, we learn high-quality attribute embeddings that capture the user’s stable preferences. Next, leveraging the session groups, we model global item multi-granularity co-occurrence and high-order sequential relationships through designed attention layers, enabling us to learn informative item embeddings that represent the user’s dynamic preferences. Finally, we further model the correlated relationship between the two user preference representations. Our approach ensures that effective attribute knowledge and global collaborative information are learned to mitigate the data sparsity issue, while comprehensively modeling global item relationships to capture the user’s complex dynamic preferences. Experimental results demonstrate that MLEUP achieves state-of-the-art performance in terms of precision, MRR, and NDCG. The experimental datasets and source code of MLEUP are publicly available at https://github.com/lean030/MLEUP.