Embedding Space Research Articles

Disentangling User Interest and Geographical Context for POI Recommendations

POI recommendation plays an important role in many applications, such as mobility prediction and location-based advertisements. Existing POI recommendation methods mainly capture the observed patterns in user visits for recommendations, without a comprehensive consideration of the underlying reasons behind the visits. Therefore, different causes of a visit, i.e., users’ interest and geographical context, are entangled. When the underlying causes change (e.g., when a user moves to a new place), the robustness of the recommendations cannot be guaranteed. To address the above challenges, we propose DUIG, a novel user interest and geographical influences disentanglement framework for POI recommendations. We first design a personalized disentanglement strategy to divide check-ins through geographical influence. Specifically, the colliding effect of causality is leveraged to the divide cause-specific check-ins, such that user interest and geographical influence can be properly disentangled in user and POI embeddings. Through this mechanism, even if the underlying reasons that affect a user’s preference change, intervention can be conducted upon the causes to make recommendations generalized to the new scenario. In addition, a geographical-aware negative sampling strategy is proposed to utilize hard negatives to regularize the embedding and disentanglement in the latent space, where a larger sampling probability is introduced for negative samples containing more geographic information. Extensive experiments on two real-world POI recommendation datasets demonstrate the superior performance of DUIG.

Can machine learning help accelerate article screening for systematic reviews? Yes, when article separability in embedding space is high

Abstract Systematic reviews play important roles but manual efforts can be time-consuming given a growing literature. There is a need to use and evaluate automated strategies to accelerate systematic reviews. Here, we comprehensively tested machine learning (ML) models from classical and deep learning model families. We also assessed the performance of prompt engineering via few-shot learning of GPT-3.5 and GPT-4 large language models (LLMs). We further attempted to understand when ML models can help automate screening. These ML models were applied to actual datasets of systematic reviews in education. Results showed that the performance of classical and deep ML models varied widely across datasets, ranging from 1.2 to 75.6% of work saved at 95% recall. LLM prompt engineering produced similarly wide performance variation. We searched for various indicators of whether and how ML screening can help. We discovered that the separability of clusters of relevant versus irrelevant articles in high-dimensional embedding space can strongly predict whether ML screening can help (overall R = 0.81). This simple and generalizable heuristic applied well across datasets and different ML model families. In conclusion, ML screening performance varies tremendously, but researchers and software developers can consider using our cluster separability heuristic in various ways in an ML-assisted screening pipeline.

Core dimensions of human material perception

Visually categorizing and comparing materials is crucial for everyday behavior, but what organizational principles underlie our mental representation of materials? Here, we used a large-scale data-driven approach to uncover core latent dimensions of material representations from behavior. First, we created an image dataset of 200 systematically sampled materials and 600 photographs (STUFF dataset, https://osf.io/myutc/). Using these images, we next collected 1.87 million triplet similarity judgments and used a computational model to derive a set of sparse, positive dimensions underlying these judgments. The resulting multidimensional embedding space predicted independent material similarity judgments and the similarity matrix of all images close to the human intersubject consistency. We found that representations of individual images were captured by a combination of 36 material dimensions that were highly reproducible and interpretable, comprising perceptual (e.g., grainy, blue) as well as conceptual (e.g., mineral, viscous) dimensions. These results provide the foundation for a comprehensive understanding of how humans make sense of materials.

Revealing the Relationship between Publication Bias and Chemical Reactivity with Contrastive Learning.

A synthetic method's substrate tolerance and generality are often showcased in a "substrate scope" table. However, substrate selection exhibits a frequently discussed publication bias: unsuccessful experiments or low-yielding results are rarely reported. In this work, we explore more deeply the relationship between such a publication bias and chemical reactivity beyond the simple analysis of yield distributions using a novel neural network training strategy, substrate scope contrastive learning. By treating reported substrates as positive samples and nonreported substrates as negative samples, our contrastive learning strategy teaches a model to group molecules within a numerical embedding space, based on historical trends in published substrate scope tables. Training on 20,798 aryl halides in the CAS Content CollectionTM, spanning thousands of publications from 2010 to 2015, we demonstrate that the learned embeddings exhibit a correlation with physical organic reactivity descriptors through both intuitive visualizations and quantitative regression analyses. Additionally, these embeddings are applicable to various reaction modeling tasks like yield prediction and regioselectivity prediction, underscoring the potential to use historical reaction data as a pretraining task. This work not only presents a chemistry-specific machine learning training strategy to learn from literature data in a new way but also represents a unique approach to uncover trends in chemical reactivity reflected by trends in substrate selection in publications.

Imaginary-Connected Embedding in Complex Space for Unseen Attribute-Object Discrimination.

Compositional Zero-Shot Learning (CZSL) aims to recognize novel compositions of seen primitives. Prior studies have attempted to either learn primitives individually (non-connected) or establish dependencies among them in the composition (fully-connected). In contrast, human comprehension of composition diverges from the aforementioned methods as humans possess the ability to make composition-aware adaptation for these primitives, instead of inferring them rigidly through the aforementioned methods. However, developing a comprehension of compositions akin to human cognition proves challenging within the confines of real space. This arises from the limitation of real-space-based methods, which often categorize attributes, objects, and compositions using three independent measures, without establishing a direct dynamic connection. To tackle this challenge, we expand the CZSL distance metric scheme to encompass complex spaces to unify the independent measures, and we establish an imaginary-connected embedding in complex space to model human understanding of attributes. To achieve this representation, we introduce an innovative visual bias-based attribute extraction module that selectively extracts attributes based on object prototypes. As a result, we are able to incorporate phase information in training and inference, serving as a metric for attribute-object dependencies while preserving the independent acquisition of primitives. We evaluate the effectiveness of our proposed approach on three benchmark datasets, illustrating its superiority compared to baseline methods. Our code is available at https://github.com/LanchJL/IMAX.

Deep clustering analysis via variational autoencoder with Gamma mixture latent embeddings

Deep clustering analysis via variational autoencoder with Gamma mixture latent embeddings

MDMNet: Multi-dimensional multi-modal network to identify organ system limitation in cardiopulmonary exercise testing.

MDMNet: Multi-dimensional multi-modal network to identify organ system limitation in cardiopulmonary exercise testing.

CareerBERT: Matching resumes to ESCO jobs in a shared embedding space for generic job recommendations

CareerBERT: Matching resumes to ESCO jobs in a shared embedding space for generic job recommendations

Enhancing cross-domain recommendations: Leveraging personality-based transfer learning with probabilistic matrix factorization

Enhancing cross-domain recommendations: Leveraging personality-based transfer learning with probabilistic matrix factorization

Delayed knowledge transfer: Cross-modal knowledge transfer from delayed stimulus to EEG for continuous attention detection based on spike-represented EEG signals.

Delayed knowledge transfer: Cross-modal knowledge transfer from delayed stimulus to EEG for continuous attention detection based on spike-represented EEG signals.

Auxiliary meta-learning strategy for cancer recognition: leveraging external data and optimized feature mapping

As reported by the International Agency for Research on Cancer (IARC), the global incidence of cancer reached nearly 20 million new cases in recent years, with cancer-related fatalities amounting to around 9.7 million. This underscores the profound impact cancer has on public health worldwide. Deep learning has become a mainstream approach in cancer recognition. Despite its significant progress, deep learning is known for its requirement of large quantities of labeled data. Few-shot learning addresses this limitation by reducing the need for extensive labeled samples. In the field of cancer recognition, data collection is particularly challenging due to the scarcity of categories compared to other fields, and current few-shot learning methods have not yielded satisfactory results. To tackle this, we propose an auxiliary meta-learning strategy for cancer recognition. During the auxiliary training phase, the feature mapping model is trained in conjunction with external data. This process neutralizes the prediction probability of misclassification, allowing the model to more readily learn distinguishing features and avoid performance degradation caused by discrepancies in external data. Additionally, the redundancy of some input principal components in the feature mapping model is reduced, while the implicit information within these components is extracted. The training process is further accelerated by utilizing depthwise over-parameterized convolutional layers. Moreover, the implementation of a three-branch structure contributes to faster training and enhanced performance. In the meta-training stage, the feature mapping model is optimized within the embedding space, utilizing category prototypes and cosine distance. During the meta-testing phase, a small number of labeled samples are employed to classify unknown data. We have conducted extensive experiments on the BreakHis, Pap smear, and ISIC 2018 datasets. The results demonstrate that our method achieves superior accuracy in cancer recognition. Furthermore, experiments on few-shot benchmark datasets indicate that our approach exhibits excellent generalization capabilities.

DM2RM: dual-mode multimodal ranking for target objects and receptacles based on open-vocabulary instructions

In this study, we aim to develop a domestic service robot (DSR) that, guided by open-vocabulary instructions, can carry everyday objects to the specified pieces of furniture. Few existing methods handle mobile manipulation tasks with open-vocabulary instructions in the image retrieval setting, and most do not identify both the target objects and the receptacles. We propose the Dual-Mode Multimodal Ranking model (DM 2 RM), which enables images of both the target objects and receptacles to be retrieved using a single model based on multimodal foundation models. We introduce a switching mechanism that leverages a mode token and phrase identification via a large language model to switch the embedding space based on the prediction target. To evaluate the DM 2 RM, we construct a novel dataset including real-world images collected from hundreds of building-scale environments and crowd-sourced instructions with referring expressions. The evaluation results show that the proposed DM 2 RM outperforms previous approaches in terms of standard metrics in image retrieval settings. Furthermore, we demonstrate the application of the DM 2 RM on a standardized real-world DSR platform including fetch-and-carry actions, where it achieves a task success rate of 82% despite the zero-shot transfer setting. Demonstration videos, code, and more materials are available at https://kkrr10.github.io/dm2rm/.

AlphaFold as a Prior: Experimental Structure Determination Conditioned on a Pretrained Neural Network.

Advances in machine learning have transformed structural biology, enabling swift and accurate prediction of protein structure from sequence. However, challenges persist in capturing sidechain packing, condition-dependent conformational dynamics, and biomolecular interactions, primarily due to scarcity of high-quality training data. Emerging techniques, including cryo-electron tomography (cryo-ET) and high-throughput crystallography, promise vast new sources of structural data, but translating raw experimental observations into mechanistically interpretable atomic models remains a key bottleneck. Here, we aim to address these challenges by improving the efficiency of structural analysis through combining experimental measurements with a landmark protein structure prediction method - AlphaFold2. We present an augmentation of AlphaFold2, ROCKET, that refines its predictions using cryo-EM, cryo-ET, and X-ray crystallography data, and demonstrate that this approach captures biologically important structural variation that AlphaFold2 does not. By performing structure optimization in the space of coevolutionary embeddings, rather than Cartesian coordinates, ROCKET automates difficult modeling tasks, such as flips of functional loops and domain rearrangements, that are beyond the scope of current state-of-the-art methods and, in some instances, even manual human modeling. The ability to efficiently sample these barrier-crossing rearrangements unlocks a new horizon for scalable and automated model building. Crucially, ROCKET does not require retraining of AlphaFold2 and is readily adaptable to multimers, ligand-cofolding, and other data modalities. Conversely, our differentiable crystallographic and cryo-EM target functions are capable of augmenting other structure prediction methods. ROCKET thus provides an extensible framework for the integration of experimental observables with biomolecular machine learning.

TongueTransUNet: toward effective tongue contour segmentation using well-managed dataset.

In modern telehealth and healthcare information systems medical image analysis is essential to understand the context of the images and its complex structure from large, inconsistent-quality, and distributed datasets. Achieving desired results faces a few challenges for deep learning. Examples of these challenges are date size, labeling, balancing, training, and feature extraction. These challenges made the AI model complex and expensive to be built and difficult to understand which made it a black box and produce hysteresis and irrelevant, illegal, and unethical output in some cases. In this article, lingual ultrasound is studied to extract tongue contour to understand language behavior and language signature and utilize it as biofeedback for different applications. This article introduces a design strategy that can work effectively using a well-managed dynamic-size dataset. It includes a hybrid architecture using UNet, Vision Transformer (ViT), and contrastive loss in latent space to build a foundation model cumulatively. The process starts with building a reference representation in the embedding space using human experts to validate any new input for training data. UNet and ViT encoders are used to extract the input feature representations. The contrastive loss was then compared to the new feature embedding with the reference in the embedding space. The UNet-based decoder is used to reconstruct the image to its original size. Before releasing the final results, quality control is used to assess the segmented contour, and if rejected, the algorithm requests an action from a human expert to annotate it manually. The results show an improved accuracy over the traditional techniques as it contains only high quality and relevant features.

A unified analysis of atlas single cell data.

Recent efforts to generate atlas-scale single-cell data provide opportunities for joint analysis across tissues and modalities. Existing methods use cells as the reference unit, hindering downstream gene-based analysis and removing genuine biological variations. Here we present GIANT, an integration method designed for atlas-scale gene analysis across cell types and tissues. GIANT converts datasets into gene graphs and recursively embeds genes without additional alignment. Applying GIANT to two recent atlas datasets yields unified gene embedding spaces across human tissues and data modalities. Further evaluations demonstrate GIANT's usefulness in discovering diverse gene functions and underlying gene regulations in cells from different tissues.

Learning maximally spanning representations improves protein function annotation.

Automated protein function annotation is a fundamental problem in computational biology, crucial for understanding the functional roles of proteins in biological processes, with broad implications in medicine and biotechnology. A persistent challenge in this problem is the imbalanced, long-tail distribution of available function annotations: a small set of well-studied function classes account for most annotated proteins, while many other classes have few annotated proteins, often due to investigative bias, experimental limitations, or intrinsic biases in protein evolution. As a result, existing machine learning models for protein function prediction tend to only optimize the prediction accuracy for well-studied function classes overrepresented in the training data, leading to poor accuracy for understudied functions. In this work, we develop MSRep, a novel deep learning-based protein function annotation framework designed to address this imbalance issue and improve annotation accuracy. MSRep is inspired by an intriguing phenomenon, called neural collapse (NC), commonly observed in high-accuracy deep neural networks used for classification tasks, where hidden representations in the final layer collapse to class-specific mean embeddings, while maintaining maximal inter-class separation. Given that NC consistently emerges across diverse architectures and tasks for high-accuracy models, we hypothesize that inducing NC structure in models trained on imbalanced data can enhance both prediction accuracy and generalizability. To achieve this, MSRep refines a pre-trained protein language model to produce NC-like representations by optimizing an NC-inspired loss function, which ensures that minority functions are equally represented in the embedding space as majority functions, in contrast to conventional classification methods whose embedding spaces are dominated by overrepresented classes. In evaluations across four protein function annotation tasks on the prediction of Enzyme Commission numbers, Gene3D codes, Pfam families, and Gene Ontology terms, MSRep demonstrates superior predictive performance for both well- and underrepresented classes, outperforming several state-of-the-art annotation tools. We anticipate that MSRep will enhance the annotation of understudied functions and novel, uncharacterized proteins, advancing future protein function studies and accelerating the discovery of new functional proteins. The source code of MSRep is available at https://github.com/luo-group/MSRep .

Balancing Embedding Spectrum for Recommendation

Modern recommender systems heavily rely on high-quality representations learned from high-dimensional sparse data. While significant efforts have been invested in designing powerful algorithms for extracting user preferences, the factors contributing to good representations have remained relatively unexplored. In this work, we shed light on an issue in the existing pair-wise learning paradigm ( i.e., embedding collapse), that the representations tend to span a subspace of the whole embedding space, leading to a suboptimal solution and reducing the model capacity. Specifically, we show that alignment of positive pairs is equivalent to a low pass filter causing users/items to collapse to a constant vector. While negative sampling can partially mitigate this issue by acting as a high pass filter to balance the spectrum, leading to an incomplete collapse. To tackle this issue, we present a novel learning paradigm DirectSpec, which directly optimizes the spectrum distribution to ensure that users/items effectively span the entire embedding space. We demonstrate that many self-supervised learning algorithms without explicit negative sampling can be considered as special cases of DirectSpec. Furthermore, we show that optimizing the spectrum inappropriately could also be poisonous to data representation, where the key lies in a dynamic balance between alignment of positive pairs and spectrum balancing. Finally, we propose an enhanced and practical implementation DirectSpec + to balance the embedding spectrum more adaptively and effectively. We implement DirectSpec + on two popular recommender models: MF and LightGCN. Our experimental results demonstrate its effectiveness and efficiency over competitive baselines.

Towards a latent space cartography of subjective experience in mental health.

The way that individuals subjectively experience the world greatly influences their own mental well-being. However, it remains a considerable challenge to precisely characterize the breadth and depth of such experiences. One persistent problem is the lack of objective tools for directly quantifying and comparing narrative reports of subjective experiences. Here, we develop a new approach to map and compare reports of experience using the latent space of artificial neural networks. Using a series of 31 prompts, including 30 images and one open-ended question, we quantified how the verbal reports provided by participants (n = 210, 50% female) deviate from one another and how these variations are linked to subjective experience and mental health. We found that latent space embeddings of experience can accurately predict subjective judgments of valence and arousal in a series of emotional pictures. Furthermore, we show that narrative reports to ambiguous images can accurately predict transdiagnostic factors of mental health. While distortions in the latent space of artificial neural networks are notoriously difficult to interpret, we propose a new approach to synthesize visual stimuli with generative artificial intelligence that can be used to explore semantic distortions in reported experiences. In sum, latent space cartography could offer a promising avenue for objectively quantifying distortions of subjective experience in mental health and could ultimately help identify new therapeutic targets for clinical interventions.

Out-of-distribution generalization via composition: A lens through induction heads in Transformers

Large language models (LLMs) such as GPT-4 sometimes appear to be creative, solving novel tasks often with a few demonstrations in the prompt. These tasks require the models to generalize on distributions different from those from training data-which is known as out-of-distribution (OOD) generalization. Despite the tremendous success of LLMs, how they approach OOD generalization remains an open and underexplored question. We examine OOD generalization in settings where instances are generated according to hidden rules, including in-context learning with symbolic reasoning. Models are required to infer the hidden rules behind input prompts without any fine-tuning. We empirically examined the training dynamics of Transformers on a synthetic example and conducted extensive experiments on a variety of pretrained LLMs, focusing on a type of component known as induction heads. We found that OOD generalization and composition are tied together-models can learn rules by composing two self-attention layers, thereby achieving OOD generalization. Furthermore, a shared latent subspace in the embedding (or feature) space acts as a bridge for composition by aligning early layers and later layers, which we refer to as the common bridge representation hypothesis.

Explainable Graph Spectral Clustering of text documents.

Spectral clustering methods are known for their ability to represent clusters of diverse shapes, densities etc. However, the results of such algorithms, when applied e.g. to text documents, are hard to explain to the user, especially due to embedding in the spectral space which has no obvious relation to document contents. Therefore, there is an urgent need to elaborate methods for explaining the outcome of the clustering. We have constructed in this paper a theoretical bridge linking the clusters resulting from Graph Spectral Clustering and the actual document content, given that similarities between documents are computed as cosine measures in tf or tfidf representation. This link enables to provide with explanation of cluster membership in clusters produced by GSA. We present a proposal of explanation of the results of combinatorial and normalized Laplacian based graph spectral clustering. For this purpose, we show (approximate) equivalence of combinatorial Laplacian embedding and of K-embedding (proposed in this paper) and term vector space embedding. We performed an experimental study showing that K-embedding approximates well Laplacian embedding under favourable block matrix conditions and show that approximation is good enough under other conditions. We show also perfect equivalence of normalized Laplacian embedding and the [Formula: see text]-embedding (proposed in this paper) and (weighted) term vector space embedding. Hence a bridge is constructed between the textual contents and the clustering results using both combinatorial and normalized Laplacian based Graph Spectral Clustering methods. We provide a theoretical background for our approach. An initial version of this paper is available at arXiv, (Starosta B 2023). The Reader may refer to that text to get acquainted with formal aspects of our method and find a detailed overview of motivation.