Embedding Space Research Articles

Self-supervised Learning Approach for Anomaly Detection in Rotating Machinery

Early fault detection in rotating machinery needs careful expert analysis of vibration data for monitoring a component state. Online methods that automatically set a threshold and raise an alarm when the vibration signature is anomalous are meant to efficiently manage key assets in a preventive maintenance plan. In recent years a focus has raised on data driven methods in parallel with the increasing attention towards machine learning and, particularly, deep learning models. In this regard, for rotating equipment components, an important aspect relates to labelled data scarcity for supervised training. On the other hand, the advent of the Internet of Things allows to gather data from multiple assets with relevant information on the asset state itself. Self-supervised learning methods in deep learning application are currently tackling this problem. Investigating Self-learning approaches to integrate domain knowledge and learn relevant features from unlabeled data is therefore important for condition monitoring applications. In this paper a methodology is proposed based on cycle consistency representation learning for training an embedder network on univariate unlabeled data. In order to learn a distance metric in the embedding space the original data are transformed to generate sequences of augmented inputs to enforce learnable pattern similarity in the augmented pairs. A differentiable cycle-consistency loss is chosen to maximize the numbers of augmented pairs in the learned embedding space that have minimum features distance. The pretext task in the described self-supervised setting aims to train a feature extractor for discriminating dissimilar samples in the embedding space by a distance metric and to provide a useful representation for down-stream tasks. The paper analyzes the performance of the approach for anomaly detection in rotating machinery. The methodology is tested on vibration data provided by the Center for Intelligent Maintenance Systems (IMS), University of Cincinnati, considering different accelerated life test campaigns. The data were collected to monitor the fault development in bearings and the model shows how the learned embedding space discriminates effectively anomalous samples from normal ones in the degradation stages of the bearings.

Toward Accelerating Discovery via Physics-Driven and Interactive Multifidelity Bayesian Optimization

Abstract Both computational and experimental material discovery bring forth the challenge of exploring multidimensional and often nondifferentiable parameter spaces, such as phase diagrams of Hamiltonians with multiple interactions, composition spaces of combinatorial libraries, processing spaces, and molecular embedding spaces. Often these systems are expensive or time consuming to evaluate a single instance, and hence classical approaches based on exhaustive grid or random search are too data intensive. This resulted in strong interest toward active learning methods such as Bayesian optimization (BO) where the adaptive exploration occurs based on human learning (discovery) objective. However, classical BO is based on a predefined optimization target, and policies balancing exploration and exploitation are purely data driven. In practical settings, the domain expert can pose prior knowledge of the system in the form of partially known physics laws and exploration policies often vary during the experiment. Here, we propose an interactive workflow building on multifidelity BO (MFBO), starting with classical (data-driven) MFBO, then expand to a proposed structured (physics-driven) structured MFBO (sMFBO), and finally extend it to allow human-in-the-loop interactive interactive MFBO (iMFBO) workflows for adaptive and domain expert aligned exploration. These approaches are demonstrated over highly nonsmooth multifidelity simulation data generated from an Ising model, considering spin–spin interaction as parameter space, lattice sizes as fidelity spaces, and the objective as maximizing heat capacity. Detailed analysis and comparison show the impact of physics knowledge injection and real-time human decisions for improved exploration with increased alignment to ground truth. The associated notebooks allow to reproduce the reported analyses and apply them to other systems.2

A deep learning method to integrate extracelluar miRNA with mRNA for cancer studies.

Extracellular miRNAs (exmiRs) and intracellular mRNAs both can serve as promising biomarkers and therapeutic targets for various diseases. However, exmiR expression data is often noisy, and obtaining intracellular mRNA expression data usually involves intrusive procedures. To gain valuable insights into disease mechanisms, it is thus essential to improve the quality of exmiR expression data and develop noninvasive methods for assessing intracellular mRNA expression. We developed CrossPred, a deep-learning multi-encoder model for the cross-prediction of exmiRs and mRNAs. Utilizing contrastive learning, we created a shared embedding space to integrate exmiRs and mRNAs. This shared embedding was then used to predict intracellular mRNA expression from noisy exmiR data and to predict exmiR expression from intracellular mRNA data. We evaluated CrossPred on three types of cancers and assessed its effectiveness in predicting the expression levels of exmiRs and mRNAs. CrossPred outperformed the baseline encoder-decoder model, exmiR or mRNA-based models, and variational autoencoder models. Moreover, the integration of exmiR and mRNA data uncovered important exmiRs and mRNAs associated with cancer. Our study offers new insights into the bidirectional relationship between mRNAs and exmiRs. The datasets and tool are available at https://doi.org/10.5281/zenodo.13891508.

Chasing the Neutrino Blazar Candidates

In our study of the correlations between IceCube-detected neutrino events and γ-ray properties of blazars, we recognize the inherent challenges posed by the limited detection of neutrinos. In this paper, we explore few-shot learning to deal with the class imbalance and few-shot issues presented in the incremental version of the 12 yr Fermi-LAT γ-ray source catalog (4FGL_ DR3). Specifically, we train a triplet network to transform the blazars with neutrino emission (NBs) and nonblazar samples into an embedding space where their similarities can be measured. With two-way three-shot learning, 199 out of 3708 blazars without neutrino emission (non-NBs) are considered as the potential blazars emitting neutrinos (NB candidates, or NBCs for short), with a similarity score against NBs exceeding 98%. Moreover, the Kolmogorov–Smirnov test supports our identification of NBCs.

Adaptive generic prototype network with geodesic distance for cross-domain few-shot fault diagnosis

Since the fault samples of equipment are limited and the working condition is often variable, it is valuable for researching the cross-domain few-shot fault diagnosis method to assure the safe operation of various machines. As the well-known few-shot classification approaches, the traditional prototype networks are difficult to process the complex sample distributions caused by variable operating condition data and the substantial distributional discrepancies between different machines, which seriously affects the accuracy of cross-domain fault diagnosis. To address these issues, this study proposes a new adaptive geodesic prototype network (AGPN), which can extract the category prototypes with enhanced adaptability and generalization capabilities. Firstly, a geodesic distance-driven learning strategy is developed to better measure the distance between complex samples in the embedding space. Secondly, an adaptive area prototype with a dynamic expansion coefficient is proposed, which allows for more flexible representation of different data categories. Furthermore, an adaptive momentum prototype method is put forward via a model-agnostic adaptive momentum factor, which can reduce the prototype oscillation during training and maximize the learning ability of model. The proposed AGPN is successfully applied to fault diagnosis across bearings with different operating conditions. Compared with the existing few-shot diagnosis methods, the proposed method possesses higher diagnostic accuracy and training stability, thus it is more suitable for cross-domain few-shot fault diagnosis.

Network community detection via neural embeddings

Recent advances in machine learning research have produced powerful neural graph embedding methods, which learn useful, low-dimensional vector representations of network data. These neural methods for graph embedding excel in graph machine learning tasks and are now widely adopted. However, how and why these methods work—particularly how network structure gets encoded in the embedding—remain largely unexplained. Here, we show that node2vec—shallow, linear neural network—encodes communities into separable clusters better than random partitioning down to the information-theoretic detectability limit for the stochastic block models. We show that this is due to the equivalence between the embedding learned by node2vec and the spectral embedding via the eigenvectors of the symmetric normalized Laplacian matrix. Numerical simulations demonstrate that node2vec is capable of learning communities on sparse graphs generated by the stochastic blockmodel, as well as on sparse degree-heterogeneous networks. Our results highlight the features of graph neural networks that enable them to separate communities in the embedding space.

Exploring Inherent Consistency for Semi-Supervised Anatomical Structure Segmentation in Medical Imaging.

Due to the exorbitant expense of obtaining labeled data in the field of medical image analysis, semi-supervised learning has emerged as a favorable method for the segmentation of anatomical structures. Although semi-supervised learning techniques have shown great potential in this field, existing methods only utilize image-level spatial consistency to impose unsupervised regularization on data in label space. Considering that anatomical structures often possess inherent anatomical properties that have not been focused on in previous works, this study introduces the inherent consistency into semi-supervised anatomical structure segmentation. First, the prediction and the ground-truth are projected into an embedding space to obtain latent representations that encapsulate the inherent anatomical properties of the structures. Then, two inherent consistency constraints are designed to leverage these inherent properties by aligning these latent representations. The proposed method is plug-and-play and can be seamlessly integrated with existing methods, thereby collaborating to improve segmentation performance and enhance the anatomical plausibility of the results. To evaluate the effectiveness of the proposed method, experiments are conducted on three public datasets (ACDC, LA, and Pancreas). Extensive experimental results demonstrate that the proposed method exhibits good generalizability and outperforms several state-of-the-art methods.

Weakly supervised video anomaly detection based on hyperbolic space

In recent years, there has been a proliferation of weakly supervised methods in the field of video anomaly detection. Despite significant progress in existing research, these efforts have primarily focused on addressing this issue within Euclidean space. Conducting weakly supervised video anomaly detection in Euclidean space imposes a fundamental limitation by constraining the ability to model complex patterns due to the dimensionality constraints of the embedding space and lacking the capacity to model long-term contextual information. This inadequacy can lead to misjudgments of anomalous events due to insufficient video representation. However, hyperbolic space has shown significant potential for modeling complex data, offering new insights. In this paper, we rethink weakly supervised video anomaly detection with a novel perspective: transforming video features from Euclidean space into hyperbolic space may enable the network to learn implicit relationships in normal and anomalous videos, thereby enhancing its ability to effectively distinguish between them. Finally, to validate our approach, we conducted extensive experiments on the UCF-Crime and XD-Violence datasets. Experimental results show that our method not only has the lowest number of parameters but also achieves state-of-the-art performance on the XD-Violence dataset using only RGB information.

Enhancing cross-domain recommendations: Leveraging personality-based transfer learning with probabilistic matrix factorization

The conventional method of computing personality scores through extensive questionnaire-based surveys poses practical challenges in real-world scenarios. An alternate route is to predict personality scores from user reviews by analyzing various linguistic features such as writing style, word choices, and specific phrases. However, the reviews are domain-dependent and classification models trained on one domain cannot be readily applied to other domains. To mitigate this challenge, we propose a cross-domain recommendation framework called PEMF-CD which leverages a novel mixing strategy to integrate user reviews from multiple domains with common joint embedding space and predict user personality scores using a transformer model. By capturing the underlying semantics and latent representations within the textual data, the transformer architecture can effectively model the linguistic cues to infer users’ personality traits, and the learning is transferred across domains. To further enhance the recommendation process, our model integrates personality-wise and rating pattern-based similarities of users into a probabilistic matrix factorization method that fosters user neighborhoods based on similarity scores among users. Comprehensive experiments were conducted using five real-world datasets from TripAdvisor and Amazon with varied numbers of users, items, and reviews of up to 44187, 26386, and 426791, respectively. The performance has been benchmarked against thirteen baseline algorithms and the experimental results demonstrate a significant improvements of up to 24.72%, 64.28%, 48.79%, and 61% in RMSE, and 55.9%, 76.7%, 67.6%, and 71.5% in MAE for a 90:10 train-test split with Digital Music, Fashion, Magazine Subscriptions and Video Games datasets from Amazon, respectively. Similar results have been observed for the 80:20 train-test split.

Conformal bootstrap equations from the embedding space operator product expansion

We describe how to implement the conformal bootstrap program in the context of the embedding space OPE formalism introduced in previous work. To take maximal advantage of the known properties of the scalar conformal blocks for symmetric-traceless exchange, we construct tensorial generalizations of the three-point and four-point scalar conformal blocks that have many nice properties. Further, we present a special basis of tensor structures for three-point correlation functions endowed with the remarkable simplifying property that it does not mix under permutations of the external quasi-primary operators. We find that in this approach, we can write the M-point conformal bootstrap equations explicitly in terms of the standard position space cross-ratios without the need to project back to position space, thus effectively deriving all conformal bootstrap equations directly from the embedding space. Finally, we lay out an algorithm for generating the conformal bootstrap equations in this formalism. Collectively, the tensorial generalizations, the new basis of tensor structures, as well as the procedure for deriving the conformal bootstrap equations lead to four-point bootstrap equations for quasi-primary operators in arbitrary Lorentz representations expressed as linear combinations of the standard scalar conformal blocks for spin-ℓ exchange, with finite ℓ-independent terms. Moreover, the OPE coefficients in these equations conveniently feature trivial symmetry properties. The only inputs necessary are the relevant projection operators and tensor structures, which are all fixed by group theory. To illustrate the procedure, we present one nontrivial example involving scalars S and vectors V, namely ⟨SSSV⟩.

Kinematic matrix: One-shot human action recognition using kinematic data structure

One-shot action recognition, which refers to recognizing human-performed actions using only a single training example, holds significant promise in advancing video analysis, particularly in domains requiring rapid adaptation to new actions. However, existing algorithms for one-shot action recognition face multiple challenges, including high computational complexity, limited accuracy, and difficulties in generalization to unseen actions. To address these issues, we propose a novel kinematic-based skeleton representation that effectively reduces computational demands while enhancing recognition performance. This representation leverages skeleton locations, velocities, and accelerations to formulate the one-shot action recognition task as a metric learning problem, where a model projects kinematic data into an embedding space. In this space, actions are distinguished based on Euclidean distances, facilitating efficient nearest-neighbour searches among activity reference samples. Our approach not only reduces computational complexity but also achieves higher accuracy and better generalization compared to existing methods. Specifically, our model achieved a validation accuracy of 78.5%, outperforming state-of-the-art methods by 8.66% under comparable training conditions. These findings underscore the potential of our method for practical applications in real-time action recognition systems.

Developing healthcare language model embedding spaces

Pre-trained Large Language Models (LLMs) have revolutionised Natural Language Processing (NLP) tasks, but often struggle when applied to specialised domains such as healthcare. The traditional approach of pre-training on large datasets followed by task-specific fine-tuning is resource-intensive and poorly aligned with the constraints of many healthcare settings. This presents a significant challenge for deploying LLM-based NLP solutions in medical contexts, where data privacy, computational resources, and domain-specific language pose unique obstacles.This study aims to develop and evaluate efficient methods for adapting smaller LLMs to healthcare-specific datasets and tasks. We seek to identify pre-training approaches that can effectively instil healthcare competency in compact LLMs under tight computational budgets, a crucial capability for responsible and sustainable deployment in local healthcare settings.We explore three specialised pre-training methods to adapt smaller LLMs to different healthcare datasets: traditional Masked Language modelling (MLM), Deep Contrastive Learning for Unsupervised Textual Representations (DeCLUTR), and a novel approach utilising metadata categories from healthcare settings. These methods are assessed across multiple healthcare datasets, with a focus on downstream document classification tasks. We evaluate the performance of the resulting LLMs through classification accuracy and analysis of the derived embedding spaces.Contrastively trained models consistently outperform other approaches on classification tasks, delivering strong performance with limited labelled data and fewer model parameter updates. While our novel metadata-based pre-training does not further improve classifications across datasets, it yields interesting embedding cluster separability. Importantly, all domain-adapted LLMs outperform their publicly available, general-purpose base models, validating the importance of domain specialisation.This research demonstrates the efficacy of specialised pre-training methods in adapting compact LLMs to healthcare tasks, even under resource constraints. We provide guidelines for pre-training specialised healthcare LLMs and motivate continued inquiry into contrastive objectives. Our findings underscore the potential of these approaches for aligning small LLMs with privacy-sensitive medical tasks, offering a path toward more efficient and responsible NLP deployment in healthcare settings. This work contributes to the broader goal of making advanced NLP techniques accessible and effective in specialised domains, particularly where resource limitations and data sensitivity are significant concerns.

Few-shot Learning for Sign Language Recognition with Embedding Propagation

Sign language is a primary channel for the deaf and hard-hearing to communicate. Sign language consists of many signs with different variations in hand shapes, motion patterns, and positioning of hands, faces, and body parts. This makes sign language recognition (SLR) a challenging field in computer vision research. This paper tackles the problem of few-shot SLR, where models trained on known sign classes are utilized to recognize instances of unseen signs with only a few examples. In this approach, a transformer encoder is employed to learn the spatial and temporal features of sign gestures, and an embedding propagation technique is used to project these features into the embedding space. Subsequently, a label propagation method is applied to smooth the resulting embeddings. The obtained results demonstrate that combining embedding propagation with label propagation enhances the performance of the SLR system and achieved an accuracy of 76.6%, which surpasses the traditional few-shot prototypical network's accuracy of 72.4%.

Chimeric U-Net – Modifying the standard U-Net towards explainability

Healthcare guided by semantic segmentation has the potential to improve our quality of life through early and accurate disease detection. Convolutional Neural Networks, especially the U-Net-based architectures, are currently the state-of-the-art learning-based segmentation methods and have given unprecedented performances. However, their decision-making processes are still an active field of research. In order to reliably utilize such methods in healthcare, explainability of how the segmentation was performed is mandated. To date, explainability is studied and applied heavily in classification tasks. In this work, we propose the Chimeric U-Net, a U-Net architecture with an invertible decoder unit, that inherently brings explainability into semantic segmentation tasks. We find that having the restriction of an invertible decoder does not hinder the performance of the segmentation task. However, the invertible decoder helps to disentangle the class information in the latent space embedding and to construct meaningful saliency maps. Furthermore, we found that with a simple k-Nearest-Neighbours classifier, we could predict the Intersection over Union scores of unseen data, demonstrating that the latent space, constructed by the Chimeric U-Net, encodes an interpretable representation of the segmentation quality. Explainability is an emerging field, and in this work, we propose an alternative approach, that is, rather than building tools for explaining a generic architecture, we propose constraints on the architecture which induce explainability. With this approach, we could peer into the architecture to reveal its class correlations and local contextual dependencies, taking an insightful step towards trustworthy and reliable AI. Code to build and utilize the Chimeric U-Net is made available under:https://github.com/kenrickschulze/Chimeric-UNet---Half-invertible-UNet-in-Pytorch

Analyzing Semantic Properties of Word Embeddings Using Eigenvectors

Dense word vectors have demonstrated their efficacy in several downstream natural language processing (NLP) tasks in recent years. Nevertheless, the interpretability of these embeddings' dimensions remains challenging. In this paper, we investigate how eigenvectors can reveal different semantic properties captured by word embedding models. Hence, we train word embeddings (e.g., Word2Vec) on English Wikipedia corpus to analyze the top eigenvectors, identify specific semantic properties (e.g., sentiment, formality) associated with each, and explore how these properties are encoded in the embedding space. This paper also discussed the limitations and potential benefits of this approach compared to other methods for analyzing word embeddings.

Anomaly detection scheme for lung CT images using vector quantized variational auto-encoder with support vector data description.

This study aims to develop an anomaly-detection scheme for lesions in CT images. Our database consists of lung CT images obtained from 1500 examinees. It includes 1200 normal and 300 abnormal cases. In this study, SVDD (Support Vector Data Description) mapping the normal latent variables into a hypersphere as small as possible on the latent space is introduced to VQ-VAE (Vector Quantized-Variational Auto-Encoder). VQ-VAE with SVDD is constructed from two encoders, two decoders, and an embedding space. The first encoder compresses the input image into the latent-variable map, whereas the second encoder maps the normal latent variables into a hypersphere as small as possible. The first decoder then up-samples the mapped latent variables into a latent-variable map with the original size. The second decoder finally reconstructs the input image from the latent-variable map replaced by the embedding representations. The data of each examinee is classified as abnormal or normal based on the anomaly score defined as the combination of the difference between the input image and the reconstructed image and the distance between the latent variables and the center of the hypersphere. The area under the ROC curve for VQ-VAE with SVDD was 0.76, showing an improvement when compared with the conventional VAE (0.63, p < .001). VQ-VAE with SVDD developed in this study can yield higher anomaly-detection accuracy than the conventional VAE. The proposed method is expected to be useful for identifying examinees with lesions and reducing interpretation time in CT screening.

Embedding space approach to Lorentzian CFT amplitudes and causal spherical functions

Conformal field theory in a Minkowski setting is discussed in an embedding space approach, paying special attention to causality constraints for four-point amplitudes. The physics of dilatation and Lorentz boost is emphasized in specifying the noncompact maximal Abelian subgroup of SO(d,2). Reduction of a conformal field theory four-point amplitudes as functions of cross ratios is shown to be equivalent to enforcing H bi-invariance, i.e., F(hgh′)=F(g), with g∈SO(d,2) and H an appropriate subgroup. Causality is imposed by introducing appropriate semigroups. Causal zonal spherical functions are constructed, making contact with Minkowski conformal blocks introduced previously. Published by the American Physical Society 2024

Conformal Yang-Mills field in (A)dS space

Ordinary-derivative (second-derivative) Lagrangian formulation of classical conformal Yang-Mills field in the (A)dS space of six, eight, and ten dimensions is developed. For such conformal field, we develop two gauge invariant Lagrangian formulations which we refer to as generic formulation and decoupled formulation. In both formulations, the usual Yang-Mills field is accompanied by additional vector and scalar fields where the scalar fields are realized as Stueckelberg fields. In the generic formulation, the usual Yang-Mills field is realized as a primary field, while the additional vector fields are realized as auxiliary fields. In the decoupled formulation, the usual Yang-Mills field is realized as massless field, while the additional vector fields together with the Stueckelberg are realized as massive fields. Some massless/massive fields appear with the wrong sign of kinetic terms, hence demonstrating explicitly that the considered models are not unitary. The use of embedding space method allows us to treat the isometry symmetries of (A)dS space manifestly and obtain conformal transformations of fields in a relatively straightforward way. By accompanying each vector field by the respective gauge parameter, we introduce an extended gauge algebra. Levy-Maltsev decomposition of such algebra is noted. Use of the extended gauge algebra setup allows us to present concise form for the Lagrangian and gauge transformations of the conformal Yang-Mills field. Higher-derivative representation of the Lagrangian is also obtained.

Multi-view visual semantic embedding for cross-modal image–text retrieval

Visual Semantic Embedding (VSE) is a dominant method for cross-modal image–text retrieval. The purpose of VSE is to learn an embedding space where images can be embedded close to the corresponding captions. However, there are large intra-class variations in image–text data. Multiple captions describing the same image may be described from different views, and descriptions of different views are often dissimilar. The VSE method embeds samples from the same class in similar positions, which suppresses intra-class variations and leads to inferior generalization. This paper proposes a Multi-View Visual Semantic Embedding (MV-VSE) framework that learns multiple embeddings for an image, explicitly modeling intra-class variation. To optimize the MV-VSE framework, a multi-view triplet loss is proposed, which jointly optimizes multi-view embeddings while retaining intra-class variation. Recently, large-scale Vision-Language Pre-training (VLP) has become a new paradigm for cross-modal image–text retrieval. To allow our framework to be flexibly applied to the traditional VSE models and VSE-based VLP models, we incorporate the contrastive loss commonly used in VLP and the triplet loss into a unified loss, and further propose a multi-view unified loss. Our framework can be applied plug-and-play to traditional VSE models and VSE-based VLP models without excessively increasing model complexity. Experimental results on the image–text retrieval benchmark datasets demonstrate that applying our framework can boost the retrieval performance of current VSE models. The code is available at https://github.com/AAA-Zheng/MV-VSE.

Batch effects correction in scRNA-seq based on biological-noise decoupling autoencoder and central-cross loss

Technical or biologically irrelevant differences caused by different experiments, times, or sequencing platforms can generate batch effects that mask the true biological information. Therefore, batch effects are typically removed when analyzing single-cell RNA sequencing (scRNA-seq) datasets for downstream tasks. Existing batch correction methods usually mitigate batch effects by reducing the data from different batches to a lower dimensional space before clustering, potentially leading to the loss of rare cell types. To address this problem, we introduce a novel single-cell data batch effect correction model using Biological-noise Decoupling Autoencoder (BDA) and Central-cross Loss termed BDACL. The model initially reconstructs raw data using an auto-encoder and conducts preliminary clustering. We then construct a similarity matrix and a hierarchical clustering tree to delineate relationships within and between different batches. Finally, we introduce a Central-cross Loss (CL). This loss leverages cross-entropy loss to prompt the model to better distinguish between different cluster labels. Additionally, it employs the Central Loss to encourage samples to form more compact clusters in the embedding space, thereby enhancing the consistency and interpretability of clustering results to mitigate differences between different batches. The primary innovation of this model lies in reconstructing data with an auto-encoder and gradually merging smaller clusters into larger ones using a hierarchical clustering tree. By using reallocated cluster labels as training labels and employing the Central-cross Loss, the model effectively eliminates batch effects in an unsupervised manner. Compared to current methods, BDACL can mitigate batch effects without losing rare cell types.