Representation Learning Research Articles

The Role of Embodiment in Learning Representations: Discovering Space Topology via Sensory-Motor Prediction

This paper explores the crucial role of embodiment in learning representations for space topology in robotics. Embodiment, the ability of an agent to interact with its environment and receive sensory feedback, is fundamental to developing accurate and efficient representations. In this work, we investigate this by applying an action-conditional prediction algorithm to data collected from a simulated environment, aiming to learn the topology of the environment through sequences of random interactions. Using a simple mobile-robot-like scenario, by leveraging sensory-motor interactions we demonstrate how the agent can discover the topology of its environment. Our results demonstrate the importance of embodiment in the development of representations and potential applicability in robotic tasks, and a simple but effective method of integrating actions into a learning loop. We suggest that building abstract representations through the use of action-conditional prediction is a step towards unification of the representations used in robotics.

Wav2vec2 Without Attention: Do You Need Hopfield Networks for Self-Supervised Learning of Speech Representations?

Wav2vec2 Without Attention: Do You Need Hopfield Networks for Self-Supervised Learning of Speech Representations?

Enhancing machine learning recommendation systems with CNN architectures: Applications and innovations in computer vision and speech recognition

Abstract. This paper provides an in-depth analysis of enhancing machine learning recommendation systems using Convolutional Neural Network (CNN) architectures, with a focus on their applications in computer vision and speech recognition. Traditional recommendation systems often struggle with scalability and the complexity of high-dimensional data. By integrating CNNs, these systems can utilize advanced feature extraction and representation learning to more effectively process and analyze diverse data sources. Our study demonstrates significant improvements in recommendation accuracy and personalization through CNN-enhanced systems. We discuss the architecture, design principles, and advantages of CNNs, supported by case studies across various domains. Our findings illustrate the potential of CNNs to revolutionize recommendation systems by addressing existing limitations and offering innovative solutions for real-time, high-quality recommendations. This research emphasizes the importance of advanced machine learning techniques in creating robust and scalable recommendation systems, paving the way for future advancements and applications in multiple fields.

SCARF: Scalable Continual Learning Framework for Memory‐efficient Multiple Neural Radiance Fields

AbstractThis paper introduces a novel continual learning framework for synthesising novel views of multiple scenes, learning multiple 3D scenes incrementally, and updating the network parameters only with the training data of the upcoming new scene. We build on Neural Radiance Fields (NeRF), which uses multi‐layer perceptron to model the density and radiance field of a scene as the implicit function. While NeRF and its extensions have shown a powerful capability of rendering photo‐realistic novel views in a single 3D scene, managing these growing 3D NeRF assets efficiently is a new scientific problem. Very few works focus on the efficient representation or continuous learning capability of multiple scenes, which is crucial for the practical applications of NeRF. To achieve these goals, our key idea is to represent multiple scenes as the linear combination of a cross‐scene weight matrix and a set of scene‐specific weight matrices generated from a global parameter generator. Furthermore, we propose an uncertain surface knowledge distillation strategy to transfer the radiance field knowledge of previous scenes to the new model. Representing multiple 3D scenes with such weight matrices significantly reduces memory requirements. At the same time, the uncertain surface distillation strategy greatly overcomes the catastrophic forgetting problem and maintains the photo‐realistic rendering quality of previous scenes. Experiments show that the proposed approach achieves state‐of‐the‐art rendering quality of continual learning NeRF on NeRF‐Synthetic, LLFF, and TanksAndTemples datasets while preserving extra low storage cost.

Integration of molecular coarse-grained model into geometric representation learning framework for protein-protein complex property prediction

Structure-based machine learning algorithms have been utilized to predict the properties of protein-protein interaction (PPI) complexes, such as binding affinity, which is critical for understanding biological mechanisms and disease treatments. While most existing algorithms represent PPI complex graph structures at the atom-scale or residue-scale, these representations can be computationally expensive or may not sufficiently integrate finer chemical-plausible interaction details for improving predictions. Here, we introduce MCGLPPI, a geometric representation learning framework that combines graph neural networks (GNNs) with MARTINI molecular coarse-grained (CG) models to predict PPI overall properties accurately and efficiently. Extensive experiments on three types of downstream PPI property prediction tasks demonstrate that at the CG-scale, MCGLPPI achieves competitive performance compared with the counterparts at the atom- and residue-scale, but with only a third of computational resource consumption. Furthermore, CG-scale pre-training on protein domain-domain interaction structures enhances its predictive capabilities for PPI tasks. MCGLPPI offers an effective and efficient solution for PPI overall property predictions, serving as a promising tool for the large-scale analysis of biomolecular interactions.

Network Architecture Search Methods for Constructing Efficient Fault Diagnosis Models in Rotating Machinery

Abstract The development of high-performance fault diagnosis models for specific tasks requires substantial expertise. Neural Architecture Search (NAS) offers a promising solution, but most NAS methodologies are hampered by lengthy search durations and low efficiency, and few researchers have applied these methods within the fault diagnosis domain. This paper introduces a novel differentiable architecture search (DARTS) method tailored for constructing efficient fault diagnosis models for rotating machinery, designed to rapidly and effectively search for network models suitable for specific datasets. Specifically, this study constructs a completely new and advanced search space, incorporating various efficient, lightweight convolutional operations to reduce computational complexity. To enhance the stability of the differentiable network architecture search process and reduce fluctuations in model accuracy, this study proposes a novel Multi-scale Pyramid Squeeze Attention (MPSA) module. This module aids in the learning of richer multi-scale feature representations and adaptively recalibrates the weights of multi-dimensional channel attention. The proposed method was validated on two rotating machinery fault datasets, demonstrating superior performance compared to manually designed networks and general network search methods, with notably improved diagnostic effectiveness.

Adaptive Granularity-Fused Keypoint Detection for 6D Pose Estimation of Space Targets

Estimating the 6D pose of a space target is an intricate task due to factors such as occlusions, changes in visual appearance, and background clutter. Accurate pose determination requires robust algorithms capable of handling these complexities while maintaining reliability under various environmental conditions. Conventional pose estimation for space targets unfolds in two stages: establishing 2D–3D correspondences using keypoint detection networks and 3D models, followed by pose estimation via the perspective-n-point algorithm. The accuracy of this process hinges critically on the initial keypoint detection, which is currently limited by predominantly singular-scale detection techniques and fails to exploit sufficient information. To tackle the aforementioned challenges, we propose an adaptive dual-stream aggregation network (ADSAN), which enables the learning of finer local representations and the acquisition of abundant spatial and semantic information by merging features from both inter-layer and intra-layer perspectives through a multi-grained approach, consolidating features within individual layers and amplifying the interaction of distinct resolution features between layers. Furthermore, our ADSAN implements the selective keypoint focus module (SKFM) algorithm to alleviate problems caused by partial occlusions and viewpoint alterations. This mechanism places greater emphasis on the most challenging keypoints, ensuring the network prioritizes and optimizes its learning around these critical points. Benefiting from the finer and more robust information of space objects extracted by the ADSAN and SKFM, our method surpasses the SOTA method PoET (5.8°, 8.1°/0.0351%, 0.0744%) by 0.5°, 0.9°, and 0.0084%, 0.0354%, achieving 5.3°, 7.2° in rotation angle errors and 0.0267%, 0.0390% in normalized translation errors on the Speed and SwissCube datasets, respectively.

Edge Detection of Source Body from Magnetic Anomaly Based on ResNet

Utilizing magnetic anomaly data for effective edge detection of source bodies can provide crucial evidence for the delineation of geological units and the division of fault structures. However, the existing edge detection methods of source bodies from magnetic anomalies are influenced by factors such as the source bodies’ burial depth, magnetization direction, and mutual interference of magnetic anomalies, leading to errors in subsequent interpretation tasks. The advanced convolutional neural network possesses robust capabilities for feature representation and deep learning, prompting this paper to introduce an edge detection method for source bodies based on convolutional neural networks. The issue is initially framed as a semantic segmentation problem, and four network architectures aimed at edge detection of a source body from magnetic anomaly are designed and modified based on the U-Net and ResNet. Subsequently, a multitude of high-quality sample data sets are constructed using models with varying locations, scales, quantities, and physical properties to train the network. This paper then details model experiments that escalate from simple to complex, taking into account the combined effects of burial depth and inclined magnetization on edge detection. Compared to conventional edge detection methods, the method proposed in this paper is shown to accurately identify edges of source bodies at various depths with little impact from inclined magnetization and can automatically extract edge information without manual intervention. The method’s efficacy is corroborated through real data tests.

Multi-modal representation learning in retinal imaging using self-supervised learning for enhanced clinical predictions

Self-supervised learning has become the cornerstone of building generalizable and transferable artificial intelligence systems in medical imaging. In particular, contrastive representation learning techniques trained on large multi-modal datasets have demonstrated impressive capabilities of producing highly transferable representations for different downstream tasks. In ophthalmology, large multi-modal datasets are abundantly available and conveniently accessible as modern retinal imaging scanners acquire both 2D fundus images and 3D optical coherence tomography (OCT) scans to assess the eye. In this context, we introduce a novel multi-modal contrastive learning-based pipeline to facilitate learning joint representations for the two retinal imaging modalities. After self-supervised pre-training on 153,306 scan pairs, we show that such a pre-training framework can provide both a retrieval system and encoders that produce comprehensive OCT and fundus image representations that generalize well for various downstream tasks on three independent external datasets, explicitly focusing on clinically pertinent prediction tasks. In addition, we show that interchanging OCT with lower-cost fundus imaging can preserve the predictive power of the trained models.

Inductive Subgraph Embedding for Link Prediction

AbstractLink prediction, which aims to infer missing edges or predict future edges based on currently observed graph connections, has emerged as a powerful technique for diverse applications such as recommendation, relation completion, etc. While there is rich literature on link prediction based on node representation learning, direct link embedding is relatively less studied and less understood. One common practice in previous work characterizes a link by manipulate the embeddings of its incident node pairs, which is not capable of capturing effective link features. Moreover, common link prediction methods such as random walks and graph auto-encoder usually rely on full-graph training, suffering from poor scalability and high resource consumption on large-scale graphs. In this paper, we propose Inductive Subgraph Embedding for Link Prediciton (SE4LP) — an end-to-end scalable representation learning framework for link prediction, which utilizes the strong correlation between central links and their neighborhood subgraphs to characterize links. We sample the “link-centric induced subgraphs” as input, with a subgraph-level contrastive discrimination as pretext task, to learn the intrinsic and structural link features via subgraph classification. Extensive experiments on five datasets demonstrate that SE4LP has significant superiority in link prediction in terms of performance and scalability, when compared with state-of-the-art methods. Moreover, further analysis demonstrate that introducing self-supervision in link prediction can significantly reduce the dependence on training data and improve the generalization and scalability of model.

AmbiBias Contrast: Enhancing debiasing networks via disentangled space from ambiguity-bias clusters

The goal of debiasing in classification tasks is to train models to be less sensitive to correlations between a sample’s target attribution and periodically occurring contextual attributes to achieve accurate classification. A prevalent method involves applying re-weighing techniques to lower the weight of bias-aligned samples that contribute to bias, thereby focusing the training on bias-conflicting samples that deviate from the bias patterns. Our empirical analysis indicates that this approach is effective in datasets where bias-conflicting samples constitute a minority compared to bias-aligned samples, yet its effectiveness diminishes in datasets with similar proportions of both. This ineffectiveness in varied dataset compositions suggests that the traditional method cannot be practical in diverse environments as it overlooks the dynamic nature of dataset-induced biases. To address this issue, we introduce a contrastive approach named “AmbiBias Contrast”, which is robust across various dataset compositions. This method accounts for “ambiguity bias”— the variable nature of bias elements across datasets, which cannot be clearly defined. Given the challenge of defining bias due to the fluctuating compositions of datasets, we designed a method of representation learning that accommodates this ambiguity. Our experiments across a range of and dataset configurations verify the robustness of our method, delivering state-of-the-art performance.

Multi-view graph contrastive representation learning for bundle recommendation

Bundle recommendation can recommend a collection of associated items that can be consumed together to a user rather than recommending these items separately, making it extremely suitable for some scenarios such as product bundle recommendation and game bundle recommendation. Recent bundle recommendation approaches consider auxiliary data to mitigate sparse user-bundle interactions. However, these approaches obtain the node embeddings directly from the established user-bundle graph and do not explicitly exploit the relationships between users (bundles) when constructing recommendation models. Moreover, bundle recommendation approaches based on graph contrastive learning usually construct contrastive views by randomly discarding nodes (edges) in the graph, while discarding some essential nodes or edges will destroy the structure of the original graph, thereby deteriorating the quality of the learned node embeddings. Aiming at these limitations, we propose a bundle recommendation approach based on multi-view graph contrastive representation learning. First, we present a multi-view modeling method to model the relations between entities as several views from different perspectives. These views serve as inputs of graph neural networks for graph representation learning and provide contrastive views for the contrastive learning tasks. Second, we propose a novel framework for bundle recommendation. This framework obtains the user (bundle) embeddings from different views by performing multi-view graph representation learning and enhances the learned user and bundle embeddings through a two-level contrastive learning strategy. On this basis, the enhanced user (bundle) embeddings are fused for prediction. Finally, we design a joint optimization objective to optimize the model parameters, combining the prediction loss that supports multiple negative samples and the contrastive losses. Experiments on the Netease and Youshu datasets reveal that our approach outperforms the state-of-the-art (SOTA) baselines. Furthermore, the average improvements of Recall@K and NDCG@K of our approach over the SOTA baselines are approximately 3.38% and 2.80% on Netease and 3.94% and 4.84% on Youshu.

Unsupervised representation learning of Kohn–Sham states and consequences for downstream predictions of many-body effects

Representation learning for the electronic structure problem is a major challenge of machine learning in computational condensed matter and materials physics. Within quantum mechanical first principles approaches, density functional theory (DFT) is the preeminent tool for understanding electronic structure, and the high-dimensional DFT wavefunctions serve as building blocks for downstream calculations of correlated many-body excitations and related physical observables. Here, we use variational autoencoders (VAE) for the unsupervised learning of DFT wavefunctions and show that these wavefunctions lie in a low-dimensional manifold within latent space. Our model autonomously determines the optimal representation of the electronic structure, avoiding limitations due to manual feature engineering. To demonstrate the utility of the latent space representation of the DFT wavefunction, we use it for the supervised training of neural networks (NN) for downstream prediction of quasiparticle bandstructures within the GW formalism. The GW prediction achieves a low error of 0.11 eV for a combined test set of two-dimensional metals and semiconductors, suggesting that the latent space representation captures key physical information from the original data. Finally, we explore the generative ability and interpretability of the VAE representation.

CIDGMed: Causal Inference-Driven Medication Recommendation with enhanced dual-granularity learning

Medication recommendation aims to integrate patients’ long-term health records to provide accurate and safe medication combinations for specific health states. Existing methods often fail to deeply explore the true causal relationships between diseases/procedures and medications, resulting in biased recommendations. Additionally, in medication representation learning, the relationships between information at different granularities of medications—coarse-grained (medication itself) and fine-grained (molecular level)—are not effectively integrated, leading to biases in representation learning. To address these limitations, we propose the Causal Inference-driven Dual-Granularity Medication Recommendation method (CIDGMed). Our approach leverages causal inference to uncover the relationships between diseases/procedures and medications, thereby enhancing the rationality and interpretability of recommendations. By integrating coarse-grained medication effects with fine-grained molecular structure information, CIDGMed provides a comprehensive representation of medications. Additionally, we employ a bias correction model during the prediction phase to further refine recommendations, ensuring both accuracy and safety. Through extensive experiments, CIDGMed significantly outperforms current state-of-the-art models across multiple metrics, achieving a 2.54% increase in accuracy, a 3.65% reduction in side effects, and a 39.42% improvement in time efficiency. Additionally, we demonstrate the rationale of CIDGMed through a case study.

Prototype Similarity Distillation for Communication-Efficient Federated Unsupervised Representation Learning

Prototype Similarity Distillation for Communication-Efficient Federated Unsupervised Representation Learning

Heterogeneous graph representation learning via mutual information estimation for fraud detection

In the fraud detection, fraudsters frequently engage with numerous benign users to disguise their activities. Consequently, the fraud graph exhibits not only homogeneous connections between the fraudsters and the same labelled nodes, but also heterogeneous connections, where fraudsters interact with the legitimate nodes. Heterogeneous graph representation learning aims at extracting the structural and semantic information and embed it into the low-dimensional node representation. Recently, maximizing the mutual information between the local node embedding and the summary representation has achieved the promising results on node classification tasks. However, existing deep graph infomax methods still have the following limitations. Firstly, attribute information of nodes in the graph is not fully utilized for capturing the semantic relationships between nodes. Secondly, the local and global supervision signal are not simultaneously exploited for the node embedding learning. Thirdly, the multiplex heterogeneous relations among nodes are ignored. To address these issues, a heterogeneous graph representation learning model by mutual information estimation (MIE-HetGRL) is proposed in this paper to identify the fraudsters in the fraud review graph. Concretely, a high-order mutual information estimation is proposed to integrate the local and global mutual information as the supervision signal. Then we devise a semantic attention fusion module to aggregate the relation-aware node embeddings into a compact node representation. Finally, a joint contrastive learning is designed for facilitating the training and optimization of model. The experimental results show that our proposed model significantly outperforms state-of-the-art baselines for fraud detection.

Supervised Representation Learning Towards Generalizable Assembly State Recognition

Supervised Representation Learning Towards Generalizable Assembly State Recognition

Enhanced landslide susceptibility mapping in data-scarce regions via unsupervised few-shot learning

Given the critical need to assess landslide hazards, producing landslide susceptibility map (LSM) in regions with scarce historical landslide inventories poses significant challenges. This study introduces a novel landslide susceptibility assessment framework that combines unsupervised learning strategies with few-shot learning methods to increase the accuracy of LSM in these areas. The framework has been practically validated in a representative geological disaster-prone area along the West-East Gas Pipeline in Shaanxi Province, China. We employed three advanced few-shot learning models: a support vector machine, meta-learning, and transfer learning. These models implement feature representation learning for weakly correlated influencing factors through an unsupervised approach, thereby constructing an effective landslide susceptibility assessment model. We compared traditional learning methods and used the receiver operating characteristic (ROC) curve and SHAP values to quantify the effectiveness of the models. The results indicate that the meta-learning algorithm outperforms both the SVM and transfer learning in areas with limited landslide data. The integration of unsupervised strategies significantly improves performance, achieving area under the curve (AUC) values of 0.9385 and 0.9861, respectively. Compared with using meta-learning alone, incorporating unsupervised learning strategies increased the AUC by 4.76%, enhancing both the predictive power of the model and the interpretability of the features. Meta-learning under unsupervised conditions effectively mitigates the evaluation difficulties caused by insufficient landslide records, providing a viable path and empirical evidence for performance improvement in similar data- scarce regions worldwide.

GeoNet enables the accurate prediction of protein-ligand binding sites through interpretable geometric deep learning.

The identification of protein binding residues is essential for understanding their functions invivo. However, it remains a computational challenge to accurately identify binding sites due to the lack of known residue binding patterns. Local residue spatial distribution and its interactive biophysical environment both determine binding patterns. Previous methods could not capture both information simultaneously, resulting in unsatisfactory performance. Here, we present GeoNet, an interpretable geometric deep learning model for predicting DNA, RNA, and protein binding sites by learning the latent residue binding patterns. GeoNet achieves this by introducing a coordinate-free geometric representation to characterize local residue distributions and generating an eigenspace to depict local interactive biophysical environments. Evaluation shows that GeoNet is superior compared to other leading predictors and it shows a strong interpretability of learned representations. We present three test cases, where interaction interfaces were successfully identified with GeoNet.

MIMO-Uformer: A Transformer-Based Image Deblurring Network for Vehicle Surveillance Scenarios

Motion blur is a common problem in the field of surveillance scenarios, and it obstructs the acquisition of valuable information. Thanks to the success of deep learning, a sequence of CNN-based architecture has been designed for image deblurring and has made great progress. As another type of neural network, transformers have exhibited powerful deep representation learning and impressive performance based on high-level vision tasks. Transformer-based networks leverage self-attention to capture the long-range dependencies in the data, yet the computational complexity is quadratic to the spatial resolution, which makes transformers infeasible for the restoration of high-resolution images. In this article, we propose an efficient transformer-based deblurring network, named MIMO-Uformer, for vehicle-surveillance scenarios. The distinct feature of the MIMO-Uformer is that the basic-window-based multi-head self-attention (W-MSA) of the Swin transformer is employed to reduce the computational complexity and then incorporated into a multi-input and multi-output U-shaped network (MIMO-UNet). The performance can benefit from the operation of multi-scale images by MIMO-UNet. However, most deblurring networks are designed for global blur, while local blur is more common under vehicle-surveillance scenarios since the motion blur is primarily caused by local moving vehicles. Based on this observation, we further propose an Intersection over Patch (IoP) factor and a supervised morphological loss to improve the performance based on local blur. Extensive experiments on a public and a self-established dataset are carried out to verify the effectiveness. As a result, the deblurring behavior based on PSNR is improved at least 0.21 dB based on GOPRO and 0.74 dB based on the self-established datasets compared to the existing benchmarks.