Self-supervised Representation Research Articles

DCPoint: Global-Local Dual Contrast for Self-Supervised Representation Learning of 3-D Point Clouds

DCPoint: Global-Local Dual Contrast for Self-Supervised Representation Learning of 3-D Point Clouds

Self-supervised learning for characterising histomorphological diversity and spatial RNA expression prediction across 23 human tissue types

As vast histological archives are digitised, there is a pressing need to be able to associate specific tissue substructures and incident pathology to disease outcomes without arduous annotation. Here, we learn self-supervised representations using a Vision Transformer, trained on 1.7 M histology images across 23 healthy tissues in 838 donors from the Genotype Tissue Expression consortium (GTEx). Using these representations, we can automatically segment tissues into their constituent tissue substructures and pathology proportions across thousands of whole slide images, outperforming other self-supervised methods (43% increase in silhouette score). Additionally, we can detect and quantify histological pathologies present, such as arterial calcification (AUROC = 0.93) and identify missing calcification diagnoses. Finally, to link gene expression to tissue morphology, we introduce RNAPath, a set of models trained on 23 tissue types that can predict and spatially localise individual RNA expression levels directly from H&E histology (mean genes significantly regressed = 5156, FDR 1%). We validate RNAPath spatial predictions with matched ground truth immunohistochemistry for several well characterised control genes, recapitulating their known spatial specificity. Together, these results demonstrate how self-supervised machine learning when applied to vast histological archives allows researchers to answer questions about tissue pathology, its spatial organisation and the interplay between morphological tissue variability and gene expression.

Self-supervised Representations and Node Embedding Graph Neural Networks for Accurate and Multi-scale Analysis of Materials

Abstract We propose a new strategy by self-supervised learning on the node and edge information of the crystal graph neural network (GNN) model, which may gain physical insights by tuning the depth of multi-scale representations. Compared with the popular manually constructed material descriptors, the self-supervised atomic representation can be applied to various machine learning models and reach better prediction performance on material properties. Applying the self-supervised atomic representation on the magnetic moment datasets, we show how they can extract rules and information from the magnetic materials. We develop the node-embedding graph neural networks (NEGNN) framework to incorporate rich physical information into the GNN model, showing significant improvements in the prediction performance. The self-supervised material representation and the NEGNN framework may investigate in-depth information from materials and can be applied to small datasets with increased prediction accuracy.

Self-supervised Representations and Node Embedding Graph Neural Networks for Accurate and Multi-scale Analysis of Materials

Abstract We propose a new strategy by self-supervised learning on the node and edge information of the crystal graph neural network (GNN) model, which may gain physical insights by tuning the depth of multi-scale representations. Compared with the popular manually constructed material descriptors, the self-supervised atomic representation can be applied to various machine learning models and reach better prediction performance on material properties. Applying the self-supervised atomic representation on the magnetic moment datasets, we show how they can extract rules and information from the magnetic materials. We develop the node-embedding graph neural networks (NEGNN) framework to incorporate rich physical information into the GNN model, showing significant improvements in the prediction performance. The self-supervised material representation and the NEGNN framework may investigate in-depth information from materials and can be applied to small datasets with increased prediction accuracy.

MICCF: A Mutual Information Constrained Clustering Framework for Learning Clustering-Oriented Feature Representations

Deep clustering is a crucial task in machine learning and data mining that focuses on acquiring feature representations conducive to clustering. Previous research relies on self-supervised representation learning for general feature representations, such features may not be optimally suited for downstream clustering tasks. In this paper, we introduce MICCF, a framework designed to bridge this gap and enhance clustering performance. MICCF enhances feature representations by combining mutual information at different levels and employs an auxiliary alignment mutual information module for learning clustering-oriented features. To be specific, we propose a dual mutual information constraints module, incorporating minimal mutual information constraints at the feature level and maximal mutual information constraints at the instance level. This reduction in feature redundancy encourages the neural network to extract more discriminative features, while maximization ensures more unbiased and robust representations. To obtain clustering-oriented representations, the auxiliary alignment mutual information module utilizes pseudo-labels to maximize mutual information through a multi-classifier network, aligning features with the clustering task. The main network and the auxiliary one work in synergy to jointly optimize feature representations that are well-suited for the clustering task. We validate the effectiveness of our method through extensive experiments on six benchmark datasets. The results indicate that our method performs well in most scenarios, particularly on fine-grained datasets, where our approach effectively distinguishes subtle differences between closely related categories. Notably, our approach achieved a remarkable accuracy of 96.4% on the ImageNet-10 dataset, surpassing other comparison methods. The code is available at https://github.com/Li-Hyn/MICCF.git .

Federated Momentum Contrastive Clustering

Self-supervised representation learning and deep clustering are mutually beneficial to learn high-quality representations and cluster data simultaneously in centralized settings. However, it is not always feasible to gather large amounts of data at a central entity, considering data privacy requirements and computational resources. Federated Learning (FL) has been developed successfully to aggregate a global model while training on distributed local data, respecting the data privacy of edge devices. However, most FL research effort focuses on supervised learning algorithms. A fully unsupervised federated clustering scheme has not been considered in the existing literature. We present federated momentum contrastive clustering (FedMCC), a generic federated clustering framework that can not only cluster data automatically but also extract discriminative representations training from distributed local data over multiple users. In FedMCC, we demonstrate a two-stage federated learning paradigm where the first stage aims to learn differentiable instance embeddings and the second stage accounts for clustering data automatically. The experimental results show that FedMCC not only achieves superior clustering performance but also outperforms several existing federated self-supervised methods for linear evaluation and semi-supervised learning tasks. Additionally, FedMCC can easily be adapted to ordinary centralized clustering through what we call momentum contrastive clustering (MCC). We show that MCC achieves state-of-the-art clustering accuracy results in certain datasets such as STL-10 and ImageNet-10. We also present a method to reduce the memory footprint of our clustering schemes.

Detecting galaxy tidal features using self-supervised representation learning

Abstract Low surface brightness substructures around galaxies, known as tidal features, are a valuable tool in the detection of past or ongoing galaxy mergers, and their properties can answer questions about the progenitor galaxies involved in the interactions. The assembly of current tidal feature samples is primarily achieved using visual classification, making it difficult to construct large samples and draw accurate and statistically robust conclusions about the galaxy evolution process. With upcoming large optical imaging surveys such as the Vera C. Rubin Observatory’s Legacy Survey of Space and Time (LSST), predicted to observe billions of galaxies, it is imperative that we refine our methods of detecting and classifying samples of merging galaxies. This paper presents promising results from a self-supervised machine learning model, trained on data from the Ultradeep layer of the Hyper Suprime-Cam Subaru Strategic Program optical imaging survey, designed to automate the detection of tidal features. We find that self-supervised models are capable of detecting tidal features, and that our model outperforms previous automated tidal feature detection methods, including a fully supervised model. An earlier method applied to real galaxy images achieved 76% completeness for 22% contamination, while our model achieves considerably higher (96%) completeness for the same level of contamination. We emphasise a number of advantages of self-supervised models over fully supervised models including maintaining excellent performance when using only 50 labelled examples for training, and the ability to perform similarity searches using a single example of a galaxy with tidal features.

Trusted 3D self-supervised representation learning with cross-modal settings

Trusted 3D self-supervised representation learning with cross-modal settings

Kiosk Recommend System Based on Self-Supervised Representation Learning of User Behaviors in Offline Retail

Kiosk Recommend System Based on Self-Supervised Representation Learning of User Behaviors in Offline Retail

Enhancing equipment safeguarding in IIoT: A self-supervised fault diagnosis paradigm based on asymmetric graph autoencoder

Thanks to the sufficient monitoring data provided by Industrial Internet of Things (IIoT), intelligent fault diagnosis technology has demonstrated remarkable performance in safeguarding equipment. However, the effectiveness of existing methods heavily relies on manually labeled data. Unfortunately, data collected from equipment often lacks labels, leading to a scarcity of fault data. Furthermore, an additional significant challenge is the feature domain shift resulting from speed variation. To address this, we propose a self-supervised paradigm based on an asymmetric graph autoencoder for fault diagnosis under domain shift, aiming to mine valuable health information from unlabeled data. Unlike Euclidean-based methods, the proposed method transforms time series samples into graphs and extracts domain invariant features through information interaction between nodes. To efficiently mine unlabeled data and enhance generalization, the self-supervised learning paradigm utilizes an asymmetric graph autoencoder architecture. This architecture includes an encoder that learns self-supervised representations from unlabeled samples and a lightweight decoder that predicts the original input. Specifically, we mask a portion of input samples and predict the original input from learned self-supervised representations. In downstream task, the pre-trained encoder is fine-tuned using limited labeled data for specific fault diagnosis task. The proposed method is evaluated on three mechanical fault simulation experiments, and the results demonstrate the its superiority and potential.

Contrastive visual clustering for improving instance-level contrastive learning as a plugin

Contrastive learning has achieved remarkable success in computer vision, however it is built on instance-level discrimination which leaves the valuable intra-class correlation in dataset unexploited. Current semantic clustering methods are proven to be helpful but they would suffer from the error accumulated in the iteration process without ground-truth guidance. In an attempt to remedy the clustering error accumulation when utilizing intra-class correlation for contrastive learning, we propose an online Contrastive Visual Clustering (CVC) method with two actions: gathering instances with highly similar feature embeddings, and penalizing instances being clustered with low confidence. CVC can integrate with not only contrastive learning but also arbitrary self-supervised learning frameworks simply as a plugin. Under various experiment settings, we show that CVC improves the linear classification performance by a large margin for models pre-trained with self-supervised representation learning, in both image and video scenarios. The code is available at https://github.com/yliu1229/CVC.

Improving inceptionV4 model based on fractional-order snow leopard optimization algorithm for diagnosing of ACL tears

In the current research study, a new method is presented to diagnose Anterior Cruciate Ligament (ACL) tears by introducing an optimized version of the InceptionV4 model. Our proposed methodology utilizes a custom-made variant of the Snow Leopard Optimization Algorithm, known as the Fractional-order Snow Leopard Optimization Algorithm (FO-LOA), to extract essential features from knee magnetic resonance imaging (MRI) images. This results in a substantial improvement in the accuracy of ACL tear detection. By effectively extracting critical features from knee MRI images, our proposed methodology significantly enhances diagnostic accuracy, potentially reducing false negatives and false positives. The enhanced model based on FO-LOA underwent thorough testing using the MRNet dataset, demonstrating exceptional performance metrics including an accuracy rate of 98.00%, sensitivity of 98.00%, precision of 97.00%, specificity of 98.00%, F1-score of 98.00%, and Matthews Correlation Coefficient (MCC) of 88.00%. These findings surpass current methodologies like Convolutional Neural Network (CNN), Inception-v3, Deep Belief Networks and Improved Honey Badger Algorithm (DBN/IHBA), integration of the CNN with an Amended Cooking Training-based Optimizer version (CNN/ACTO), Self-Supervised Representation Learning (SSRL), signifying a significant breakthrough in ACL injury diagnosis. Using FO-SLO to optimize the InceptionV4 framework shows promise in improving the accuracy of ACL tear identification, enabling prompt and efficient treatment interventions.

A novel self-supervised representation learning framework based on time-frequency alignment and interaction for mechanical fault diagnosis

Recently, supervised learning methods for intelligent mechanical fault diagnosis have achieved considerable advancements, while they heavily rely on labeled information and neglect massive unlabeled signals. Consequently, some self-supervised research has emerged and employs contrastive learning to mine general knowledge from unlabeled samples. However, these methods emphasize independent feature extraction and lack deep feature interaction, which limits better representation learning. Furthermore, it also restricts the performance improvement of fault diagnosis. To fill this gap, a novel self-supervised representation learning framework based on time-frequency alignment and interaction (TFAI) is proposed to improve the diagnosis reliability under limited labeled data. Comprising a dual encoder and a cross-modal encoder based on Transformers, the proposed TFAI model takes paired time-frequency data as input for self-supervised pretraining. Two self-supervised learning strategies, namely time-frequency alignment and time-frequency interaction, are proposed to enable the model to gain valuable knowledge from unlabeled data. The former leverages a time-frequency contrastive loss to drive the dual encoder to extract profound features from unlabeled data. The latter involves utilizing a time-frequency matching loss to guide the cross-modal encoder to perform deep feature fusion in an unsupervised manner. The well pretrained TFAI model can learn informative representations and can be fine-tuned for specific downstream fault diagnosis tasks to improve the diagnosis accuracy. By conducting three experimental cases on an axial flow pump system and a public gearbox platform system, the effectiveness, generalizability, and domain adaptability of the proposed method are validated.

Advancements in Self-Supervised Learning for Remote Sensing Scene Classification: Present Innovations and Future Outlooks

Deep learning methodologies have significantly advanced the fields of computer vision and machine learning, enhancing performance across various tasks like classification, regression, and detection. In remote sensing for Earth observation, deep neural networks have propelled state-of-the-art results. However, a major drawback is their dependence on large annotated datasets, necessitating extensive human effort, especially in specialized domains like medical imaging or remote sensing. To mitigate this reliance on annotations, several self-supervised representation learning techniques have emerged, aiming to learn unsupervised image representations applicable to downstream tasks such as image classification, object detection, or semantic segmentation. Consequently, self-supervised learning approaches have gained traction in remote sensing. This article surveys the foundational principles of various self-supervised methods, focusing on scene classification tasks. We elucidate key contributions, analyze experimental setups, and synthesize findings from each study. Furthermore, we conduct comprehensive experiments on two public scene classification datasets to evaluate and benchmark different self-supervised models

Advancements in Self-Supervised Learning for Remote Sensing Scene Classification: Present Innovations and Future Outlooks

Deep learning methodologies have significantly advanced the fields of computer vision and machine learning, enhancing performance across various tasks like classification, regression, and detection. In remote sensing for Earth observation, deep neural networks have propelled state-of-the-art results. However, a major drawback is their dependence on large annotated datasets, necessitating extensive human effort, especially in specialized domains like medical imaging or remote sensing. To mitigate this reliance on annotations, several self-supervised representation learning techniques have emerged, aiming to learn unsupervised image representations applicable to downstream tasks such as image classification, object detection, or semantic segmentation. Consequently, self-supervised learning approaches have gained traction in remote sensing. This article surveys the foundational principles of various self-supervised methods, focusing on scene classification tasks. We elucidate key contributions, analyze experimental setups, and synthesize findings from each study. Furthermore, we conduct comprehensive experiments on two public scene classification datasets to evaluate and benchmark different self-supervised models

Self-Supervised Representation Learning on Electronic Health Records with Graph Kernel Infomax

Learning Electronic Health Records (EHRs) representation is a preeminent yet under-discovered research topic. It benefits various clinical decision support applications, e.g., medication outcome prediction or patient similarity search. Current approaches focus on task-specific label supervision on vectorized sequential EHR, which is not applicable to large-scale unsupervised scenarios. Recently, contrastive learning has shown great success in self-supervised representation learning problems. However, complex temporality often degrades the performance. We propose Graph Kernel Infomax, a self-supervised graph kernel learning approach on the graphical representation of EHR, to overcome the previous problems. Unlike the state-of-the-art, we do not change the graph structure to construct augmented views. Instead, we use Kernel Subspace Augmentation to embed nodes into two geometrically different manifold views. The entire framework is trained by contrasting nodes and graph representations on those two manifold views through the commonly used contrastive objectives. Empirically, using publicly available benchmark EHR datasets, our approach yields performance on clinical downstream tasks that exceeds the state-of-the-art. Theoretically, the variation in distance metrics naturally creates different views as data augmentation without changing graph structures. Practically, our method is non-ad hoc and confirms superior performance on commonly used graph benchmark datasets.

ProtoMGAE: Prototype-Aware Masked Graph Auto-Encoder for Graph Representation Learning

Graph self-supervised representation learning has gained considerable attention and demonstrated remarkable efficacy in extracting meaningful representations from graphs, particularly in the absence of labeled data. Two representative methods in this domain are graph auto-encoding and graph contrastive learning. However, the former methods primarily focus on global structures, potentially overlooking some fine-grained information during reconstruction. The latter methods emphasize node similarity across correlated views in the embedding space, potentially neglecting the inherent global graph information in the original input space. Moreover, handling incomplete graphs in real-world scenarios, where original features are unavailable for certain nodes, poses challenges for both types of methods. To alleviate these limitations, we integrate masked graph auto-encoding and prototype-aware graph contrastive learning into a unified model to learn node representations in graphs. In our method, we begin by masking a portion of node features and utilize a specific decoding strategy to reconstruct the masked information. This process facilitates the recovery of graphs from a global or macro level and enables handling incomplete graphs easily. Moreover, we treat the masked graph and the original one as a pair of contrasting views, enforcing the alignment and uniformity between their corresponding node representations at a local or micro level. Last, to capture cluster structures from a meso level and learn more discriminative representations, we introduce a prototype-aware clustering consistency loss that is jointly optimized with the preceding two complementary objectives. Extensive experiments conducted on several datasets demonstrate that the proposed method achieves significantly better or competitive performance on downstream tasks, especially for graph clustering, compared with the state-of-the-art methods, showcasing its superiority in enhancing graph representation learning.

Enabling unsupervised discovery in astronomical images through self-supervised representations

ABSTRACT Unsupervised learning, a branch of machine learning that can operate on unlabelled data, has proven to be a powerful tool for data exploration and discovery in astronomy. As large surveys and new telescopes drive a rapid increase in data size and richness, these techniques offer the promise of discovering new classes of objects and of efficient sorting of data into similar types. However, unsupervised learning techniques generally require feature extraction to derive simple but informative representations of images. In this paper, we explore the use of self-supervised deep learning as a method of automated representation learning. We apply the algorithm Bootstrap Your Own Latent to Galaxy Zoo DECaLS images to obtain a lower dimensional representation of each galaxy, known as features. We briefly validate these features using a small supervised classification problem. We then move on to apply an automated clustering algorithm, demonstrating that this fully unsupervised approach is able to successfully group together galaxies with similar morphology. The same features prove useful for anomaly detection, where we use the framework astronomaly to search for merger candidates. While the focus of this work is on optical images, we also explore the versatility of this technique by applying the exact same approach to a small radio galaxy data set. This work aims to demonstrate that applying deep representation learning is key to unlocking the potential of unsupervised discovery in future data sets from telescopes such as the Vera C. Rubin Observatory and the Square Kilometre Array.

WaSSaBi: Wafer Selection With Self-Supervised Representations and Brain-Inspired Active Learning

WaSSaBi: Wafer Selection With Self-Supervised Representations and Brain-Inspired Active Learning

SurgNet: Self-Supervised Pretraining With Semantic Consistency for Vessel and Instrument Segmentation in Surgical Images.

Blood vessel and surgical instrument segmentation is a fundamental technique for robot-assisted surgical navigation. Despite the significant progress in natural image segmentation, surgical image-based vessel and instrument segmentation are rarely studied. In this work, we propose a novel self-supervised pretraining method (SurgNet) that can effectively learn representative vessel and instrument features from unlabeled surgical images. As a result, it allows for precise and efficient segmentation of vessels and instruments with only a small amount of labeled data. Specifically, we first construct a region adjacency graph (RAG) based on local semantic consistency in unlabeled surgical images and use it as a self-supervision signal for pseudo-mask segmentation. We then use the pseudo-mask to perform guided masked image modeling (GMIM) to learn representations that integrate structural information of intraoperative objectives more effectively. Our pretrained model, paired with various segmentation methods, can be applied to perform vessel and instrument segmentation accurately using limited labeled data for fine-tuning. We build an Intraoperative Vessel and Instrument Segmentation (IVIS) dataset, comprised of ~3 million unlabeled images and over 4,000 labeled images with manual vessel and instrument annotations to evaluate the effectiveness of our self-supervised pretraining method. We also evaluated the generalizability of our method to similar tasks using two public datasets. The results demonstrate that our approach outperforms the current state-of-the-art (SOTA) self-supervised representation learning methods in various surgical image segmentation tasks.