Self-supervised Approach Research Articles

MolE: a foundation model for molecular graphs using disentangled attention.

Models that accurately predict properties based on chemical structure are valuable tools in the chemical sciences. However, for many properties, public and private training sets are typically small, making it difficult for models to generalize well outside of the training data. Recently, this lack of generalization has been mitigated by using self-supervised pretraining on large unlabeled datasets, followed by finetuning on smaller, labeled datasets. Inspired by these advances, we report MolE, a Transformer architecture adapted for molecular graphs together with a two-step pretraining strategy. The first step of pretraining is a self-supervised approach focused on learning chemical structures trained on ~842 million molecular graphs, and the second step is a massive multi-task approach to learn biological information. We show that finetuning models that were pretrained in this way perform better than the best published results on 10 of the 22 ADMET (absorption, distribution, metabolism, excretion and toxicity) tasks included in the Therapeutic Data Commons leaderboard (c. September 2023).

Utilizing graph neural networks for adverse health detection and personalized decision making in sensor-based remote monitoring for dementia care

Background:Sensor-based remote health monitoring is increasingly used to detect adverse health in people living with dementia (PLwD) at home, aiming to prevent hospitalizations and reduce caregiver burden. However, home sensor data is often noisy, overly granular, and suffers from unreliable labeling, data drift and high variability between households. Current anomaly detection methods lack generalizability and personalization, often requiring anomaly-free training data and frequent model updates. Objective:To develop a lightweight, explainable, self-supervised approach with personalized alert thresholds to detect adverse health events in PLwD, using changes in home activity. Methods:We hypothesized that health downturns manifest as detectable shifts in household movement patterns. Our approach leverages a Graph Barlow Twins contrastive model, which uses granular activity data and a macroscopic view to extract noise-robust, high-level and low-level discriminative features that represent daily activity patterns. Household-personalized alert thresholds are calculated based on clinician-set target alert rates, and daily anomaly scores are compared against these thresholds, triggering alerts for the clinical monitoring team. Model attention weights support explainability. Data were collected from a real-world dataset by the UK Dementia Research Institute (August 2019-April 2022). Results:Our model outperformed state-of-the-art temporal graph algorithms in detecting agitation and fall events across three patient cohorts, achieving 81% average recall and 88% generalizability at a target alert rate of 7%. Conclusion:We developed a novel, lightweight, explainable, and personalized Graph Barlow Twins model for real-world remote health monitoring in dementia care, with potential for broader applications in healthcare and sensor-based environments.

Light-weight neural network for intra-voxel structure analysis.

We present a novel neural network-based method for analyzing intra-voxel structures, addressing critical challenges in diffusion-weighted MRI analysis for brain connectivity and development studies. The network architecture, called the Local Neighborhood Neural Network, is designed to use the spatial correlations of neighboring voxels for an enhanced inference while reducing parameter overhead. Our model exploits these relationships to improve the analysis of complex structures and noisy data environments. We adopt a self-supervised approach to address the lack of ground truth data, generating signals of voxel neighborhoods to integrate the training set. This eliminates the need for manual annotations and facilitates training under realistic conditions. Comparative analyses show that our method outperforms the constrained spherical deconvolution (CSD) method in quantitative and qualitative validations. Using phantom images that mimic in vivo data, our approach improves angular error, volume fraction estimation accuracy, and success rate. Furthermore, a qualitative comparison of the results in actual data shows a better spatial consistency of the proposed method in areas of real brain images. This approach demonstrates enhanced intra-voxel structure analysis capabilities and holds promise for broader application in various imaging scenarios.

Vehicle Ego-Trajectory Segmentation Using Guidance Cues

Computer vision has significantly influenced recent advancements in autonomous driving by providing cutting-edge solutions for various challenges, including object detection, semantic segmentation, and comprehensive scene understanding. One specific challenge is ego-vehicle trajectory segmentation, which involves learning the vehicle’s path and describing it with a segmentation map. This can play an important role in both autonomous driving and advanced driver assistance systems, as it enhances the accuracy of perceiving and forecasting the vehicle’s movements across different driving scenarios. In this work, we propose a deep learning approach for ego-trajectory segmentation that leverages a state-of-the-art segmentation network augmented with guidance cues provided through various merging mechanisms. These mechanisms are designed to direct the vehicle’s path as intended, utilizing training data obtained with a self-supervised approach. Our results demonstrate the feasibility of using self-supervised labels for ego-trajectory segmentation and embedding directional intentions within the network’s decisions through image and guidance input concatenation, feature concatenation, or cross-attention between pixel features and various types of guidance cues. We also analyze the effectiveness of our approach in constraining the segmentation outputs and prove that our proposed improvements bring major boosts in the segmentation metrics, increasing IoU by more than 12% and 5% compared with our two baseline models. This work paves the way for further exploration into ego-trajectory segmentation methods aimed at better predicting the behavior of autonomous vehicles.

A Knowledge-Driven Self-Supervised Approach for Molecular Generation.

Due to the great successes of Graph Neural Networks (GNN) in numerous fields, growing research interests have been devoted to applying GNN to molecular learning tasks. The molecule structure can be naturally represented as graphs where atoms and bonds refer to nodes and edges respectively. However, the atoms are not haphazardly stacked together but combined into various spatial geometries. Meanwhile, since chemical reactions mainly occur in substructures such as functional groups, the substructure plays a decisive role in the molecule's properties. Therefore, directly applying GNN to molecular representation learning could ignore the molecular spatial structure and the substructure properties which in turn degrades the performance of downstream tasks. In this paper, we propose Knowledge-Driven Self-Supervised Model for Molecular Representation Learning (KSMRL) to address above problems. The KSMRL consists of two major pathways: (1) the Spatial Information (SI) based pathway which preserves the spatial information of molecular structure, (2) the Subgraph Constraint (SC) based pathway which retains the properties of substructures into the molecular representation. In this manner, both the atomic level and substructure level information can be included in modeling. According to the experimental results on multiple datasets, the proposed KSMRL can generate discriminative molecular representations. In molecular generation tasks, KSMRL combined with Autoregressive Flow (AF) models or Discrete Flow (DF) models outperforms the state-of-the-art baselines over all datasets. In addition, we demonstrate the effectiveness of KSMRL with property optimization experiments. To indicate the ability of predicting specified potential Drug-Target Interactions (DTIs), a case study for discriminating the interactions between molecule generated by KSMRL and targets is also given.

MVEB: Self-Supervised Learning With Multi-View Entropy Bottleneck.

Self-supervised learning aims to learn representation that can be effectively generalized to downstream tasks. Many self-supervised approaches regard two views of an image as both the input and the self-supervised signals, assuming that either view contains the same task-relevant information and the shared information is (approximately) sufficient for predicting downstream tasks. Recent studies show that discarding superfluous information not shared between the views can improve generalization. Hence, the ideal representation is sufficient for downstream tasks and contains minimal superfluous information, termed minimal sufficient representation. One can learn this representation by maximizing the mutual information between the representation and the supervised view while eliminating superfluous information. Nevertheless, the computation of mutual information is notoriously intractable. In this work, we propose an objective termed multi-view entropy bottleneck (MVEB) to learn minimal sufficient representation effectively. MVEB simplifies the minimal sufficient learning to maximizing both the agreement between the embeddings of two views and the differential entropy of the embedding distribution. Our experiments confirm that MVEB significantly improves performance. For example, it achieves top-1 accuracy of 76.9% on ImageNet with a vanilla ResNet-50 backbone on linear evaluation. To the best of our knowledge, this is the new state-of-the-art result with ResNet-50.

Unlocking the potential of unlabeled data: Self-supervised machine learning for battery aging diagnosis with real-world field data

Accurate aging diagnosis is crucial for the health and safety management of lithium-ion batteries in electric vehicles. Despite significant advancements achieved by data-driven methods, diagnosis accuracy remains constrained by the high costs of check-up tests and the scarcity of labeled data. This paper presents a framework utilizing self-supervised machine learning to harness the potential of unlabeled data for diagnosing battery aging in electric vehicles during field operations. We validate our method using battery degradation datasets collected over more than two years from twenty real-world electric vehicles. Our analysis comprehensively addresses cell inconsistencies, physical interpretations, and charging uncertainties in real-world applications. This is achieved through self-supervised feature extraction using random short charging sequences in the main peak of incremental capacity curves. By leveraging inexpensive unlabeled data in a self-supervised approach, our method demonstrates improvements in average root mean square errors of 74.54% and 60.50% in the best and worst cases, respectively, compared to the supervised benchmark. This work underscores the potential of employing low-cost unlabeled data with self-supervised machine learning for effective battery health and safety management in real-world scenarios.

Weakly-supervised learning-based pathology detection and localization in 3D chest CT scans.

Recent advancements in anomaly detection have paved the way for novel radiological reading assistance tools that support the identification of findings, aimed at saving time. The clinical adoption of such applications requires a low rate of false positives while maintaining high sensitivity. In light of recent interest and development in multi pathology identification, we present a novel method, based on a recent contrastive self-supervised approach, for multiple chest-related abnormality identification including low lung density area ("LLDA"), consolidation ("CONS"), nodules ("NOD") and interstitial pattern ("IP"). Our approach alerts radiologists about abnormal regions within a computed tomography (CT) scan by providing 3Dlocalization. We introduce a new method for the classification and localization of multiple chest pathologies in 3D Chest CT scans. Our goal is to distinguish four common chest-related abnormalities: "LLDA", "CONS", "NOD", "IP" and "NORMAL". This method is based on a 3D patch-based classifier with a Resnet backbone encoder pretrained leveraging recent contrastive self supervised approach and a fine-tuned classification head. We leverage the SimCLR contrastive framework for pretraining on an unannotated dataset of randomly selected patches and we then fine-tune it on a labeled dataset. During inference, this classifier generates probability maps for each abnormality across the CT volume, which are aggregated to produce a multi-label patient-level prediction. We compare different training strategies, including random initialization, ImageNet weight initialization, frozen SimCLR pretrained weights and fine-tuned SimCLR pretrained weights. Each training strategy is evaluated on a validation set for hyperparameter selection and tested on a test set. Additionally, we explore the fine-tuned SimCLR pretrained classifier for 3D pathology localization and conduct qualitativeevaluation. Validated on 111 chest scans for hyperparameter selection and subsequently tested on 251 chest scans with multi-abnormalities, our method achieves an AUROC of 0.931 (95% confidence interval [CI]: [0.9034, 0.9557], -value < 0.001) and 0.963 (95% CI: [0.952, 0.976], -value < 0.001) in the multi-label and binary (i.e., normal versus abnormal) settings, respectively. Notably, our method surpasses the area under the receiver operating characteristic (AUROC) threshold of 0.9 for two abnormalities: IP (0.974) and LLDA (0.952), while achieving values of 0.853 and 0.791 for NOD and CONS, respectively. Furthermore, our results highlight the superiority of incorporating contrastive pretraining within the patch classifier, outperforming Imagenet pretraining weights and non-pretrained counterparts with uninitialized weights (F1 score = 0.943, 0.792, and 0.677 respectively). Qualitatively, the method achieved a satisfactory 88.8% completeness rate in localization and maintained an 88.3% accuracy rate against falsepositives. The proposed method integrates self-supervised learning algorithms for pretraining, utilizes a patch-based approach for 3D pathology localization and develops an aggregation method for multi-label prediction at patient-level. It shows promise in efficiently detecting and localizing multiple anomalies within a singlescan.

Multiple representation contrastive self-supervised learning for pulmonary nodule detection✰

Self-supervised learning aims to create semantic-enriched representation from unannotated data. A prevalent strategy in this field involves training a unified representation space that is invariant to various transformation combinations. However, creating a single invariant representation to multiple transformations poses several challenges. The efficacy of such a representation space depends on factors such as the intensity, sequence, and various combination scenarios of transformations. As a result, features generated in single representation space may exhibit limited adaptability for subsequent tasks. In contrast to the conventional SSL training approach, we introduce a novel method that involves constructing multiple atomic transformation-invariant representation subspaces. Each subspace in the proposed method is invariant to a specific atomic transformation from a predefined reference set. Our method offers increased flexibility by enabling the downstream task to weigh every atomic transformation-invariant subspace based on the desired feature space. A series of experiments were conducted to compare our approach to traditional self-supervised learning methods in order to assess its effectiveness. This evaluation encompassed diverse data regimes, datasets, evaluation protocols, and perspectives on source-destination data distribution. Our results highlight the superiority of our method compared to training strategies based on single transformation-invariant representation spaces. Additionally, our proposed method demonstrated superior performance in reducing false positives in the context of pulmonary nodule detection when compared to several recent supervised and self-supervised approaches.

Correction: Self-supervised approach for diabetic retinopathy severity detection using vision transformer

Correction: Self-supervised approach for diabetic retinopathy severity detection using vision transformer

Disentangling brain atrophy heterogeneity in Alzheimer's disease: A deep self-supervised approach with interpretable latent space

Alzheimer's disease (AD) is heterogeneous, but existing methods for capturing this heterogeneity through dimensionality reduction and unsupervised clustering have limitations when it comes to extracting intricate atrophy patterns. In this study, we propose a deep learning based self-supervised framework that characterizes complex atrophy features using latent space representation. It integrates feature engineering, classification, and clustering to synergistically disentangle heterogeneity in Alzheimer's disease. Through this representation learning, we trained a clustered latent space with distinct atrophy patterns and clinical characteristics in AD, and replicated the findings in prodromal Alzheimer's disease. Moreover, we discovered that these clusters are not solely attributed to subtypes but also reflect disease progression in the latent space, representing the core dimensions of heterogeneity, namely progression and subtypes. Furthermore, longitudinal latent space analysis revealed two distinct disease progression pathways: medial temporal and parietotemporal pathways. The proposed approach enables effective latent representations that can be integrated with individual-level cognitive profiles, thereby facilitating a comprehensive understanding of AD heterogeneity.

Learning Self-distilled Features for Facial Deepfake Detection Using Visual Foundation Models: General Results and Demographic Analysis

Modern deepfake techniques produce highly realistic false media content with the potential for spreading harmful information, including fake news and incitements to violence. Deepfake detection methods aim to identify and counteract such content by employing machine learning algorithms, focusing mainly on detecting the presence of manipulation using spatial and temporal features. These methods often utilize Foundation Models trained on extensive unlabeled data through self-supervised approaches. This work extends previous research on deepfake detection, focusing on the effectiveness of these models while also considering biases, particularly concerning age, gender, and ethnicity, for ethical analysis. Experiments with DINOv2, a novel Vision Transformer-based Foundation Model, trained using the diverse Deepfake Detection Challenge Dataset, which encompasses several lighting conditions, resolutions, and demographic attributes, demonstrated improved deepfake detection when combined with a CNN classifier, with minimal bias towards these demographic characteristics.

Self-supervised approach for diabetic retinopathy severity detection using vision transformer

Self-supervised approach for diabetic retinopathy severity detection using vision transformer

Stacked collaborative transformer network with contrastive learning for video moment localization

Video grounding intends to perform temporal localization in multimedia information retrieval. The temporal bounds of the target video span are determined for the given input query. A novel interactive multi-head self-attention (IMSA) transformer is proposed to localize an unseen moment in the untrimmed video for the given image. A new semantic-trained self-supervised approach is considered in this paper to perform cross-domain learning to match the image query – video segment. It normalizes the convolution function enabling efficient correlation and collecting of semantically related video segments across time based on the image query. A double hostile Contrastive learning with Gaussian distribution parameters method is advanced to learn the representations of video. The proposed approach performs dynamically on various video components to achieve exact semantic synchronization and localization among queries and video. In the proposed approach, the IMSA model localizes frames greatly compared to other approaches. Experiments on benchmark datasets show that the proposed model can significantly increase temporal grounding accuracy. The moment occurrence is identified in the video with a start and end boundary ascertains an average recall of 86.45% and a mAP of 59.3%.

Utilizing External Knowledge to Enhance Location Prediction for Twitter/X Users in Low Resource Settings

Accurate estimates of user location are important for many online services, including event detection, disaster management and determining public opinion. Neural network-based techniques have proven to be highly effective in predicting user location. However, these models typically require a large amount of labeled training data, which can be difficult to obtain in real-world scenarios. In this paper, we present two approaches to tackle the issue of limited training data when predicting city level location. First, we consider a self-supervised approach that trains a state level model without labeled data and then integrate this knowledge into the training data set used for city level predictions. Second, we explore the option of increasing the number of training examples by utilizing external resources to generate synthetic users . Finally, we combine these two strategies, exploiting the benefits of both. We empirically evaluate our proposed techniques on multiple Twitter/X data sets and show that our models perform significantly better than the state-of-the-art with improvements of up to 6% for Acc@161 and 8% for F1 score.

Efficient human activity recognition: A deep convolutional transformer-based contrastive self-supervised approach using wearable sensors

Artificial intelligence has advanced the applications of sensor-based human motion capture and recognition technology in various engineering fields, such as human–robot collaboration and health monitoring. Deep learning methods can achieve satisfactory recognition results when provided with sufficient labeled data. However, labeling a large dataset is expensive and time-consuming in practical applications. To address this issue, this paper proposes a deep convolutional transformer-based contrastive self-supervised (DCTCSS) model under the bootstrap your own latent (BYOL) framework. The DCTCSS model aims to achieve reliable activity recognition using only a small amount of labeled data. Firstly, a deep convolutional transformer (DCT) model is proposed as the backbone of DCTCSS model, to learn high-level feature representations from unlabeled data in pre-training period. Subsequently, a simple linear classifier is trained with supervised fine-tuning using a limited amount of labeled data to recognize activities. In addition, this paper experimentally formulates a random data augmentation strategy to increase the diversity of input data. The performance of the DCTCSS model is evaluated and compared with several state-of-the-art algorithms on three datasets widely used in daily life, medical monitoring, and intelligent manufacturing. Experimental results show that the DCTCSS model achieves mean F1 scores of 95.64%, 88.39%, and 98.40% on the UCI-HAR, Skoda, and Mhealth datasets, respectively, using only 10% of the labeled data. These results outperform both supervised and unsupervised baseline models. Consequently, the DCTCSS model demonstrates its effectiveness in reducing the dependence on large amounts of labeled data while still achieving competitive activity recognition performance.

A robust self-supervised approach for fine-grained crack detection in concrete structures

This work addresses a critical issue: the deterioration of concrete structures due to fine-grained cracks, which compromises their strength and longevity. To tackle this problem, experts have turned to computer vision (CV) based automated strategies, incorporating object detection and image segmentation techniques. Recent efforts have integrated complex techniques such as deep convolutional neural networks (DCNNs) and transformers for this task. However, these techniques encounter challenges in localizing fine-grained cracks. This paper presents a self-supervised 'you only look once' (SS-YOLO) approach that utilizes a YOLOv8 model. The novel methodology amalgamates different attention approaches and pseudo-labeling techniques, effectively addressing challenges in fine-grained crack detection and segmentation in concrete structures. It utilizes convolution block attention (CBAM) and Gaussian adaptive weight distribution multi-head self-attention (GAWD-MHSA) modules to accurately identify and segment fine-grained cracks in concrete buildings. Additionally, the assimilation of curriculum learning-based self-supervised pseudo-labeling (CL-SSPL) enhances the model's ability when applied to limited-size data. The efficacy and viability of the proposed approach are demonstrated through experimentation, results, and ablation analysis. Experimental results indicate a mean average precision (mAP) of at least 90.01%, an F1 score of 87%, and an intersection over union threshold greater than 85%. It is evident from the results that the proposed method yielded at least 2.62% and 4.40% improvement in mAP and F1 values, respectively, when tested on three diverse datasets. Moreover, the inference time taken per image is 2 ms less than that of the compared methods.

Self-supervised 3D vehicle detection based on monocular images

The deep learning-based 3D object detection literature on monocular images has been dominated by methods that require supervision in the form of 3D bounding box annotations for training. However, obtaining sufficient 3D annotations is expensive, laborious and prone to introducing errors. To address this problem, we propose a monocular self-supervised approach towards 3D object detection relying solely on observed RGB data rather than 3D bounding boxes for training. We leverage differentiable rendering to apply visual alignment to depth maps, instance masks and point clouds for self-supervision. Furthermore, considering the complexity of autonomous driving scenes, we introduce a point cloud filter to reduce noise impact and design an automatic training set pruning strategy suitable for the self-supervised framework to further improve network performance. We provide detailed experiments on the KITTI benchmark and achieve competitive performance with existing self-supervised methods as well as some fully supervised methods.

Noninvasive Molecular Subtyping of Pediatric Low-Grade Glioma with Self-Supervised Transfer Learning.

Purpose To develop and externally test a scan-to-prediction deep learning pipeline for noninvasive, MRI-based BRAF mutational status classification for pediatric low-grade glioma. Materials and Methods This retrospective study included two pediatric low-grade glioma datasets with linked genomic and diagnostic T2-weighted MRI data of patients: Dana-Farber/Boston Children's Hospital (development dataset, n = 214 [113 (52.8%) male; 104 (48.6%) BRAF wild type, 60 (28.0%) BRAF fusion, and 50 (23.4%) BRAF V600E]) and the Children's Brain Tumor Network (external testing, n = 112 [55 (49.1%) male; 35 (31.2%) BRAF wild type, 60 (53.6%) BRAF fusion, and 17 (15.2%) BRAF V600E]). A deep learning pipeline was developed to classify BRAF mutational status (BRAF wild type vs BRAF fusion vs BRAF V600E) via a two-stage process: (a) three-dimensional tumor segmentation and extraction of axial tumor images and (b) section-wise, deep learning-based classification of mutational status. Knowledge-transfer and self-supervised approaches were investigated to prevent model overfitting, with a primary end point of the area under the receiver operating characteristic curve (AUC). To enhance model interpretability, a novel metric, center of mass distance, was developed to quantify the model attention around the tumor. Results A combination of transfer learning from a pretrained medical imaging-specific network and self-supervised label cross-training (TransferX) coupled with consensus logic yielded the highest classification performance with an AUC of 0.82 (95% CI: 0.72, 0.91), 0.87 (95% CI: 0.61, 0.97), and 0.85 (95% CI: 0.66, 0.95) for BRAF wild type, BRAF fusion, and BRAF V600E, respectively, on internal testing. On external testing, the pipeline yielded an AUC of 0.72 (95% CI: 0.64, 0.86), 0.78 (95% CI: 0.61, 0.89), and 0.72 (95% CI: 0.64, 0.88) for BRAF wild type, BRAF fusion, and BRAF V600E, respectively. Conclusion Transfer learning and self-supervised cross-training improved classification performance and generalizability for noninvasive pediatric low-grade glioma mutational status prediction in a limited data scenario. Keywords: Pediatrics, MRI, CNS, Brain/Brain Stem, Oncology, Feature Detection, Diagnosis, Supervised Learning, Transfer Learning, Convolutional Neural Network (CNN) Supplemental material is available for this article. © RSNA, 2024.

MaskCAE: Masked Convolutional AutoEncoder via Sensor Data Reconstruction for Self-Supervised Human Activity Recognition.

Self-supervised Human Activity Recognition (HAR) has been gradually gaining a lot of attention in ubiquitous computing community. Its current focus primarily lies in how to overcome the challenge of manually labeling complicated and intricate sensor data from wearable devices, which is often hard to interpret. However, current self-supervised algorithms encounter three main challenges: performance variability caused by data augmentations in contrastive learning paradigm, limitations imposed by traditional self-supervised models, and the computational load deployed on wearable devices by current mainstream transformer encoders. To comprehensively tackle these challenges, this paper proposes a powerful self-supervised approach for HAR from a novel perspective of denoising autoencoder, the first of its kind to explore how to reconstruct masked sensor data built on a commonly employed, well-designed, and computationally efficient fully convolutional network. Extensive experiments demonstrate that our proposed Masked Convolutional AutoEncoder (MaskCAE) outperforms current state-of-the-art algorithms in self-supervised, fully supervised, and semi-supervised situations without relying on any data augmentations, which fills the gap of masked sensor data modeling in HAR area. Visualization analyses show that our MaskCAE could effectively capture temporal semantics in time series sensor data, indicating its great potential in modeling abstracted sensor data. An actual implementation is evaluated on an embedded platform.