Self-supervised Learning Techniques Research Articles

Self-supervised adversarial adaptation network for breast cancer detection

Breast cancer is the most commonly diagnosed cancer worldwide, and early detection is essential for reducing mortality rates. Digital mammography is currently the best standard for early detection, as it can assist physicians in treating the disease. However, inaccurate diagnoses from mammography are common and can lead to patients undergoing unnecessary tests and treatments. To address this challenge, deep-learning techniques have shown promising results in improving the accuracy and reliability of breast cancer detection. However, existing methods face two primary challenges: the lack of the annotated data, and the inability to adapt to new data domains. In this paper, we propose SelfAdaptNet to address these issues. Specifically, SelfAdaptNet employs self-supervised learning techniques, such as Bootstrap Your Own Latent (BYOL) and Simple Framework for Learning of Visual Representations (SimCLR), to tackle the problem of limited annotated data. Additionally, the adversarial technique is used to address the problem of domain shift. By successfully reducing domain disparities, this strategy enhances the model’s adaptability and robustness across a variety of clinical scenarios. Overall, our contributions offer a more effective and flexible approach for early breast cancer detection, and experimental results demonstrate that SelfAdaptNet can produce promising results as compared with other methods.

A limited labeled data bearing fault diagnosis method based on self-supervised learning

The key focus of this study revolves around several crucial issues concerning bearing fault diagnosis, and presents an innovative solution. Conventional labeled approaches for bearing fault diagnosis often necessitate labeled data sets, which can be time-consuming or infeasible to obtain. To address this problem, an increasing amount of research has started to explore fault diagnosis methods that utilize limited labeled data. In our study, we introduce a framework for bearing fault diagnosis that incorporates wavelet transform and self-supervised learning techniques. The framework leverages vibration signals and transforms them into time-frequency spectrograms as inputs. To extract features, we employ the Swin Transformer as an encoder. Furthermore, we present a self-supervised learning approach named MoBy to address the challenge of limited labeled samples. Encouragingly, our approach achieves a diagnostic accuracy of 96.4% by utilizing only 1% labeled samples, through a well-trained encoder and a simple linear classification layer. This demonstrates outstanding performance in utilizing limited labeled data. To validate the superiority of our proposed approach, we conducted experiments on two rolling bearing fault datasets and achieved significant results.

Leveraging Imagery Data with Spatial Point Prior for Weakly Semi-supervised 3D Object Detection

Training high-accuracy 3D detectors necessitates massive labeled 3D annotations with 7 degree-of-freedom, which is laborious and time-consuming. Therefore, the form of point annotations is proposed to offer significant prospects for practical applications in 3D detection, which is not only more accessible and less expensive but also provides strong spatial information for object localization. In this paper, we empirically discover that it is non-trivial to merely adapt Point-DETR to its 3D form, encountering two main bottlenecks: 1) it fails to encode strong 3D prior into the model, and 2) it generates low-quality pseudo labels in distant regions due to the extreme sparsity of LiDAR points. To overcome these challenges, we introduce Point-DETR3D, a teacher-student framework for weakly semi-supervised 3D detection, designed to fully capitalize on point-wise supervision within a constrained instance-wise annotation budget. Different from Point-DETR which encodes 3D positional information solely through a point encoder, we propose an explicit positional query initialization strategy to enhance the positional prior. Considering the low quality of pseudo labels at distant regions produced by the teacher model, we enhance the detector's perception by incorporating dense imagery data through a novel Cross-Modal Deformable RoI Fusion (D-RoI). Moreover, an innovative point-guided self-supervised learning technique is proposed to allow for fully exploiting point priors, even in student models. Extensive experiments on representative nuScenes dataset demonstrate our Point-DETR3D obtains significant improvements compared to previous works. Notably, with only 5% of labeled data, Point-DETR3D achieves over 90% performance of its fully supervised counterpart.

Exploring Novel Self-Supervised Learning Techniques for Image Reconstruction Tasks

Image reconstruction tasks, such as super-resolution, inpainting, and denoising, play a crucial role in various computer vision applications. Traditional methods often rely heavily on large labeled datasets for training, which can be costly and time-consuming to acquire. Self-supervised learning has emerged as a promising alternative, aiming to reduce this dependency by leveraging the inherent structures within the data itself. In this paper, we explore novel self-supervised learning techniques tailored specifically for image reconstruction tasks. We propose approaches that exploit the inherent relationships between low and high-resolution images, utilize context-aware information for inpainting, and incorporate generative adversarial networks for denoising. Through extensive experimentation, we demonstrate the efficacy of our methods in achieving competitive performance compared to supervised approaches while significantly reducing the need for labeled data. Our findings pave the way for more efficient and scalable solutions in image reconstruction, offering practical benefits across a wide range of applications

Prediction of chemical reaction yields with large-scale multi-view pre-training

Developing machine learning models with high generalization capability for predicting chemical reaction yields is of significant interest and importance. The efficacy of such models depends heavily on the representation of chemical reactions, which has commonly been learned from SMILES or graphs of molecules using deep neural networks. However, the progression of chemical reactions is inherently determined by the molecular 3D geometric properties, which have been recently highlighted as crucial features in accurately predicting molecular properties and chemical reactions. Additionally, large-scale pre-training has been shown to be essential in enhancing the generalization capability of complex deep learning models. Based on these considerations, we propose the Reaction Multi-View Pre-training (ReaMVP) framework, which leverages self-supervised learning techniques and a two-stage pre-training strategy to predict chemical reaction yields. By incorporating multi-view learning with 3D geometric information, ReaMVP achieves state-of-the-art performance on two benchmark datasets. Notably, the experimental results indicate that ReaMVP has a significant advantage in predicting out-of-sample data, suggesting an enhanced generalization ability to predict new reactions. Scientific Contribution: This study presents the ReaMVP framework, which improves the generalization capability of machine learning models for predicting chemical reaction yields. By integrating sequential and geometric views and leveraging self-supervised learning techniques with a two-stage pre-training strategy, ReaMVP achieves state-of-the-art performance on benchmark datasets. The framework demonstrates superior predictive ability for out-of-sample data and enhances the prediction of new reactions.

Understanding Self-Supervised Learning of Speech Representation via Invariance and Redundancy Reduction

Self-supervised learning (SSL) has emerged as a promising paradigm for learning flexible speech representations from unlabeled data. By designing pretext tasks that exploit statistical regularities, SSL models can capture useful representations that are transferable to downstream tasks. Barlow Twins (BTs) is an SSL technique inspired by theories of redundancy reduction in human perception. In downstream tasks, BTs representations accelerate learning and transfer this learning across applications. This study applies BTs to speech data and evaluates the obtained representations on several downstream tasks, showing the applicability of the approach. However, limitations exist in disentangling key explanatory factors, with redundancy reduction and invariance alone being insufficient for factorization of learned latents into modular, compact, and informative codes. Our ablation study isolated gains from invariance constraints, but the gains were context-dependent. Overall, this work substantiates the potential of Barlow Twins for sample-efficient speech encoding. However, challenges remain in achieving fully hierarchical representations. The analysis methodology and insights presented in this paper pave a path for extensions incorporating further inductive priors and perceptual principles to further enhance the BTs self-supervision framework.

Self-Supervised Learning for Electroencephalography.

Decades of research have shown machine learning superiority in discovering highly nonlinear patterns embedded in electroencephalography (EEG) records compared with conventional statistical techniques. However, even the most advanced machine learning techniques require relatively large, labeled EEG repositories. EEG data collection and labeling are costly. Moreover, combining available datasets to achieve a large data volume is usually infeasible due to inconsistent experimental paradigms across trials. Self-supervised learning (SSL) solves these challenges because it enables learning from EEG records across trials with variable experimental paradigms, even when the trials explore different phenomena. It aggregates multiple EEG repositories to increase accuracy, reduce bias, and mitigate overfitting in machine learning training. In addition, SSL could be employed in situations where there is limited labeled training data, and manual labeling is costly. This article: 1) provides a brief introduction to SSL; 2) describes some SSL techniques employed in recent studies, including EEG; 3) proposes current and potential SSL techniques for future investigations in EEG studies; 4) discusses the cons and pros of different SSL techniques; and 5) proposes holistic implementation tips and potential future directions for EEG SSL practices.

In-Distribution and Out-of-Distribution Self-Supervised ECG Representation Learning for Arrhythmia Detection.

This paper presents a systematic investigation into the effectiveness of Self-Supervised Learning (SSL) methods for Electrocardiogram (ECG) arrhythmia detection. We begin by conducting a novel analysis of the data distributions on three popular ECG-based arrhythmia datasets: PTB-XL, Chapman, and Ribeiro. To the best of our knowledge, our study is the first to quantitatively explore and characterize these distributions in the area. We then perform a comprehensive set of experiments using different augmentations and parameters to evaluate the effectiveness of various SSL methods, namely SimCRL, BYOL, and SwAV, for ECG representation learning, where we observe the best performance achieved by SwAV. Furthermore, our analysis shows that SSL methods achieve highly competitive results to those achieved by supervised state-of-the-art methods. To further assess the performance of these methods on both In-Distribution (ID) and Out-of-Distribution (OOD) ECG data, we conduct cross-dataset training and testing experiments. Our comprehensive experiments show almost identical results when comparing ID and OOD schemes, indicating that SSL techniques can learn highly effective representations that generalize well across different OOD datasets. This finding can have major implications for ECG-based arrhythmia detection. Lastly, to further analyze our results, we perform detailed per-disease studies on the performance of the SSL methods on the three datasets.

Self-supervised learning based on Transformer for flow reconstruction and prediction

Machine learning has great potential for efficient reconstruction and prediction of flow fields. However, existing datasets may have highly diversified labels for different flow scenarios, which are not applicable for training a model. To this end, we make a first attempt to apply the self-supervised learning (SSL) technique to fluid dynamics, which disregards data labels for pre-training the model. The SSL technique embraces a large amount of data (8000 snapshots) at Reynolds numbers of Re = 200, 300, 400, and 500 without discriminating between them, which improves the generalization of the model. The Transformer model is pre-trained via a specially designed pretext task, where it reconstructs the complete flow fields after randomly masking 20% data points in each snapshot. For the downstream task of flow reconstruction, the pre-trained model is fine-tuned separately with 256 snapshots for each Reynolds number. The fine-tuned models accurately reconstruct the complete flow fields based on less than 5% random data points within a limited window even for Re = 250 and 600, whose data were not seen in the pre-trained phase. For the other downstream task of flow prediction, the pre-training model is fine-tuned separately with 128 consecutive snapshot pairs for each corresponding Reynolds number. The fine-tuned models then correctly predict the evolution of the flow fields over many periods of cycles. We compare all results generated by models trained via SSL and models trained via supervised learning, where the former has unequivocally superior performance. We expect that the methodology presented here will have wider applications in fluid mechanics.

Gamma-ray blazar classification using machine learning with advanced weight initialization and self-supervised learning techniques

ABSTRACT Machine learning has emerged as a powerful tool in the field of gamma-ray astrophysics. The algorithms can distinguish between different source types, such as blazars and pulsars, and help uncover new insights into the high-energy universe. The Large Area Telescope onboard the Fermi gamma-ray telescope has significantly advanced our understanding of the Universe. The instrument has detected a large number of gamma-ray-emitting sources, among which a significant number of objects have been identified as active galactic nuclei. The sample is primarily composed of blazars; however, more than one-third of these sources are either of an unknown class or lack a definite association with a low-energy counterpart. In this work, we employ multiple machine learning algorithms to classify the sources based on their other physical properties. In particular, we utilized smart initialization techniques and self-supervised learning for classifying blazars into BL Lacertae (BL Lac, also BLL) objects and flat-spectrum radio quasars (FSRQs). The core advantage of the algorithm is its simplicity, usage of minimum number of features and easy deployment due to lesser number of parameters without compromising on the performance along with increase in inference speed (at least seven times more than existing algorithms). As a result, the best-performing model is deployed on multiple platforms so that any user irrespective of their coding background can use the tool. The model predicts that out of the 1115 sources of uncertain type in the 4FGL-DR3 catalogue, 820 can be classified as BL Lacs and 295 can be classified as FSRQs.

Masked Modeling-Based Ultrasound Image Classification via Self-Supervised Learning.

Recently, deep learning-based methods have emerged as the preferred approach for ultrasound data analysis. However, these methods often require large-scale annotated datasets for training deep models, which are not readily available in practical scenarios. Additionally, the presence of speckle noise and other imaging artifacts can introduce numerous hard examples for ultrasound data classification. In this paper, drawing inspiration from self-supervised learning techniques, we present a pre-training method based on mask modeling specifically designed for ultrasound data. Our study investigates three different mask modeling strategies: random masking, vertical masking, and horizontal masking. By employing these strategies, our pre-training approach aims to predict the masked portion of the ultrasound images. Notably, our method does not rely on externally labeled data, allowing us to extract representative features without the need for human annotation. Consequently, we can leverage unlabeled datasets for pre-training. Furthermore, to address the challenges posed by hard samples in ultrasound data, we propose a novel hard sample mining strategy. To evaluate the effectiveness of our proposed method, we conduct experiments on two datasets. The experimental results demonstrate that our approach outperforms other state-of-the-art methods in ultrasound image classification. This indicates the superiority of our pre-training method and its ability to extract discriminative features from ultrasound data, even in the presence of hard examples.

MTS-LOF: Medical Time-Series Representation Learning via Occlusion-Invariant Features.

Medical time series data are indispensable in healthcare, providing critical insights for disease diagnosis, treatment planning, and patient management. The exponential growth in data complexity, driven by advanced sensor technologies, has presented challenges related to data labeling. Self-supervised learning (SSL) has emerged as a transformative approach to address these challenges, eliminating the need for extensive human annotation. In this study, we introduce a novel framework for Medical Time Series Representation Learning, known as MTS-LOF. MTS-LOF leverages the strengths of Joint-Embedding SSL and Masked Autoencoder (MAE) methods, offering a unique approach to representation learning for medical time series data. By combining these techniques, MTS-LOF enhances the potential of healthcare applications by providing more sophisticated, context-rich representations. Additionally, MTS-LOF employs a multi-masking strategy to facilitate occlusion-invariant feature learning. This approach allows the model to create multiple views of the data by masking portions of it. By minimizing the discrepancy between the representations of these masked patches and the fully visible patches, MTS-LOF learns to capture rich contextual information within medical time series datasets. The results of experiments conducted on diverse medical time series datasets demonstrate the superiority of MTS-LOF over other methods. These findings hold promise for significantly enhancing healthcare applications by improving representation learning. Furthermore, our work delves into the integration of Joint-Embedding SSL and MAE techniques, shedding light on the intricate interplay between temporal and structural dependencies in healthcare data. This understanding is crucial, as it allows us to grasp the complexities of healthcare data analysis.

Eigenimage2Eigenimage (E2E): A Self-Supervised Deep Learning Network for Hyperspectral Image Denoising.

The performance of deep learning-based denoisers highly depends on the quantity and quality of training data. However, paired noisy-clean training images are generally unavailable in hyperspectral remote sensing areas. To solve this problem, this work resorts to the self-supervised learning technique, where our proposed model can train itself to learn one part of noisy input from another part of noisy input. We study a general hyperspectral image (HSI) denoising framework, called Eigenimage2Eigenimage (E2E), which turns the HSI denoising problem into an eigenimage (i.e., the subspace representation coefficients of the HSI) denoising problem and proposes a learning strategy to generate noisy-noisy paired training eigenimages from noisy eigenimages. Consequently, the E2E denoising framework can be trained without clean data and applied to denoise HSIs without the constraint with the number of frequency bands. Experimental results are provided to demonstrate the performance of the proposed method that is better than the other existing deep learning methods for denoising HSIs. A MATLAB demo of this work is available at https://github.com/LinaZhuang/HSI-denoiser-Eigenimage2Eigenimagehttps://github.com/LinaZhuang/HSI-denoiser-Eigenimage2Eigenimage for the sake of reproducibility.

Measuring near-field wave elevation with monocular vision using self-supervised learning techniques

Ocean waves are among the most influential factors affecting marine mechanical systems. The existing methods for measuring wide-area wave-field data have limitations and require high maintenance costs. Therefore, it is crucial to develop a reliable and efficient field measurement method for wide-area wave fields. This study proposes a monocular vision-based approach for wave field measurement, where the depth of each pixel is equalised to summation of the depth of the mean water surface and the depth variation induced by wave elevation. A deep network was trained using a self-supervised algorithm to approximate the depth variation induced by wave elevation. A loss of representative features was proposed to improve the performance of deep networks. The images of an open-source dataset were used to train and validate the proposed model. By using the proposed method, the images were utilised to reconstruct the three-dimensional wave field of the test set. The time history and wave spectrum of the wave fields were compared with those calculated by existing method and measured by wave gauge. The influence of the training set on the measurement results was also investigated. The results indicate that the proposed method is capable of performing effective near-field wave measurements by monocular images, effectively reducing the device requirements, and increasing update frequency for ocean wave measurement.

Attention De-sparsification Matters: Inducing diversity in digital pathology representation learning

We propose DiRL, a Diversity-inducing Representation Learning technique for histopathology imaging. Self-supervised learning (SSL) techniques, such as contrastive and non-contrastive approaches, have been shown to learn rich and effective representations of digitized tissue samples with limited pathologist supervision. Our analysis of vanilla SSL-pretrained models’ attention distribution reveals an insightful observation: sparsity in attention, i.e, models tends to localize most of their attention to some prominent patterns in the image. Although attention sparsity can be beneficial in natural images due to these prominent patterns being the object of interest itself, this can be sub-optimal in digital pathology; this is because, unlike natural images, digital pathology scans are not object-centric, but rather a complex phenotype of various spatially intermixed biological components. Inadequate diversification of attention in these complex images could result in crucial information loss. To address this, we leverage cell segmentation to densely extract multiple histopathology-specific representations, and then propose a prior-guided dense pretext task, designed to match the multiple corresponding representations between the views. Through this, the model learns to attend to various components more closely and evenly, thus inducing adequate diversification in attention for capturing context-rich representations. Through quantitative and qualitative analysis on multiple tasks across cancer types, we demonstrate the efficacy of our method and observe that the attention is more globally distributed.

A knowledge-guided pre-training framework for improving molecular representation learning

Learning effective molecular feature representation to facilitate molecular property prediction is of great significance for drug discovery. Recently, there has been a surge of interest in pre-training graph neural networks (GNNs) via self-supervised learning techniques to overcome the challenge of data scarcity in molecular property prediction. However, current self-supervised learning-based methods suffer from two main obstacles: the lack of a well-defined self-supervised learning strategy and the limited capacity of GNNs. Here, we propose Knowledge-guided Pre-training of Graph Transformer (KPGT), a self-supervised learning framework to alleviate the aforementioned issues and provide generalizable and robust molecular representations. The KPGT framework integrates a graph transformer specifically designed for molecular graphs and a knowledge-guided pre-training strategy, to fully capture both structural and semantic knowledge of molecules. Through extensive computational tests on 63 datasets, KPGT exhibits superior performance in predicting molecular properties across various domains. Moreover, the practical applicability of KPGT in drug discovery has been validated by identifying potential inhibitors of two antitumor targets: hematopoietic progenitor kinase 1 (HPK1) and fibroblast growth factor receptor 1 (FGFR1). Overall, KPGT can provide a powerful and useful tool for advancing the artificial intelligence (AI)-aided drug discovery process.

CLOVER: Contrastive Learning for Onboard Vision-Enabled Robotics

Current deep-learning models employed by the planetary science community are constrained by a dearth of annotated training data for planetary images. Current models also frequently suffer from inductive bias due to domain shifts when using the same model on data obtained from different spacecraft or different time periods. Moreover, power and compute constraints preclude state-of-the-art vision models from being implemented on robotic spacecraft. In this research, we propose a self-supervised learning (SSL) framework that leverages contrastive learning techniques to improve upon state-of-the-art performance on several published Mars computer vision benchmarks. Our SSL framework enables models to be trained using fewer labels, generalize well to different tasks, and achieve higher computational efficiency. Results on published Mars computer vision benchmarks show that contrastive pretraining outperforms plain supervised learning by 2–10%. We further investigate the importance of dataset heterogeneity in mixed-domain contrastive pretraining. Using self-supervised distillation, we were also able to train a compact ResNet-18 student model to achieve better accuracy than its ResNet-152 teacher model while having 5.2 times fewer parameters. We expect that these SSL techniques will be relevant to the planning of future robotic missions, and remote sensing identification of target destinations with high scientific value.

DoubleStrokeNet: Bigram-Level Keystroke Authentication

Keystroke authentication is a well-established biometric technique that has gained significant attention due to its non-intrusive and continuous characteristics. The method analyzes the unique typing patterns of individuals to verify their identity while interacting with the keyboard, both virtual and hardware. Current deep-learning approaches like TypeNet and TypeFormer focus on generating biometric signatures as embeddings for the entire typing sequence. The authentication process is defined using the Euclidean distances between the new typing embedding and the saved biometric signatures. This paper introduces a novel approach called DoubleStrokeNet for authenticating users through keystroke analysis using bigram embeddings. Unlike conventional methods, our model targets the temporal features of bigrams to generate user embeddings. This is achieved using a Transformer-based neural network that distinguishes between different bigrams. Furthermore, we employ self-supervised learning techniques to compute embeddings for both bigrams and users. By harnessing the power of the Transformer’s attention mechanism, the DoubleStrokeNet approach represents a significant departure from existing methods. It allows for a more precise and accurate assessment of user authenticity, specifically emphasizing the temporal characteristics and latent representations of bigrams in deriving user embeddings. Our experiments were conducted using the Aalto University keystrokes datasets, which include 136 million keystrokes from 168,000 subjects using physical keyboards and 63 million keystrokes acquired on mobile devices from 60,000 subjects. The DoubleStrokeNet outperforms the TypeNet-based authentication system using 10 enrollment typing sequences, achieving Equal Error Rate (EER) values of 0.75% and 2.35% for physical and touchscreen keyboards, respectively.

Self-supervised Multimodal Graph Convolutional Network for collaborative filtering

Collaborative filtering (CF) is a central solution for capturing various user-item relationships in building recommender systems. However, when the relationships are sparsely observed, it is challenging to obtain enough signals to infer precise user preferences. Recent studies have attempted to address the sparsity issue by incorporating multimodal information (e.g., image and text) into CF models. However, existing methods mainly focus on capturing modal-specific user preference with multiple unimodal graphs, ignoring the complex nature of user behavior, which is determined by an intricate fusion of multimodal information. Therefore, we develop a Self-supervised Multimodal Graph Convolutional Network (SMGCN), which aims to learn the cross-modal user preferences over multiple modalities with an expressive multimodal fusion on a single graph.More importantly, to facilitate and enhance multimodal fusion in SMGCN, we devise two novel self-supervised learning techniques. 1) Collaborative Multimodal Alignment (CMA) uses contrastive learning to align the domain-specific multimodal semantics with the user-item relational semantics. 2) Multimodal Consistency Regularization (MCR) alleviates the sensitivity on a certain modality and increases model robustness. The experimental results demonstrate that our model consistently outperforms advanced multimodal models on three benchmark datasets.

Intent Disentanglement and Feature Self-Supervision for Novel Recommendation

One key property in recommender systems is the long-tail distribution in user-item interactions where most items only have few user feedback. Improving the recommendation of tail items can promote novelty and bring positive effects to both users and providers, and thus is a desirable property of recommender systems. Current novel recommendation studies over-emphasize the importance of tail items without differentiating the degree of users' intent on popularity and often incur a sharp decline of accuracy. Moreover, none of existing methods has ever taken the extreme case of tail items, i.e., cold-start items without any interaction, into consideration. In this work, we first disclose the mechanism that drives a user's interaction towards popular or niche items by disentangling her intent into conformity influence (popularity) and personal interests (preference). We then present a unified end-to-end framework to simultaneously optimize accuracy and novelty targets based on the disentangled intent of popularity and that of preference. We further develop a new paradigm for novel recommendation of cold-start items which exploits the self-supervised learning technique to model the correlation between collaborative features and content features. We conduct extensive experimental results on three real-world datasets. The results demonstrate that our proposed model yields significant improvements over the state-of-the-art baselines in terms of accuracy, novelty, coverage, and trade-off.