Self-supervised Learning Techniques Research Articles

Adaptive node feature extraction in graph-based neural networks for brain diseases diagnosis using self-supervised learning

Electroencephalography (EEG) has demonstrated significant value in diagnosing brain diseases. In particular, brain networks have gained prominence as they offer additional valuable insights by establishing connections between EEG signal channels. While brain connections are typically delineated by channel signal similarity, there lacks a consistent and reliable strategy for ascertaining node characteristics. Conventional node features such as temporal and frequency domain properties of EEG signals prove inadequate for capturing the extensive EEG information. In our investigation, we introduce a novel adaptive method for extracting node features from EEG signals utilizing a distinctive task-induced self-supervised learning technique. By amalgamating these extracted node features with fundamental edge features constructed using Pearson correlation coefficients, we showed that the proposed approach can function as a plug-in module that can be integrated to many common GNN networks (e.g., GCN, GraphSAGE, GAT) as a replacement of node feature selections module. Comprehensive experiments are then conducted to demonstrate the consistently superior performance and high generality of the proposed method over other feature selection methods in various of brain disorder prediction tasks, such as depression, schizophrenia, and Parkinson’s disease. Furthermore, compared to other node features, our approach unveils profound spatial patterns through graph pooling and structural learning, shedding light on pivotal brain regions influencing various brain disorder prediction based on derived features.

Selfredepth

Depth maps produced by consumer-grade sensors suffer from inaccurate measurements and missing data from either system or scene-specific sources. Data-driven denoising algorithms can mitigate such problems; however, they require vast amounts of ground truth depth data. Recent research has tackled this limitation using self-supervised learning techniques, but it requires multiple RGB-D sensors. Moreover, most existing approaches focus on denoising single isolated depth maps or specific subjects of interest highlighting a need for methods that can effectively denoise depth maps in real-time dynamic environments. This paper extends state-of-the-art approaches for depth-denoising commodity depth devices, proposing SelfReDepth, a self-supervised deep learning technique for depth restoration, via denoising and hole-filling by inpainting of full-depth maps captured with RGB-D sensors. The algorithm targets depth data in video streams, utilizing multiple sequential depth frames coupled with color data to achieve high-quality depth videos with temporal coherence. Finally, SelfReDepth is designed to be compatible with various RGB-D sensors and usable in real-time scenarios as a pre-processing step before applying other depth-dependent algorithms. Our results demonstrate our approach’s real-time performance on real-world datasets shows that it outperforms state-of-the-art methods in denoising and restoration performance at over 30 fps on Commercial Depth Cameras, with potential benefits for augmented and mixed-reality applications.

A cognition-driven framework for few-shot class-incremental learning

Few-Shot Class-Incremental Learning (FSCIL) aims at integrating new concepts from a minimal set of instances while preserving previously acquired knowledge. This study explores the potential of Vision Transformers (ViTs) in addressing the challenges inherent in the FSCIL paradigm, such as catastrophic forgetting and overfitting problems. Drawing insights from cognitive neuroscience, we propose a Cognition-Driven Framework (CoDF) for FSCIL, leveraging Vision Transformers to emulate human cognitive processes from the aspects of Intuitive Acquisition and Structured Cognition. On the one hand, we employ self-supervised learning techniques to imbue the representations with richer information, thereby facilitating a more intuitive acquisition of essential information to solve FSCIL tasks. On the other hand, we propose to structure the learned representations by introducing biases into the prior distribution of the latent factors, which involves a multivariate Gaussian Mixture Model (GMM) and an intra-class distribution assumption. Furthermore, we apply an innovative extended warm-up strategy to effectively harness the acquired representations for downstream tasks. Comprehensive experiments on three public datasets substantiate the efficacy of our proposed framework.

Generation diffusion degradation: Simple and efficient design for blind super-resolution

In the domain of computer vision, blind super-resolution is a key area focused on generating high-resolution images with enhanced visual quality from low-resolution counterparts affected by indeterminate degradation factors. This area is primarily advanced through self-supervised learning techniques utilizing GANs. Despite their prominence, GAN-based methods encounter challenges including unstable training dynamics and limited diversity, compounded by the intricate necessity to configure degradation models to mimic various blur effects and noise types. Lately, denoising diffusion models have shown promising results in image restoration, yet their sampling efficiency constraints impede their deployment in real-time scenarios. This study introduces the Generation Diffusion Degradation (GDD) model, a novel and efficient technique for replicating image degradation by applying random Gaussian noise in a sequential manner via a parametric Markov chain, followed by a progressive reconstruction of the initial image through a U-net-based noise predictor. This method adeptly mirrors the inherent degradation distribution observed in actual degraded images. Furthermore, we present an innovative training strategy that utilizes a composite loss function to train the GDD model, ensuring stable training, improving the authenticity of the generated degraded images, and precisely reflecting the degradation patterns of target images. Extensive experimental analyses underscore the superior performance of the proposed GDD model, both in objective metrics and subjective visual quality. The code is available at https://github.com/lgylab/GDD.

Self-supervised adversarial adaptation network for breast cancer detection

Breast cancer is the most commonly diagnosed cancer worldwide, and early detection is essential for reducing mortality rates. Digital mammography is currently the best standard for early detection, as it can assist physicians in treating the disease. However, inaccurate diagnoses from mammography are common and can lead to patients undergoing unnecessary tests and treatments. To address this challenge, deep-learning techniques have shown promising results in improving the accuracy and reliability of breast cancer detection. However, existing methods face two primary challenges: the lack of the annotated data, and the inability to adapt to new data domains. In this paper, we propose SelfAdaptNet to address these issues. Specifically, SelfAdaptNet employs self-supervised learning techniques, such as Bootstrap Your Own Latent (BYOL) and Simple Framework for Learning of Visual Representations (SimCLR), to tackle the problem of limited annotated data. Additionally, the adversarial technique is used to address the problem of domain shift. By successfully reducing domain disparities, this strategy enhances the model’s adaptability and robustness across a variety of clinical scenarios. Overall, our contributions offer a more effective and flexible approach for early breast cancer detection, and experimental results demonstrate that SelfAdaptNet can produce promising results as compared with other methods.

A limited labeled data bearing fault diagnosis method based on self-supervised learning

The key focus of this study revolves around several crucial issues concerning bearing fault diagnosis, and presents an innovative solution. Conventional labeled approaches for bearing fault diagnosis often necessitate labeled data sets, which can be time-consuming or infeasible to obtain. To address this problem, an increasing amount of research has started to explore fault diagnosis methods that utilize limited labeled data. In our study, we introduce a framework for bearing fault diagnosis that incorporates wavelet transform and self-supervised learning techniques. The framework leverages vibration signals and transforms them into time-frequency spectrograms as inputs. To extract features, we employ the Swin Transformer as an encoder. Furthermore, we present a self-supervised learning approach named MoBy to address the challenge of limited labeled samples. Encouragingly, our approach achieves a diagnostic accuracy of 96.4% by utilizing only 1% labeled samples, through a well-trained encoder and a simple linear classification layer. This demonstrates outstanding performance in utilizing limited labeled data. To validate the superiority of our proposed approach, we conducted experiments on two rolling bearing fault datasets and achieved significant results.

Leveraging Imagery Data with Spatial Point Prior for Weakly Semi-supervised 3D Object Detection

Training high-accuracy 3D detectors necessitates massive labeled 3D annotations with 7 degree-of-freedom, which is laborious and time-consuming. Therefore, the form of point annotations is proposed to offer significant prospects for practical applications in 3D detection, which is not only more accessible and less expensive but also provides strong spatial information for object localization. In this paper, we empirically discover that it is non-trivial to merely adapt Point-DETR to its 3D form, encountering two main bottlenecks: 1) it fails to encode strong 3D prior into the model, and 2) it generates low-quality pseudo labels in distant regions due to the extreme sparsity of LiDAR points. To overcome these challenges, we introduce Point-DETR3D, a teacher-student framework for weakly semi-supervised 3D detection, designed to fully capitalize on point-wise supervision within a constrained instance-wise annotation budget. Different from Point-DETR which encodes 3D positional information solely through a point encoder, we propose an explicit positional query initialization strategy to enhance the positional prior. Considering the low quality of pseudo labels at distant regions produced by the teacher model, we enhance the detector's perception by incorporating dense imagery data through a novel Cross-Modal Deformable RoI Fusion (D-RoI). Moreover, an innovative point-guided self-supervised learning technique is proposed to allow for fully exploiting point priors, even in student models. Extensive experiments on representative nuScenes dataset demonstrate our Point-DETR3D obtains significant improvements compared to previous works. Notably, with only 5% of labeled data, Point-DETR3D achieves over 90% performance of its fully supervised counterpart.

Exploring Novel Self-Supervised Learning Techniques for Image Reconstruction Tasks

Image reconstruction tasks, such as super-resolution, inpainting, and denoising, play a crucial role in various computer vision applications. Traditional methods often rely heavily on large labeled datasets for training, which can be costly and time-consuming to acquire. Self-supervised learning has emerged as a promising alternative, aiming to reduce this dependency by leveraging the inherent structures within the data itself. In this paper, we explore novel self-supervised learning techniques tailored specifically for image reconstruction tasks. We propose approaches that exploit the inherent relationships between low and high-resolution images, utilize context-aware information for inpainting, and incorporate generative adversarial networks for denoising. Through extensive experimentation, we demonstrate the efficacy of our methods in achieving competitive performance compared to supervised approaches while significantly reducing the need for labeled data. Our findings pave the way for more efficient and scalable solutions in image reconstruction, offering practical benefits across a wide range of applications

Prediction of chemical reaction yields with large-scale multi-view pre-training

Developing machine learning models with high generalization capability for predicting chemical reaction yields is of significant interest and importance. The efficacy of such models depends heavily on the representation of chemical reactions, which has commonly been learned from SMILES or graphs of molecules using deep neural networks. However, the progression of chemical reactions is inherently determined by the molecular 3D geometric properties, which have been recently highlighted as crucial features in accurately predicting molecular properties and chemical reactions. Additionally, large-scale pre-training has been shown to be essential in enhancing the generalization capability of complex deep learning models. Based on these considerations, we propose the Reaction Multi-View Pre-training (ReaMVP) framework, which leverages self-supervised learning techniques and a two-stage pre-training strategy to predict chemical reaction yields. By incorporating multi-view learning with 3D geometric information, ReaMVP achieves state-of-the-art performance on two benchmark datasets. Notably, the experimental results indicate that ReaMVP has a significant advantage in predicting out-of-sample data, suggesting an enhanced generalization ability to predict new reactions. Scientific Contribution: This study presents the ReaMVP framework, which improves the generalization capability of machine learning models for predicting chemical reaction yields. By integrating sequential and geometric views and leveraging self-supervised learning techniques with a two-stage pre-training strategy, ReaMVP achieves state-of-the-art performance on benchmark datasets. The framework demonstrates superior predictive ability for out-of-sample data and enhances the prediction of new reactions.

Understanding Self-Supervised Learning of Speech Representation via Invariance and Redundancy Reduction

Self-supervised learning (SSL) has emerged as a promising paradigm for learning flexible speech representations from unlabeled data. By designing pretext tasks that exploit statistical regularities, SSL models can capture useful representations that are transferable to downstream tasks. Barlow Twins (BTs) is an SSL technique inspired by theories of redundancy reduction in human perception. In downstream tasks, BTs representations accelerate learning and transfer this learning across applications. This study applies BTs to speech data and evaluates the obtained representations on several downstream tasks, showing the applicability of the approach. However, limitations exist in disentangling key explanatory factors, with redundancy reduction and invariance alone being insufficient for factorization of learned latents into modular, compact, and informative codes. Our ablation study isolated gains from invariance constraints, but the gains were context-dependent. Overall, this work substantiates the potential of Barlow Twins for sample-efficient speech encoding. However, challenges remain in achieving fully hierarchical representations. The analysis methodology and insights presented in this paper pave a path for extensions incorporating further inductive priors and perceptual principles to further enhance the BTs self-supervision framework.

Self-Supervised Learning for Electroencephalography.

Decades of research have shown machine learning superiority in discovering highly nonlinear patterns embedded in electroencephalography (EEG) records compared with conventional statistical techniques. However, even the most advanced machine learning techniques require relatively large, labeled EEG repositories. EEG data collection and labeling are costly. Moreover, combining available datasets to achieve a large data volume is usually infeasible due to inconsistent experimental paradigms across trials. Self-supervised learning (SSL) solves these challenges because it enables learning from EEG records across trials with variable experimental paradigms, even when the trials explore different phenomena. It aggregates multiple EEG repositories to increase accuracy, reduce bias, and mitigate overfitting in machine learning training. In addition, SSL could be employed in situations where there is limited labeled training data, and manual labeling is costly. This article: 1) provides a brief introduction to SSL; 2) describes some SSL techniques employed in recent studies, including EEG; 3) proposes current and potential SSL techniques for future investigations in EEG studies; 4) discusses the cons and pros of different SSL techniques; and 5) proposes holistic implementation tips and potential future directions for EEG SSL practices.

In-Distribution and Out-of-Distribution Self-Supervised ECG Representation Learning for Arrhythmia Detection.

This paper presents a systematic investigation into the effectiveness of Self-Supervised Learning (SSL) methods for Electrocardiogram (ECG) arrhythmia detection. We begin by conducting a novel analysis of the data distributions on three popular ECG-based arrhythmia datasets: PTB-XL, Chapman, and Ribeiro. To the best of our knowledge, our study is the first to quantitatively explore and characterize these distributions in the area. We then perform a comprehensive set of experiments using different augmentations and parameters to evaluate the effectiveness of various SSL methods, namely SimCRL, BYOL, and SwAV, for ECG representation learning, where we observe the best performance achieved by SwAV. Furthermore, our analysis shows that SSL methods achieve highly competitive results to those achieved by supervised state-of-the-art methods. To further assess the performance of these methods on both In-Distribution (ID) and Out-of-Distribution (OOD) ECG data, we conduct cross-dataset training and testing experiments. Our comprehensive experiments show almost identical results when comparing ID and OOD schemes, indicating that SSL techniques can learn highly effective representations that generalize well across different OOD datasets. This finding can have major implications for ECG-based arrhythmia detection. Lastly, to further analyze our results, we perform detailed per-disease studies on the performance of the SSL methods on the three datasets.

Self-supervised learning based on Transformer for flow reconstruction and prediction

Machine learning has great potential for efficient reconstruction and prediction of flow fields. However, existing datasets may have highly diversified labels for different flow scenarios, which are not applicable for training a model. To this end, we make a first attempt to apply the self-supervised learning (SSL) technique to fluid dynamics, which disregards data labels for pre-training the model. The SSL technique embraces a large amount of data (8000 snapshots) at Reynolds numbers of Re = 200, 300, 400, and 500 without discriminating between them, which improves the generalization of the model. The Transformer model is pre-trained via a specially designed pretext task, where it reconstructs the complete flow fields after randomly masking 20% data points in each snapshot. For the downstream task of flow reconstruction, the pre-trained model is fine-tuned separately with 256 snapshots for each Reynolds number. The fine-tuned models accurately reconstruct the complete flow fields based on less than 5% random data points within a limited window even for Re = 250 and 600, whose data were not seen in the pre-trained phase. For the other downstream task of flow prediction, the pre-training model is fine-tuned separately with 128 consecutive snapshot pairs for each corresponding Reynolds number. The fine-tuned models then correctly predict the evolution of the flow fields over many periods of cycles. We compare all results generated by models trained via SSL and models trained via supervised learning, where the former has unequivocally superior performance. We expect that the methodology presented here will have wider applications in fluid mechanics.

Gamma-ray blazar classification using machine learning with advanced weight initialization and self-supervised learning techniques

ABSTRACT Machine learning has emerged as a powerful tool in the field of gamma-ray astrophysics. The algorithms can distinguish between different source types, such as blazars and pulsars, and help uncover new insights into the high-energy universe. The Large Area Telescope onboard the Fermi gamma-ray telescope has significantly advanced our understanding of the Universe. The instrument has detected a large number of gamma-ray-emitting sources, among which a significant number of objects have been identified as active galactic nuclei. The sample is primarily composed of blazars; however, more than one-third of these sources are either of an unknown class or lack a definite association with a low-energy counterpart. In this work, we employ multiple machine learning algorithms to classify the sources based on their other physical properties. In particular, we utilized smart initialization techniques and self-supervised learning for classifying blazars into BL Lacertae (BL Lac, also BLL) objects and flat-spectrum radio quasars (FSRQs). The core advantage of the algorithm is its simplicity, usage of minimum number of features and easy deployment due to lesser number of parameters without compromising on the performance along with increase in inference speed (at least seven times more than existing algorithms). As a result, the best-performing model is deployed on multiple platforms so that any user irrespective of their coding background can use the tool. The model predicts that out of the 1115 sources of uncertain type in the 4FGL-DR3 catalogue, 820 can be classified as BL Lacs and 295 can be classified as FSRQs.

Masked Modeling-Based Ultrasound Image Classification via Self-Supervised Learning.

Recently, deep learning-based methods have emerged as the preferred approach for ultrasound data analysis. However, these methods often require large-scale annotated datasets for training deep models, which are not readily available in practical scenarios. Additionally, the presence of speckle noise and other imaging artifacts can introduce numerous hard examples for ultrasound data classification. In this paper, drawing inspiration from self-supervised learning techniques, we present a pre-training method based on mask modeling specifically designed for ultrasound data. Our study investigates three different mask modeling strategies: random masking, vertical masking, and horizontal masking. By employing these strategies, our pre-training approach aims to predict the masked portion of the ultrasound images. Notably, our method does not rely on externally labeled data, allowing us to extract representative features without the need for human annotation. Consequently, we can leverage unlabeled datasets for pre-training. Furthermore, to address the challenges posed by hard samples in ultrasound data, we propose a novel hard sample mining strategy. To evaluate the effectiveness of our proposed method, we conduct experiments on two datasets. The experimental results demonstrate that our approach outperforms other state-of-the-art methods in ultrasound image classification. This indicates the superiority of our pre-training method and its ability to extract discriminative features from ultrasound data, even in the presence of hard examples.

A Generalisable Heartbeat Classifier Leveraging Self-Supervised Learning for ECG Analysis During Magnetic Resonance Imaging.

Electrocardiogram (ECG) is acquired during Magnetic Resonance Imaging (MRI) to monitor patients and synchronize image acquisition with the heart motion. ECG signals are highly distorted during MRI due to the complex electromagnetic environment. Automated ECG analysis is therefore complicated in this context and there is no reference technique in MRI to classify pathological heartbeats. Imaging arrhythmic patients is hence difficult in MRI. Deep Learning based heartbeat classifier have been suggested but require large databases whereas existing annotated sets of ECG in MRI are very small. We proposed a Siamese network to leverage a large database of unannotated ECGs outside MRI. This was used to develop an efficient representation of ECG signals, further used to develop a heartbeat classifier. We extensively tested several data augmentations and self-supervised learning (SSL) techniques and assessed the generalization of the obtained classifier to ECG signals acquired in MRI. These augmentations included random noises and a model simulating MRI specific artefacts. SSL pretraining improved the generalizability of heartbeat classifiers in MRI (F1=0.75) compared to Deep Learning not relying on SSL (F1=0.46) and another classical machine learning approach (F1=0.40). These promising results seem to indicate that the use of SSL techniques can learn efficient ECG signal representation, and are useful for the development of Deep Learning models even when only scarce annotated medical data are available.

MTS-LOF: Medical Time-Series Representation Learning via Occlusion-Invariant Features.

Medical time series data are indispensable in healthcare, providing critical insights for disease diagnosis, treatment planning, and patient management. The exponential growth in data complexity, driven by advanced sensor technologies, has presented challenges related to data labeling. Self-supervised learning (SSL) has emerged as a transformative approach to address these challenges, eliminating the need for extensive human annotation. In this study, we introduce a novel framework for Medical Time Series Representation Learning, known as MTS-LOF. MTS-LOF leverages the strengths of Joint-Embedding SSL and Masked Autoencoder (MAE) methods, offering a unique approach to representation learning for medical time series data. By combining these techniques, MTS-LOF enhances the potential of healthcare applications by providing more sophisticated, context-rich representations. Additionally, MTS-LOF employs a multi-masking strategy to facilitate occlusion-invariant feature learning. This approach allows the model to create multiple views of the data by masking portions of it. By minimizing the discrepancy between the representations of these masked patches and the fully visible patches, MTS-LOF learns to capture rich contextual information within medical time series datasets. The results of experiments conducted on diverse medical time series datasets demonstrate the superiority of MTS-LOF over other methods. These findings hold promise for significantly enhancing healthcare applications by improving representation learning. Furthermore, our work delves into the integration of Joint-Embedding SSL and MAE techniques, shedding light on the intricate interplay between temporal and structural dependencies in healthcare data. This understanding is crucial, as it allows us to grasp the complexities of healthcare data analysis.

Eigenimage2Eigenimage (E2E): A Self-Supervised Deep Learning Network for Hyperspectral Image Denoising.

The performance of deep learning-based denoisers highly depends on the quantity and quality of training data. However, paired noisy-clean training images are generally unavailable in hyperspectral remote sensing areas. To solve this problem, this work resorts to the self-supervised learning technique, where our proposed model can train itself to learn one part of noisy input from another part of noisy input. We study a general hyperspectral image (HSI) denoising framework, called Eigenimage2Eigenimage (E2E), which turns the HSI denoising problem into an eigenimage (i.e., the subspace representation coefficients of the HSI) denoising problem and proposes a learning strategy to generate noisy-noisy paired training eigenimages from noisy eigenimages. Consequently, the E2E denoising framework can be trained without clean data and applied to denoise HSIs without the constraint with the number of frequency bands. Experimental results are provided to demonstrate the performance of the proposed method that is better than the other existing deep learning methods for denoising HSIs. A MATLAB demo of this work is available at https://github.com/LinaZhuang/HSI-denoiser-Eigenimage2Eigenimagehttps://github.com/LinaZhuang/HSI-denoiser-Eigenimage2Eigenimage for the sake of reproducibility.

Measuring near-field wave elevation with monocular vision using self-supervised learning techniques

Ocean waves are among the most influential factors affecting marine mechanical systems. The existing methods for measuring wide-area wave-field data have limitations and require high maintenance costs. Therefore, it is crucial to develop a reliable and efficient field measurement method for wide-area wave fields. This study proposes a monocular vision-based approach for wave field measurement, where the depth of each pixel is equalised to summation of the depth of the mean water surface and the depth variation induced by wave elevation. A deep network was trained using a self-supervised algorithm to approximate the depth variation induced by wave elevation. A loss of representative features was proposed to improve the performance of deep networks. The images of an open-source dataset were used to train and validate the proposed model. By using the proposed method, the images were utilised to reconstruct the three-dimensional wave field of the test set. The time history and wave spectrum of the wave fields were compared with those calculated by existing method and measured by wave gauge. The influence of the training set on the measurement results was also investigated. The results indicate that the proposed method is capable of performing effective near-field wave measurements by monocular images, effectively reducing the device requirements, and increasing update frequency for ocean wave measurement.

Attention De-sparsification Matters: Inducing diversity in digital pathology representation learning

We propose DiRL, a Diversity-inducing Representation Learning technique for histopathology imaging. Self-supervised learning (SSL) techniques, such as contrastive and non-contrastive approaches, have been shown to learn rich and effective representations of digitized tissue samples with limited pathologist supervision. Our analysis of vanilla SSL-pretrained models’ attention distribution reveals an insightful observation: sparsity in attention, i.e, models tends to localize most of their attention to some prominent patterns in the image. Although attention sparsity can be beneficial in natural images due to these prominent patterns being the object of interest itself, this can be sub-optimal in digital pathology; this is because, unlike natural images, digital pathology scans are not object-centric, but rather a complex phenotype of various spatially intermixed biological components. Inadequate diversification of attention in these complex images could result in crucial information loss. To address this, we leverage cell segmentation to densely extract multiple histopathology-specific representations, and then propose a prior-guided dense pretext task, designed to match the multiple corresponding representations between the views. Through this, the model learns to attend to various components more closely and evenly, thus inducing adequate diversification in attention for capturing context-rich representations. Through quantitative and qualitative analysis on multiple tasks across cancer types, we demonstrate the efficacy of our method and observe that the attention is more globally distributed.