Self-supervised Learning Research Articles

Data-Driven End-to-End Optimization of Radio Over Fiber Transmission System Based on Self-Supervised Learning

Data-Driven End-to-End Optimization of Radio Over Fiber Transmission System Based on Self-Supervised Learning

Self-supervised learning for denoising of multidimensional MRI data.

To develop a fast denoising framework for high-dimensional MRI data based on a self-supervised learning scheme, which does not require ground truth clean image. Quantitative MRI faces limitations in SNR, because the variation of signal amplitude in a large set of images is the key mechanism for quantification. In addition, the complex non-linear signal models make the fitting process vulnerable to noise. To address these issues, we propose a fast deep-learning framework for denoising, which efficiently exploits the redundancy in multidimensional MRI data. A self-supervised model was designed to use only noisy images for training, bypassing the challenge of clean data paucity in clinical practice. For validation, we used two different datasets of simulated magnetization transfer contrast MR fingerprinting (MTC-MRF) dataset and in vivo DWI image dataset to show the generalizability. The proposed method drastically improved denoising performance in the presence of mild-to-severe noise regardless of noise distributions compared to previous methods of the BM3D, tMPPCA, and Patch2self. The improvements were even pronounced in the following quantification results from the denoised images. The proposed MD-S2S (Multidimensional-Self2Self) denoising technique could be further applied to various multi-dimensional MRI data and improve the quantification accuracy of tissue parameter maps.

Acoustic fingerprints in nature: A self-supervised learning approach for ecosystem activity monitoring

According to the World Health Organization, healthy communities rely on well-functioning ecosystems. Clean air, fresh water, and nutritious food are inextricably linked to ecosystem health. Changes in biological activity convey important information about ecosystem dynamics, and understanding such changes is crucial for the survival of our species. Scientific edge cyberinfrastructures collect distributed data and process it in situ, often using machine learning algorithms. Most current machine learning algorithms deployed on edge cyberinfrastructures, however, are trained on data that does not accurately represent the real stream of data collected at the edge. In this work we explore the applicability of two new self-supervised learning algorithms for characterizing an insufficiently curated, imbalanced, and unlabeled dataset collected by using a set of nine microphones at different locations at the Morton Arboretum, an internationally recognized tree-focused botanical garden and research center in Lisle, IL. Our implementations showed completely autonomous characterization capabilities, such as the separation of spectrograms by recording location, month, week, and hour of the day. The models also showed the ability to discriminate spectrograms by biological and atmospheric activity, including rain, insects, and bird activity, in a completely unsupervised fashion. We validated our findings using a supervised deep learning approach and with a dataset labeled by experts, confirming competitive performance in several features. Toward explainability of our self-supervised learning approach, we used acoustic indices and false color spectrograms, showing that the topology and orientation of the clouds of points in the output space over a 24-h period are strongly linked to the unfolding of biological activity. Our findings show that self-supervised learning has the potential to learn from and process data collected at the edge, characterizing it with minimal human intervention. We believe that further research is crucial to extending this approach for complete autonomous characterization of raw data collected on distributed sensors at the edge.

Interpreting pretext tasks for active learning: a reinforcement learning approach

As the amount of labeled data increases, the performance of deep neural networks tends to improve. However, annotating a large volume of data can be expensive. Active learning addresses this challenge by selectively annotating unlabeled data. There have been recent attempts to incorporate self-supervised learning into active learning, but there are issues in utilizing the results of self-supervised learning, i.e., it is uncertain how these should be interpreted in the context of active learning. To address this issue, we propose a multi-armed bandit approach to handle the information provided by self-supervised learning in active learning. Furthermore, we devise a data sampling process so that reinforcement learning can be effectively performed. We evaluate the proposed method on various image classification benchmarks, including CIFAR-10, CIFAR-100, Caltech-101, SVHN, and ImageNet, where the proposed method significantly improves previous approaches.

Contrastive signal-dependent plasticity: Self-supervised learning in spiking neural circuits.

Brain-inspired machine intelligence research seeks to develop computational models that emulate the information processing and adaptability that distinguishes biological systems of neurons. This has led to the development of spiking neural networks, a class of models that promisingly addresses the biological implausibility and the lack of energy efficiency inherent to modern-day deep neural networks. In this work, we address the challenge of designing neurobiologically motivated schemes for adjusting the synapses of spiking networks and propose contrastive signal-dependent plasticity, a process which generalizes ideas behind self-supervised learning to facilitate local adaptation in architectures of event-based neuronal layers that operate in parallel. Our experimental simulations demonstrate a consistent advantage over other biologically plausible approaches when training recurrent spiking networks, crucially side-stepping the need for extra structure such as feedback synapses.

Recurrence prediction of gastric cancer based on multi-resolution feature fusion and context information

Pathological images of gastric cancer serve as the gold standard for diagnosing this malignancy. However, the recurrence prediction task often encounters challenges such as insignificant morphological features of the lesions, insufficient fusion of multi-resolution features, and inability to leverage contextual information effectively. To address these issues, a three-stage recurrence prediction method based on pathological images of gastric cancer is proposed. In the first stage, the self-supervised learning framework SimCLR was adopted to train low-resolution patch images, aiming to diminish the interdependence among diverse tissue images and yield decoupled enhanced features. In the second stage, the obtained low-resolution enhanced features were fused with the corresponding high-resolution unenhanced features to achieve feature complementation across multiple resolutions. In the third stage, to address the position encoding difficulty caused by the large difference in the number of patch images, we performed position encoding based on multi-scale local neighborhoods and employed self-attention mechanism to obtain features with contextual information. The resulting contextual features were further combined with the local features extracted by the convolutional neural network. The evaluation results on clinically collected data showed that, compared with the best performance of traditional methods, the proposed network provided the best accuracy and area under curve (AUC), which were improved by 7.63% and 4.51%, respectively. These results have effectively validated the usefulness of this method in predicting gastric cancer recurrence.

Self-supervised learning for accurately modelling hierarchical evolutionary patterns of cerebrovasculature

Cerebrovascular abnormalities are critical indicators of stroke and neurodegenerative diseases like Alzheimer’s disease (AD). Understanding the normal evolution of brain vessels is essential for detecting early deviations and enabling timely interventions. Here, for the first time, we proposed a pipeline exploring the joint evolution of cortical volumes (CVs) and arterial volumes (AVs) in a large cohort of 2841 individuals. Using advanced deep learning for vessel segmentation, we built normative models of CVs and AVs across spatially hierarchical brain regions. We found that while AVs generally decline with age, distinct trends appear in regions like the circle of Willis. Comparing healthy individuals with those affected by AD or stroke, we identified significant reductions in both CVs and AVs, wherein patients with AD showing the most severe impact. Our findings reveal gender-specific effects and provide critical insights into how these conditions alter brain structure, potentially guiding future clinical assessments and interventions.

GDMol: Generative Double-Masking Self-Supervised Learning for Molecular Property Prediction.

Effective molecular feature representation is crucial for drug property prediction. Recent years have seen increased attention on graph neural networks (GNNs) that are pre-trained using self-supervised learning techniques, aiming to overcome the scarcity of labeled data in molecular property prediction. Traditional GNNs in self-supervised molecular property prediction typically perform a single masking operation on the nodes and edges of the input molecular graph, masking only local information and insufficient for thorough self-supervised training. Hence, we propose a model for molecular property prediction based on generative double-masking self-supervised learning, termed as GDMol. This integrates generative learning into the self-supervised learning framework for latent representation, and applies a second round of masking to these latent representations, enabling the model to better capture global information and semantic knowledge of the molecules for a richer, more informative representation, thereby achieving more accurate and robust molecular property prediction. Our experiments on 5 datasets demonstrated superior performance of GDMol in predicting molecular properties across different domains. Moreover, we used the masking operation to traverse through the gradient changes of each node, the magnitude and sign of which reflect the positive and negative contribution respectively of the local structure in the molecule to the prediction outcome. This in-depth interpretative analysis not only enhances the model's interpretability, but also provides more targeted insights and direction for optimizing drug molecules. In summary, this research offers novel insights on improving molecular property prediction tasks, and paves the way for further research on the application of generative learning and self-supervised learning in the field of chemistry.

A Self-Supervised One-Shot Learning Approach for Seismic Noise Reduction

A Self-Supervised One-Shot Learning Approach for Seismic Noise Reduction

Point-MPP: Point Cloud Self-Supervised Learning From Masked Position Prediction.

Masked autoencoding has gained momentum for improving fine-tuning performance in many downstream tasks. However, it tends to focus on low-level reconstruction details, lacking high-level semantics and resulting in weak transfer capability. This article presents a novel jigsaw puzzle solver inspired by the idea that predicting the positions of disordered point cloud patches provides more semantic information, similar to how children learn by solving jigsaw puzzles. Our method adopts the mask-then-predict paradigm, erasing the positions of selected point patches rather than their contents. We first partition input point clouds into irregular patches and randomly erase the positions of some patches. Then, a Transformer-based model is used to learn high-level semantic features and regress the positions of the masked patches. This approach forces the model to focus on learning transfer-robust semantics while paying less attention to low-level details. To tie the predictions within the encoding space, we further introduce a consistency constraint on their latent representations to encourage the encoded features to contain more semantic cues. We demonstrate that a standard Transformer backbone with our pretraining scheme can capture discriminative point cloud semantic information. Furthermore, extensive experiments indicate that our method outperforms the previous best competitor across six popular downstream vision tasks, achieving new state-of-the-art performance. Codes will be available at https://git.openi.org.cn/OpenPointCloud/Point-MPP.

Self-Supervised Learning for Generic Raman Spectrum Denoising.

Raman and surface-enhanced Raman scattering (SERS) spectroscopy are highly specific and sensitive optical modalities that have been extensively investigated in diverse applications. Noise reduction is demanding in the preprocessing procedure, especially for weak Raman/SERS spectra. Existing denoising methods require manual optimization of parameters, which is time-consuming and laborious and cannot always achieve satisfactory performance. Deep learning has been increasingly applied in Raman/SERS spectral denoising but usually requires massive training data, where the true labels may not exist. Aiming at these challenges, this work presents a generic Raman spectrum denoising algorithm with self-supervised learning for accurate, rapid, and robust noise reduction. A specialized network based on U-Net is established, which first extracts high-level features and then restores key peak profiles of the spectra. A subsampling strategy is proposed to refine the raw Raman spectrum and avoid the underlying biased interference. The effectiveness of the proposed approach has been validated by a broad range of spectral data, exhibiting its strong generalization ability. In the context of photosafe detection of deep-seated tumors, our method achieved signal-to-noise ratio enhancement by over 400%, which resulted in a significant increase in the limit of detection thickness from 10 to 18 cm. Our approach demonstrates superior denoising performance compared to the state-of-the-art denoising methods. The occlusion method further showed that the proposed algorithm automatically focuses on characterized peaks, enhancing the interpretability of our approach explicitly in Raman and SERS spectroscopy.

Pre-training strategy for antiviral drug screening with low-data graph neural network: A case study in HIV-1 K103N reverse transcriptase.

Graph neural networks (GNN) offer an alternative approach to boost the screening effectiveness in drug discovery. However, their efficacy is often hindered by limited datasets. To address this limitation, we introduced a robust GNN training framework, applied to various chemical databases to identify potent non-nucleoside reverse transcriptase inhibitors (NNRTIs) against the challenging K103N-mutated HIV-1 RT. Leveraging self-supervised learning (SSL) pre-training to tackle data scarcity, we screened 1,824,367 compounds, using multi-step approach that incorporated machine learning (ML)-based screening, analysis of absorption, distribution, metabolism, and excretion (ADME) prediction, drug-likeness properties, and molecular docking. Ultimately, 45 compounds were left as potential candidates with 17 of the compounds were previously identified as NNRTIs, exemplifying the model's efficacy. The remaining 28 compounds are anticipated to be repurposed for new uses. Molecular dynamics (MD) simulations on repurposed candidates unveiled two promising preclinical drugs: one designed against Plasmodium falciparum and the other serving as an antibacterial agent. Both have superior binding affinity compared to anti-HIV drugs. This conceptual framework could be adapted for other disease-specific therapeutics, facilitating the identification of potent compounds effective against both WT and mutants while revealing novel scaffolds for drug design and discovery.

DRSS: a multimodal sentiment analysis approach based on dual representation and self-supervised learning strategy

DRSS: a multimodal sentiment analysis approach based on dual representation and self-supervised learning strategy

PointSmile: point self-supervised learning via curriculum mutual information

PointSmile: point self-supervised learning via curriculum mutual information

Review of deep representation learning techniques for brain-computer interfaces.

In the field of brain-computer interfaces (BCIs), the potential for leveraging deep learning techniques for representing electroencephalogram (EEG) signals has gained substantial interest. This review synthesizes empirical findings from a collection of articles using deep representation learning techniques for BCI decoding, to provide a comprehensive analysis of the current state-of-the-art. Each article was scrutinized based on three criteria: (1) the deep representation learning technique employed, (2) the underlying motivation for its utilization, and (3) the approaches adopted for characterizing the learned representations. Among the 81 articles finally reviewed in depth, our analysis reveals a predominance of 31 articles using autoencoders. We identified 13 studies employing self-supervised learning (SSL) techniques, among which ten were published in 2022 or later, attesting to the relative youth of the field. However, at the time being, none of these have led to standard foundation models that are picked up by the BCI community. Likewise, only a few studies have introspected their learned representations. We observed that the motivation in most studies for using representation learning techniques is for solving transfer learning tasks, but we also found more specific motivations such as to learn robustness or invariances, as an algorithmic bridge, or finally to uncover the structure of the data. Given the potential of foundation models to effectively tackle these challenges, we advocate for a continued dedication to the advancement of foundation models specifically designed for EEG signal decoding by using SSL techniques. We also underline the imperative of establishing specialized benchmarks and datasets to facilitate the development and continuous improvement of such foundation models.

On the Generalization and Causal Explanation in Self-Supervised Learning

On the Generalization and Causal Explanation in Self-Supervised Learning

SVQ-MAE: an efficient speech pre-training framework with constrained computational resources

Self-supervised learning for speech pre-training models has achieved remarkable success in acquiring superior speech contextual representations by learning from unlabeled audio, excelling in numerous downstream speech tasks. However, the pre-training of these models necessitates significant computational resources and training duration, presenting a high barrier to entry into the realm of pre-training learning. In our efforts, by amalgamating the resource-efficient benefits of the generative learning model, Masked Auto Encoder, with the efficacy of the vector quantization method in discriminative learning, we introduce a novel pre-training framework: Speech Vector Quantization Masked Auto Encoder (SVQ-MAE). Distinct from the majority of SSL frameworks, which require simultaneous construction of speech contextual representations and mask reconstruction within an encoder-only module, we have exclusively designed a decoupled decoder for pre-training SVQ-MAE. This allows the additional decoupled decoder to undertake the mask reconstruction task solely, reducing the learning complexity of pretext tasks and enhancing the encoder’s efficiency in extracting speech contextual representations. Owing to this innovation, by using only 4 GPUs, SVQ-NAE can achieve high performance comparable to wav2vec 2.0, which requires 64 GPUs for training. In the Speech Processing Universal Performance Benchmark, SVQ-MAE surpasses wav2vec 2.0 in both keyword spotting and emotion recognition tasks. Furthermore, in cross-lingual ASR for Mandarin, upon fine-tuning on AISHELL-1, SVQ-MAE achieves a Character Error Rate of 4.09%, outperforming all supervised ASR models.

GCLmf: A Novel Molecular Graph Contrastive Learning Framework Based on Hard Negatives and Application in Toxicity Prediction.

In silico methods for prediction of chemical toxicity can decrease the cost and increase the efficiency in the early stage of drug discovery. However, due to low accessibility of sufficient and reliable toxicity data, constructing robust and accurate prediction models is challenging. Contrastive learning, a type of self-supervised learning, leverages large unlabeled data to obtain more expressive molecular representations, which can boost the prediction performance on downstream tasks. While molecular graph contrastive learning has gathered growing attentions, current models neglect the quality of negative data set. Here, we proposed a self-supervised pretraining deep learning framework named GCLmf. We first utilized molecular fragments that meet specific conditions as hard negative samples to boost the quality of the negative set and thus increase the difficulty of the proxy tasks during pre-training to learn informative representations. GCLmf has shown excellent predictive power on various molecular property benchmarks and demonstrates high performance in 33 toxicity tasks in comparison with multiple baselines. In addition, we further investigated the necessity of introducing hard negatives in model building and the impact of the proportion of hard negatives on the model.

Self-supervised learning from images: No negative pairs, no cluster-balancing

Learning with self-derived targets provides a non-contrastive method for unsupervised image representation learning, where the variety in targets is crucial. Recent work has achieved good performance by learning with targets obtained via cluster-balancing. However, the equal-cluster-size constraint becomes too restrictive for handling data with imbalanced categories or coming in small batches. In this paper, we propose a new clustering-based approach for non-contrastive image representation learning with no need for a particular architecture design or extra memory bank and no explicit constraints on cluster size. A key formulation is to learn embedding consistency and variable decorrelation in the cluster space by tweaking the batch-wise cross-correlation matrix towards an identity one. With this identitization loss incorporated, predicted cluster assignments of two randomly augmented views of the same image serve as targets for each other. We carried out comprehensive experimental studies of linear classification with learned representations of benchmark image datasets. Our results show that the proposed approach significantly outperforms state-of-the-art approaches and is more robust to class imbalance than those with cluster balancing.

Graph attention automatic encoder based on contrastive learning for domain recognition of spatial transcriptomics

Spatial transcriptomics is an emerging technology that enables the profiling of gene expression in tissues while preserving spatial location information. This innovative approach is anticipated to provide a comprehensive understanding of the spatial distribution of different cells within tissues and facilitate in-depth analysis of tissue structure. To accurately recognize spatial domains from spatial transcriptomics, we have introduced a generalized deep learning method called GAAEST (Graph Attention-based Autoencoder for Spatial Transcriptomics). Our proposed approach effectively integrates both spatial location information and gene expression data from spatial transcriptomics. Specifically, it leverages spatial location details to construct a neighborhood graph and employs a graph attention network-based encoder to embed gene expression information into a spatially informed space. At the same time, to further optimize the learned potential embedding, self-supervised contrastive learning is introduced to capture spatial information at three levels: local, global and contextual feature of spots. Finally, the decoder reconstructs gene expressions, which are then clustered to identify spatial domains with similar expression patterns and spatial proximity. Based on our experiments conducted on multiple datasets, GAAEST consistently outperforms existing state-of-the-art methods. The proposed GAAEST demonstrates excellent capabilities in spatial domain recognition, positioning it as an ideal tool for advancing spatial transcriptomics research.