Self-supervised Network Research Articles

Enhanced Inter-Trace Variations Extraction via a Self-Supervised Network for Pre-Stack Seismic Analysis

Enhanced Inter-Trace Variations Extraction via a Self-Supervised Network for Pre-Stack Seismic Analysis

MSL-CCRN: Multi-stage self-supervised learning based cross-modality contrastive representation network for infrared and visible image fusion

Infrared and visible image fusion (IVIF) facing different information in two modal scenarios, the focus of research is to better extract different information. In this work, we propose a multi-stage self-supervised learning based cross-modality contrastive representation network for infrared and visible image fusion (MSL-CCRN). Firstly, considering that the scene differences between different modalities affect the fusion of cross-modal images, we propose a contrastive representation network (CRN). CRN enhances the interaction between the fused image and the source image, and significantly improves the similarity between the meaningful features in each modality and the fused image. Secondly, due to the lack of ground truth in IVIF, the quality of directly obtained fused image is seriously affected. We design a multi-stage fusion strategy to address the loss of important information in this process. Notably, our method is a self-supervised network. In fusion stage I, we reconstruct the initial fused image as the new view of fusion stage II. In fusion stage II, we use the fused image obtained in the previous stage to carry out three-view contrastive representation, thereby constraining the feature extraction of the source image. This makes the final fused image introduce more important information in the source image. Through a large number of qualitative, quantitative experiments and downstream object detection experiments, our propose method shows excellent performance compared with most advanced methods.

Self-supervised network for oriented synthetic aperture radar ship detection based on self-distillation

Self-supervised network for oriented synthetic aperture radar ship detection based on self-distillation

A Foundation Language-Image Model of the Retina (FLAIR): encoding expert knowledge in text supervision

Foundation vision-language models are currently transforming computer vision, and are on the rise in medical imaging fueled by their very promising generalization capabilities. However, the initial attempts to transfer this new paradigm to medical imaging have shown less impressive performances than those observed in other domains, due to the significant domain shift and the complex, expert domain knowledge inherent to medical-imaging tasks. Motivated by the need for domain-expert foundation models, we present FLAIR, a pre-trained vision-language model for universal retinal fundus image understanding. To this end, we compiled 38 open-access, mostly categorical fundus imaging datasets from various sources, with up to 101 different target conditions and 288,307 images. We integrate the expert’s domain knowledge in the form of descriptive textual prompts, during both pre-training and zero-shot inference, enhancing the less-informative categorical supervision of the data. Such a textual expert’s knowledge, which we compiled from the relevant clinical literature and community standards, describes the fine-grained features of the pathologies as well as the hierarchies and dependencies between them. We report comprehensive evaluations, which illustrate the benefit of integrating expert knowledge and the strong generalization capabilities of FLAIR under difficult scenarios with domain shifts or unseen categories. When adapted with a lightweight linear probe, FLAIR outperforms fully-trained, dataset-focused models, more so in the few-shot regimes. Interestingly, FLAIR outperforms by a wide margin larger-scale generalist image-language models and retina domain-specific self-supervised networks, which emphasizes the potential of embedding experts’ domain knowledge and the limitations of generalist models in medical imaging. The pre-trained model is available at: https://github.com/jusiro/FLAIR.

Lightweight monocular depth estimation using a fusion-improved transformer

The existing deep estimation networks often overlook the issue of computational efficiency while pursuing high accuracy. This paper proposes a lightweight self-supervised network that combines convolutional neural networks (CNN) and Transformers as the feature extraction and encoding layers for images, enabling the network to capture both local geometric and global semantic features for depth estimation. First, depth-separable convolution is used to construct a dilated convolution residual module based on a shallow network to improve the shallow CNN feature extraction receptive field. In the transformer, a multidepth separable convolution head transposed attention module is proposed to reduce the computational burden of spatial self-attention. In the feedforward network, a two-step gating mechanism is proposed to improve the nonlinear representation ability of the feedforward network. Finally, the CNN and transformer are integrated to implement a depth estimation network with a local-global context interaction function. Compared with other lightweight models, this model has fewer model parameters and higher estimation accuracy. It also has better generalizability for different outdoor datasets. Additionally, the inference speed can reach 87 FPS, achieving better real-time performance and accounting for both inference speed and estimation accuracy.

PSSS-EEG: A Probabilistic-masking Self-Supervised Swin-transformer model for EEG-based drowsiness recognition

Drowsiness recognition based on Electroencephalography (EEG) signals is a critical task with wide-ranging applications in areas such as transportation safety. However, existing methods for EEG-based drowsiness recognition have demonstrated limited performance in data-limited scenarios and struggle with capturing the diverse patterns among different subjects. In this paper, we propose a novel approach, the Probabilistic-masking Self-Supervised Swin-transformer model (PSSS-EEG), for accurate and robust EEG-based drowsiness recognition. The PSSS-EEG model integrates data augmentation and self-supervised learning techniques to improve the recognition accuracy and generalization ability of drowsiness recognition. Specifically, we introduce the incomplete probabilistic-masking operation to the data augmentation. Moreover, the swin-transformer architecture is employed as the backbone of the self-supervised network to capture long-range dependencies at different scales in EEG data. The retention weights are introduced into the loss function as a linkage between data augmentation and self-supervised network. Experimental results demonstrate that our model outperforms existing methods in EEG-based drowsiness recognition accuracy, achieving an accuracy exceeding 80%. In addition, this work highlights the importance of advanced data augmentation techniques and self-supervised learning models. It also contributes to filling a crucial gap in the existing literature about the integration of data augmentation and self-supervised learning, providing a more effective and adaptable solution for EEG-based recognition.

Improving correlation based super-resolution microscopy images through image fusion by self-supervised deep learning.

Super-resolution imaging is a powerful tool in modern biological research, allowing for the optical observation of subcellular structures with great detail. In this paper, we present a deep learning approach for image fusion of intensity and super-resolution optical fluctuation imaging (SOFI) microscopy images. We construct a network that can successfully combine the advantages of these two imaging methods, producing a fused image with a resolution comparable to that of SOFI and an SNR comparable to that of the intensity image. We also demonstrate the effectiveness of our approach experimentally, specifically on cell samples where microtubules were stained with ATTO647N and imaged using a confocal microscope with a single photon fiber bundle camera, allowing for the simultaneous acquisition of an image scanning microscopy (ISM) image and a SOFISM (ISM and SOFI) image. Our network is designed as a self-supervised network and shows the ability to train on a single pair of images and to generalize to other image pairs without the need for additional training. Our approach offers a flexible and efficient way to combine the strengths of correlation based imaging techniques along with traditional intensity based microscopy, and can be readily applied to other fluctuation based imaging modalities.

Self-supervised network for low-light traffic image enhancement based on deep noise and artifacts removal

In the intelligent transportation system (ITS), detecting vehicles and pedestrians in low-light conditions is challenging due to the low contrast between objects and the background. Recently, many works have enhanced low-light images using deep learning-based methods, but these methods require paired images during training, which are impractical to obtain in real-world traffic scenarios. Therefore, we propose a self-supervised network (SSN) for low-light traffic image enhancement that can be trained without paired images. To avoid amplifying noise and artifacts in the processed image during enhancement, we first proposed a denoising net to reduce the noise and artifacts in the input image. Then the processed image can be enhanced by the enhancement net. Considering the compression of the traffic image, we designed an artifacts removal net to improve the quality of the enhanced image. We proposed several effective and differential losses to make SSN trainable with low-light images only. To better integrate the extracted features from different levels in the network, we also proposed an attention module named the multi-head non-local block. In experiments, we evaluated SSN and other low-light image enhancement methods on two low-light traffic image sets: the Berkeley Deep Drive (BDD) dataset and the Hong Kong night-time multi-class vehicle (HK) dataset. The results indicated that SSN significantly improves upon other methods in visual comparison and some blind image quality metrics. We also conducted comparisons on classical ITS tasks like vehicle detection on the images enhanced by SSN and other methods, which further verified its effectiveness.

Self-supervised network distillation: An effective approach to exploration in sparse reward environments

Reinforcement learning can solve decision-making problems and train an agent to behave in an environment according to a predesigned reward function. However, such an approach becomes very problematic if the reward is too sparse and so the agent does not come across the reward during the environmental exploration. The solution to such a problem may be to equip the agent with an intrinsic motivation that will provide informed exploration during which the agent is likely to also encounter external reward. Novelty detection is one of the promising branches of intrinsic motivation research. We present Self-supervised Network Distillation (SND), a class of intrinsic motivation algorithms based on the distillation error as a novelty indicator, where the predictor model and the target model are both trained. We adapted three existing self-supervised methods for this purpose and experimentally tested them on a set of ten environments that are considered difficult to explore. The results show that our approach achieves faster growth and higher external reward for the same training time compared to the baseline models, which implies improved exploration in a very sparse reward environment.11The source code is available at https://github.com/Iskandor/SND and https://github.com/michalnand/reinforcement_learning. The video of our trained agents is available at https://youtu.be/-vDg_r2ZetI. In addition, the analytical methods we applied provide valuable explanatory insights into our proposed models.

Partial Multi-View Clustering via Self-Supervised Network

Partial multi-view clustering is a challenging and practical research problem for data analysis in real-world applications, due to the potential data missing issue in different views. However, most existing methods have not fully explored the correlation information among various incomplete views. In addition, these existing clustering methods always ignore discovering discriminative features inside the data itself in this unsupervised task. To tackle these challenges, we propose Partial Multi-View Clustering via Self-Supervised \textbf{N}etwork (PVC-SSN) in this paper. Specifically, we employ contrastive learning to obtain a more discriminative and consistent subspace representation, which is guided by a self-supervised module. Self-supervised learning can exploit effective cluster information through the data itself to guide the learning process of clustering tasks. Thus, it can pull together embedding features from the same cluster and push apart these from different clusters. Extensive experiments on several benchmark datasets show that the proposed PVC-SCN method outperforms several state-of-the-art clustering methods.

DSCNet: lightweight and efficient self-supervised network via depthwise separable cross convolution blocks for speckle image matching.

Speckle structured light has become a research hotspot due to its ability to acquire target three-dimensional information with single image projection in recent years. To address the challenges of a low number of extracted speckle feature points, high mismatch rate and poor real-time performance in traditional algorithms, as well as the obstacle of requiring expensive annotation data in deep learning-based methods, a lightweight and efficient self-supervised convolutional neural network (CNN) is proposed to achieve high-precision and rapid matching of speckle images. First, to efficiently utilize the speckle projection information, a feature extraction backbone based on the depthwise separable cross convolution blocks is proposed. Second, in the feature detection module, a softargmax detection head is designed to refine the coordinates of speckle feature points to sub-pixel accuracy. In the feature description module, a coarse-to-fine module is presented to further refine matching accuracy. Third, we adopt strategies of transfer learning and self-supervised learning to improve the generalization and feature representation capabilities of the model. Data augmentation and real-time training techniques are used to improve the robustness of the model. The experimental results show that the proposed method achieves a mean matching accuracy of 91.62% for speckle feature points on the pilot's helmet, with mere 0.95% mismatch rate. The full model runs at 42ms for a speckle image pair on an RTX 3060.

Joint MAPLE: Accelerated joint T1 and mapping with scan-specific self-supervised networks.

Quantitative MRI finds important applications in clinical and research studies. However, it is encoding intensive and may suffer from prohibitively long scan times. Accelerated MR parameter mapping techniques have been developed to help address these challenges. Here, an accelerated joint T1, , frequency and proton density mapping technique with scan-specific self-supervised network reconstruction is proposed to synergistically combine parallel imaging, model-based, and deep learning approaches to speed up parameter mapping. Proposed framework, Joint MAPLE, includes parallel imaging, signal modeling, and data consistency blocks which are optimized jointly in a combined loss function. A scan-specific self-supervised reconstruction is embedded into the framework, which takes advantage of multi-contrast data from a multi-echo, multi-flip angle, gradient echo acquisition. In comparison with parallel reconstruction techniques powered by low-rank methods, emerging scan specific networks, and model-based estimation approaches, the proposed framework reduces the reconstruction error in parameter maps by approximately two-fold on average at acceleration rates as high as R = 16 with uniform sampling. It can outperform evaluated parallel reconstruction techniques up to four-fold on average in the presence of challenging sub-sampling masks. It is observed that Joint MAPLE performs well at extreme acceleration rates of R = 25 and R = 36 with error values less than 20%. Joint MAPLE enables higher fidelity parameter estimation at high acceleration rates by synergistically combining parallel imaging and model-based parameter mapping and exploiting multi-echo, multi-flip angle datasets. Utilizing a scan-specific self-supervised reconstruction obviates the need for large data sets for training while improving the parameter estimation ability.

Monocular 3D Reconstruction of Blast Furnace Burden Surface Based on Cross-Domain Generative Self-Supervised Network

Monocular 3D Reconstruction of Blast Furnace Burden Surface Based on Cross-Domain Generative Self-Supervised Network

DSMT-Net: Dual Self-Supervised Multi-Operator Transformation for Multi-Source Endoscopic Ultrasound Diagnosis.

Pancreatic cancer has the worst prognosis of all cancers. The clinical application of endoscopic ultrasound (EUS) for the assessment of pancreatic cancer risk and of deep learning for the classification of EUS images have been hindered by inter-grader variability and labeling capability. One of the key reasons for these difficulties is that EUS images are obtained from multiple sources with varying resolutions, effective regions, and interference signals, making the distribution of the data highly variable and negatively impacting the performance of deep learning models. Additionally, manual labeling of images is time-consuming and requires significant effort, leading to the desire to effectively utilize a large amount of unlabeled data for network training. To address these challenges, this study proposes the Dual Self-supervised Multi-Operator Transformation Network (DSMT-Net) for multi-source EUS diagnosis. The DSMT-Net includes a multi-operator transformation approach to standardize the extraction of regions of interest in EUS images and eliminate irrelevant pixels. Furthermore, a transformer-based dual self-supervised network is designed to integrate unlabeled EUS images for pre-training the representation model, which can be transferred to supervised tasks such as classification, detection, and segmentation. A large-scale EUS-based pancreas image dataset (LEPset) has been collected, including 3,500 pathologically proven labeled EUS images (from pancreatic and non-pancreatic cancers) and 8,000 unlabeled EUS images for model development. The self-supervised method has also been applied to breast cancer diagnosis and was compared to state-of-the-art deep learning models on both datasets. The results demonstrate that the DSMT-Net significantly improves the accuracy of pancreatic and breast cancer diagnosis.

Thick Cloud Removal in Multitemporal Remote Sensing Images via Low-Rank Regularized Self-Supervised Network

Thick Cloud Removal in Multitemporal Remote Sensing Images via Low-Rank Regularized Self-Supervised Network

Neighborhood Difference-Based Self-Supervised Network for Detecting Small Changes From Synthetic Aperture Radar Images

Neighborhood Difference-Based Self-Supervised Network for Detecting Small Changes From Synthetic Aperture Radar Images

Self-Supervised Network Distillation for Exploration

Reinforcement learning is currently applicable across a range of domains, including robotics, gaming, and natural language processing. However, the approach faces difficulties in environments with sparse rewards. Random network distillation (RND) is a good intrinsic reward solution to this problem. Nevertheless, the RND method’s effectiveness hinges on excellent initialization, and the reliance on random features somewhat constrains the agent’s exploration capabilities. This paper proposes a self-supervised network distillation (SSND) exploration method, addressing the drawbacks of RND’s reliance on initializing random networks while enhancing the agent’s exploration capability in sparse reward environments. The method uses distillation error as intrinsic rewards, with the target network trained using self-supervised learning. During the training of the predictor network, we noticed fluctuations in both loss values and intrinsic rewards, which have a detrimental impact on the performance of the intelligent agent. To resolve this issue, we introduce batch normalization layers to the target network, which helps mitigate intrinsic reward anomalies stemming from the target network’s instability. Experiments show that the self-supervised network distillation is better than RND in terms of exploration speed and performance.

S[formula omitted]U-PVNet: Arbitrary-scale point cloud upsampling via Point-Voxel Network based on Siamese Self-Supervised Learning

Recently, benefiting from the rapid development of deep learning, many research on point cloud upsampling task has been successfully proposed with remarkable performance. The quality of the upsampled point clouds is also vital for the quality of the reconstructed meshes in 3D surface reconstruction task. However, most existing methods either independently train a specific upsampling network for each scale factor or heavily rely on the paired training data as the supervision information. To address these limitations which are both inefficient and impractical for storage and computation in real applications, we present a Point-Voxel Network based on Siamese Self-supervised learning for arbitrary-Scale point cloud Upsampling (S3U-PVNet), which mainly includes a Down–Up pipeline and an Up–Down pipeline, to support self-supervised point cloud upsampling with arbitrary scale factors by a single network. The core of our network is a series of stacked point-voxel feature fusion (PVFF) modules, which can effectively and efficiently extract and fuse multi-granularity features from input point clouds. Each module includes two branches, a point-based branch that represents the sparse inputs in points to adaptively learn the spatial relationship among points, and a voxel-based branch that extract features in voxels to reduce the problem of inaccurate feature extraction caused by the irregularity and sparsity of point clouds. Furthermore, we also propose an end-to-end training objective for our siamese self-supervised network encapsulating reconstruction loss and similarity loss, which considers both global shape constraint and local geometric constraint. We achieve new state-of-the-art unsupervised upsampling results on various synthetic datasets and demonstrate the generalization of the proposed method on challenging real-world data.

TSSFN: Transformer-based self-supervised fusion network for low-quality fundus image enhancement

Fundus images are used to assist the diagnoses of ocular diseases, and a high-quality fundus image with more details makes clinical diagnostic results more reliable. However, the quality of fundus images is often unsatisfactory due to the turbidity of refractive medium and the doctor-patient cooperation. To enhance the low-quality fundus images, a transformer-based self-supervised network is proposed. During the training phase, an encoder-decoder-based network is introduced. To counteract the drawbacks of establishing long-term dependencies in the convolutional neural network (CNN), the encoder composed of vision transformer and CNN is proposed so that the global and local information of fundus images is fully extracted. On this basis, three reconstruction tasks with self-supervised constraints are designed to enable the network to extract features from different degenerated images. During the testing phase, a low-quality fundus image is decomposed into three feature layers of reverse, illumination, and detail, and then the multi-layer features are fused via the network. To demonstrate the effectiveness of the proposed method, the non-uniform illumination and blurry fundus images are tested. The average scores of NIQE on underexposed, blurred, and overexposed fundus images are 3.03, 2.98, and 2.80, respectively. The average scores of BRISQUE on underexposed, blurred, and overexposed fundus images are 40.32, 40.55, and 39.76, respectively. The average score of subjective evaluation by three ophthalmologists is 61.17%. Compared with the existing methods, the proposed method achieves the superior performance in both subjective and objective evaluations.

An Approach to Recognise Lung Diseases Using Segmentation and Classification

Abstract Lung cancer is one of the most common causes of death in people worldwide. One of the key procedures for early detection of cancer is segmentation or analysis and classification or assessment of lung images. Radiotherapists have to invest a lot of effort into the manual segmentation of medical images. To solve this issue, early-stage lung cancer is detected using Computed Tomography (CT) scan images. The proposed system for diagnosing lung cancer is divided into two main components: the first part is an analyser component built on the upper layer of the U-shaped Network Transformer (UNT), and the second component is an assessment component built on the upper layer of the self-supervised network, which is used to categorise the output segmentation component as benign or cancerous. The proposed method provides a powerful tool for the early detection and treatment of lung cancer by combining CT scan data with 2D input. Numerous experiments are conducted to improve the analysis and evaluation of the findings. Using the public dataset, both test and training experiments were conducted. New state-of-the-art performances were achieved with experimental results: an analyser accuracy of 96.9% and an assessment accuracy of 96.98%. The proposed approach provides a new powerful tool for leveraging 2D-input CT scan data for early detection and treatment of lung cancer using a variety of methods.