Self-supervised Technique Research Articles

Self-supervised learning for small-scale medical imaging dataset

In recent years, advancements in deep learning have dramatically improved performance in medical image analysis, yet these models typically rely on large-scale labeled datasets, which are often unattainable in medical settings due to privacy concerns, limited data availability, and high annotation costs. This study explores the application of self-supervised learning (SSL) techniques to overcome these limitations and effectively utilize small-scale medical imaging datasets. By leveraging SSL, we enable models to learn useful feature representations without requiring extensive labeled data. We investigate various self-supervised approaches, including contrastive learning and masked image modeling, and evaluate their effectiveness on a limited dataset of medical images. Our experiments demonstrate that SSL-based models can achieve competitive performance, even when trained on a fraction of the labeled data typically required for supervised methods. Additionally, we explore the impact of SSL on model robustness and generalization across diverse medical imaging modalities. The findings suggest that self-supervised techniques could reduce dependency on annotated data, paving the way for broader, more scalable applications in medical imaging. This research contributes to the development of efficient, scalable diagnostic tools that can be deployed in data-constrained environments, potentially improving diagnostic accuracy and accessibility in smaller healthcare facilities.

Self-supervised feature learning for acoustic data analysis

Acoustic surveys play a pivotal role in fisheries management. During the surveys, acoustic signals are sent into the water and the strength of the reflection, so-called backscatter, is recorded. The collected data are typically annotated manually, a process that is both labor-intensive and time-consuming, to support acoustic target classification (ATC). The primary objective of this study is to develop an annotation-free deep learning model that extracts acoustic features and improves the representation of acoustic data. For this purpose, we adopt a self-supervised method inspired by the Self DIstillation with NO Labels (DINO) model. Extracting useful acoustic features is an intricate task due to the inherent variability and complexity in biological targets, as well as environmental and technical factors influencing sound interactions. The proposed model is trained with three sampling methods: random sampling, which ignores class imbalance present in the acoustic survey data; class-balanced sampling, which ensures equal representation of known categories; and intensity-based sampling, which selects data to capture backscatter variations. The quality of extracted features is then evaluated and compared. We show that the extracted features lead to improvement, in comparison to using the untreated data, in the discriminative power of several machine learning methods (k-nearest neighbor (kNN), linear regression, multinomial logistic regression) for ATC. The improvement was measured through higher accuracy in kNN (77.55% vs. 71.93%), Macro AUC in logistic regression (0.92 vs. 0.80), and R2 in linear regression (0.69 vs. 0.45) when comparing extracted features to the untreated data. Our findings highlight the advantage of applying emerging self-supervised techniques in fisheries acoustics. This study thus contributes to the ongoing efforts to improve the efficiency of acoustic surveys in fisheries management.

Contrastive Learned Self-Supervised Technique for Fatty Liver and Chronic Liver Identification

PurposeLiver diseases are life-threatening problems when not cured in time. The root of the prevention lies in the early identification of fatty liver diseases, which may be one of the primary causes of most chronic liver diseases. Computer-aided diagnosis (CAD) through deep learning techniques proved to be a success in the medical field for early detections when implementing progressive algorithms. MethodsThis paper proposes a self-supervised Siamese neural network (SNN) for fatty liver identification. SNN is influenced by the model optimization property of unsupervised and manual annotation property of supervised learning. This technique is based on contrastive learning of the joint embedding network, which can learn more subtle representations from the unlabelled medical images for the classification task, with just one or few labelled images required from each class for training. Detailed appearance differences between two images are contrasted during training for learning the representations. It makes use of feedback supervisory signals across the dataset to learn the data structure itself without relying on labelling whole data. ResultsThe efficiency of the proposed model has been validated on liver ultrasound images to classify them into normal, fatty liver grade-I (mild), II (moderate), III (severe), and chronic liver diseases in a two-class classifier (normal/abnormal) and five-class classifier by minimizing contrastive loss to obtain best classification accuracy of 99.90% and 99.77% respectively. ConclusionOverall experimental results achieved from the proposed model proved reliable enough to assist medical specialists in screening of liver diseases.

CLRiuS: Contrastive Learning for intrinsically unordered Steel Scrap

There has been remarkable progress in the field of Deep Learning and Computer Vision, but there is a lack of freely available labeled data, especially when it comes to data for specific industrial applications. However, large volumes of structured, semi-structured and unstructured data are generated in industrial environments, from which meaningful representations can be learned. The effort required for manual labeling is extremely high and can often only be carried out by domain experts. Self-supervised methods have proven their effectiveness in recent years in a wide variety of areas such as natural language processing or computer vision. In contrast to supervised methods, self-supervised techniques are rarely used in real industrial applications. In this paper, we present a self-supervised contrastive learning approach that outperforms existing supervised approaches on the used scrap dataset. We use different types of augmentations to extract the fine-grained structures that are typical for this type of images of intrinsically unordered items. This extracts a wider range of features and encodes more aspects of the input image. This approach makes it possible to learn characteristics from images that are common for applications in the industry, such as quality control. In addition, we show that this self-supervised learning approach can be successfully applied to scene-like images for classification.

Series2vec: similarity-based self-supervised representation learning for time series classification

We argue that time series analysis is fundamentally different in nature to either vision or natural language processing with respect to the forms of meaningful self-supervised learning tasks that can be defined. Motivated by this insight, we introduce a novel approach called Series2Vec for self-supervised representation learning. Unlike the state-of-the-art methods in time series which rely on hand-crafted data augmentation, Series2Vec is trained by predicting the similarity between two series in both temporal and spectral domains through a self-supervised task. By leveraging the similarity prediction task, which has inherent meaning for a wide range of time series analysis tasks, Series2Vec eliminates the need for hand-crafted data augmentation. To further enforce the network to learn similar representations for similar time series, we propose a novel approach that applies order-invariant attention to each representation within the batch during training. Our evaluation of Series2Vec on nine large real-world datasets, along with the UCR/UEA archive, shows enhanced performance compared to current state-of-the-art self-supervised techniques for time series. Additionally, our extensive experiments show that Series2Vec performs comparably with fully supervised training and offers high efficiency in datasets with limited-labeled data. Finally, we show that the fusion of Series2Vec with other representation learning models leads to enhanced performance for time series classification. Code and models are open-source at https://github.com/Navidfoumani/Series2Vec

RDLR: A Robust Deep Learning-Based Image Registration Method for Pediatric Retinal Images.

Retinal diseases stand as a primary cause of childhood blindness. Analyzing the progression of these diseases requires close attention to lesion morphology and spatial information. Standard image registration methods fail to accurately reconstruct pediatric fundus images containing significant distortion and blurring. To address this challenge, we proposed a robust deep learning-based image registration method (RDLR). The method consisted of two modules: registration module (RM) and panoramic view module (PVM). RM effectively integrated global and local feature information and learned prior information related to the orientation of images. PVM was capable of reconstructing spatial information in panoramic images. Furthermore, as the registration model was trained on over 280,000 pediatric fundus images, we introduced a registration annotation automatic generation process coupled with a quality control module to ensure the reliability of training data. We compared the performance of RDLR to the other methods, including conventional registration pipeline (CRP), voxel morph (WM), generalizable image matcher (GIM), and self-supervised techniques (SS). RDLR achieved significantly higher registration accuracy (average Dice score of 0.948) than the other methods (ranging from 0.491 to 0.802). The resulting panoramic retinal maps reconstructed by RDLR also demonstrated substantially higher fidelity (average Dice score of 0.960) compared to the other methods (ranging from 0.720 to 0.783). Overall, the proposed method addressed key challenges in pediatric retinal imaging, providing an effective solution to enhance disease diagnosis. Our source code is available at https://github.com/wuwusky/RobustDeepLeraningRegistration .

MV-MR: Multi-Views and Multi-Representations for Self-Supervised Learning and Knowledge Distillation.

We present a new method of self-supervised learning and knowledge distillation based on multi-views and multi-representations (MV-MR). MV-MR is based on the maximization of dependence between learnable embeddings from augmented and non-augmented views, jointly with the maximization of dependence between learnable embeddings from the augmented view and multiple non-learnable representations from the non-augmented view. We show that the proposed method can be used for efficient self-supervised classification and model-agnostic knowledge distillation. Unlike other self-supervised techniques, our approach does not use any contrastive learning, clustering, or stop gradients. MV-MR is a generic framework allowing the incorporation of constraints on the learnable embeddings via the usage of image multi-representations as regularizers. The proposed method is used for knowledge distillation. MV-MR provides state-of-the-art self-supervised performance on the STL10 and CIFAR20 datasets in a linear evaluation setup. We show that a low-complexity ResNet50 model pretrained using proposed knowledge distillation based on the CLIP ViT model achieves state-of-the-art performance on STL10 and CIFAR100 datasets.

View adaptive unified self-supervised technique for abdominal organ segmentation

Automatic abdominal organ segmentation is an essential prerequisite for accurate volumetric analysis, disease diagnosis, and tracking by medical practitioners. However, the deformable shapes, variable locations, overlapping with nearby organs, and similar contrast make the segmentation challenging. Moreover, the requirement of a large manually labeled dataset makes it harder. Hence, a semi-supervised contrastive learning approach is utilized to perform the automatic abdominal organ segmentation. Existing 3D deep learning models based on contrastive learning are not able to capture the 3D context of medical volumetric data along three planes/views: axial, sagittal, and coronal views. In this work, a semi-supervised view-adaptive unified model (VAU-model) is proposed to make the 3D deep learning model as view-adaptive to learn 3D context along each view in a unified manner. This method utilizes the novel optimization function that assists the 3D model to learn the 3D context of volumetric medical data along each view in a single model. The effectiveness of the proposed approach is validated on the three types of datasets: BTCV, NIH, and MSD quantitatively and qualitatively. The results demonstrate that the VAU model achieves an average Dice score of 81.61% which is a 3.89% improvement compared to the previous best results for pancreas segmentation in multi-organ dataset BTCV. It also achieves an average Dice score of 77.76% and 76.76% for the pancreas under the single organ non-pathological NIH dataset, and pathological MSD dataset.

Improving drug response prediction via integrating gene relationships with deep learning.

Predicting the drug response of cancer cell lines is crucial for advancing personalized cancer treatment, yet remains challenging due to tumor heterogeneity and individual diversity. In this study, we present a deep learning-based framework named Deep neural network Integrating Prior Knowledge (DIPK) (DIPK), which adopts self-supervised techniques to integrate multiple valuable information, including gene interaction relationships, gene expression profiles and molecular topologies, to enhance prediction accuracy and robustness. We demonstrated the superior performance of DIPK compared to existing methods on both known and novel cells and drugs, underscoring the importance of gene interaction relationships in drug response prediction. In addition, DIPK extends its applicability to single-cell RNA sequencing data, showcasing its capability for single-cell-level response prediction and cell identification. Further, we assess the applicability of DIPK on clinical data. DIPK accurately predicted a higher response to paclitaxel in the pathological complete response (pCR) group compared to the residual disease group, affirming the better response of the pCR group to the chemotherapy compound. We believe that the integration of DIPK into clinical decision-making processes has the potential to enhance individualized treatment strategies for cancer patients.

Energy-Efficient Self-Supervised Technique to Identify Abnormal User Over 5G Network for E-Commerce

Energy-Efficient Self-Supervised Technique to Identify Abnormal User Over 5G Network for E-Commerce

InverseTime: A Self-Supervised Technique for Semi-Supervised Classification of Time Series

InverseTime: A Self-Supervised Technique for Semi-Supervised Classification of Time Series

Self-Supervised Cloud Classification

Abstract Low-level marine clouds play a pivotal role in Earth’s weather and climate through their interactions with radiation, heat and moisture transport, and the hydrological cycle. These interactions depend on a range of dynamical and microphysical processes that result in a broad diversity of cloud types and spatial structures, and a comprehensive understanding of cloud morphology is critical for continued improvement of our atmospheric modeling and prediction capabilities moving forward. Deep learning has recently accelerated our ability to study clouds using satellite remote sensing, and machine learning classifiers have enabled detailed studies of cloud morphology. A major limitation of deep learning approaches to this problem, however, is the large number of hand-labeled samples that are required for training. This work applies a recently developed self-supervised learning scheme to train a deep convolutional neural network (CNN) to map marine cloud imagery to vector embeddings that capture information about mesoscale cloud morphology and can be used for satellite image classification. The model is evaluated against existing cloud classification datasets and several use cases are demonstrated, including training cloud classifiers with very few labeled samples, interrogation of the CNN’s learned internal feature representations, cross-instrument application, and resilience against sensor calibration drift and changing scene brightness. The self-supervised approach learns meaningful internal representations of cloud structures and achieves comparable classification accuracy to supervised deep learning methods without the expense of creating large hand-annotated training datasets. Significance Statement Marine clouds heavily influence Earth’s weather and climate, and improved understanding of marine clouds is required to improve our atmospheric modeling capabilities and physical understanding of the atmosphere. Recently, deep learning has emerged as a powerful research tool that can be used to identify and study specific marine cloud types in the vast number of images collected by Earth-observing satellites. While powerful, these approaches require hand-labeling of training data, which is prohibitively time intensive. This study evaluates a recently developed self-supervised deep learning method that does not require human-labeled training data for processing images of clouds. We show that the trained algorithm performs competitively with algorithms trained on hand-labeled data for image classification tasks. We also discuss potential downstream uses and demonstrate some exciting features of the approach including application to multiple satellite instruments, resilience against changing image brightness, and its learned internal representations of cloud types. The self-supervised technique removes one of the major hurdles for applying deep learning to very large atmospheric datasets.

Memorization Weights for Instance Reweighting in Adversarial Training

Adversarial training is an effective way to defend deep neural networks (DNN) against adversarial examples. However, there are atypical samples that are rare and hard to learn, or even hurt DNNs' generalization performance on test data. In this paper, we propose a novel algorithm to reweight the training samples based on self-supervised techniques to mitigate the negative effects of the atypical samples. Specifically, a memory bank is built to record the popular samples as prototypes and calculate the memorization weight for each sample, evaluating the "typicalness" of a sample. All the training samples are reweigthed based on the proposed memorization weights to reduce the negative effects of atypical samples. Experimental results show the proposed method is flexible to boost state-of-the-art adversarial training methods, improving both robustness and standard accuracy of DNNs.

Self-supervised next view prediction for limited-angle optical projection tomography.

Optical projection tomography captures 2-D projections of rotating biological samples and computationally reconstructs 3-D structures from these projections, where hundreds of views with an angular range of π radian is desired for a reliable reconstruction. Limited-angle tomography tries to recover the structures of the sample using fewer angles of projections. However, the result is far from satisfactory due to the missing of wedge information. Here we introduce a novel view prediction technique, which is able to extending the angular range of captured views for the limited-angle tomography. Following a self-supervised technique that learns the relationship between the captured limited-angle views, unseen views can be computationally synthesized without any prior label data required. Combined with an optical tomography system, the proposed approach can robustly generate new projections of unknown biological samples and extends the angles of the projections from the original 60° to nearly 180°, thereby yielding high-quality 3-D reconstructions of samples even with highly incomplete measurement.

Self-Supervised Approach for Facial Movement Based Optical Flow

Computing optical flow is a fundamental problem in computer vision. However, deep learning-based optical flow techniques do not perform well for non-rigid movements such as those found in faces, primarily due to lack of the training data representing the fine facial motion. We hypothesize that learning optical flow on face motion data will improve the quality of predicted flow on faces. This work aims to: (1) exploring self-supervised techniques to generate optical flow ground truth for face images; (2) computing baseline results on the effects of using face data to train Convolutional Neural Networks (CNN) for predicting optical flow; and (3) using the learned optical flow in micro-expression recognition to demonstrate its effectiveness. We generate optical flow ground truth using facial key-points in the BP4D-Spontaneous dataset. This optical flow is used to train the FlowNetS architecture to test its performance on the Extended Cohn-Kanade dataset and a portion of the generated dataset. The performance of FlowNetS trained on face images surpassed that of other optical flow CNN architectures. Our optical flow features are further compared with other methods using the STSTNet micro-expression classifier, and the results indicate that the optical flow obtained using this work has promising applications in facial expression analysis.

COVID-19-Associated Lung Lesion Detection by Annotating Medical Image with Semi Self-Supervised Technique

Diagnosing COVID-19 infection through the classification of chest images using machine learning techniques faces many controversial problems owing to the intrinsic nature of medical image data and classification architectures. The detection of lesions caused by COVID-19 in the human lung with properties such as location, size, and distribution is more practical and meaningful to medical workers for severity assessment, progress monitoring, and treatment, thus improving patients’ recovery. We proposed a COVID-19-associated lung lesion detector based on an object detection architecture. It correctly learns disease-relevant features by focusing on lung lesion annotation data of medical images. An annotated COVID-19 image dataset is currently nonexistent. We designed our semi-self-supervised method, which can extract knowledge from available annotated pneumonia image data and guide a novice in annotating lesions on COVID-19 images in the absence of a medical specialist. We prepared a sufficient dataset with nearly 8000 lung lesion annotations to train our deep learning model. We comprehensively evaluated our model on a test dataset with nearly 1500 annotations. The results demonstrated that the COVID-19 images annotated by our method significantly enhanced the model’s accuracy by as much as 1.68 times, and our model competes with commercialized solutions. Finally, all experimental data from multiple sources with different annotation data formats are standardized into a unified COCO format and publicly available to the research community to accelerate research on the detection of COVID-19 using deep learning.

CONTRASTIVE SELF-SUPERVISED DATA FUSION FOR SATELLITE IMAGERY

Abstract. Self-supervised learning has great potential for the remote sensing domain, where unlabelled observations are abundant, but labels are hard to obtain. This work leverages unlabelled multi-modal remote sensing data for augmentation-free contrastive self-supervised learning. Deep neural network models are trained to maximize the similarity of latent representations obtained with different sensing techniques from the same location, while distinguishing them from other locations. We showcase this idea with two self-supervised data fusion methods and compare against standard supervised and self-supervised learning approaches on a land-cover classification task. Our results show that contrastive data fusion is a powerful self-supervised technique to train image encoders that are capable of producing meaningful representations: Simple linear probing performs on par with fully supervised approaches and fine-tuning with as little as 10% of the labelled data results in higher accuracy than supervised training on the entire dataset.

RL-SSI Model: Adapting a Supervised Learning Approach to a Semi-Supervised Approach for Human Action Recognition

Generally, the action recognition task requires a vast amount of labeled data, which represents a time-consuming human annotation effort. To mitigate the dependency on labeled data, this study proposes Semi-Supervised and Iterative Reinforcement Learning (RL-SSI), which adapts a supervised approach that uses 100% labeled data to a semi-supervised and iterative approach using reinforcement learning for human action recognition in videos. The JIGSAWS and Breakfast datasets were used to evaluate the RL-SSI model, because they are commonly used in the action segmentation task. The same applies to the performance metrics used in this work-F-Score (F1) and Edit Score-which are commonly applied for such tasks. In JIGSAWS tests, we observed that the RL-SSI outperformed previously developed state-of-the-art techniques in all quantitative measures, while using only 65% of the labeled data. When analysing the Breakfast tests, we compared the effectiveness of RL-SSI with the results of the self-supervised technique called SSTDA. We have found that RL-SSI outperformed SSTDA with an accuracy of 66.44% versus 65.8%, but RL-SSI was surpassed by the F1@10 segmentation measure, which presented an accuracy of 67.33% versus 69.3% for SSTDA. Despite this, our experiment only used 55.8% of the labeled data, while SSTDA used 65%. We conclude that our approach outperformed equivalent supervised learning methods and is comparable to SSTDA, when evaluated on multiple datasets of human action recognition, proving to be an important innovative method to successfully building solutions to reduce the amount of fully labeled data, leveraging the work of human specialists in the task of data labeling of videos, and their respectives frames, for human action recognition, thus reducing the required resources to accomplish it.

Self-supervised learning for medieval handwriting identification: A case study from the Vatican Apostolic Library

In this paper, we consider the task of automatically identifying whether different parts of medieval and modern manuscripts can be traced back to the same copyist/scribe, a problem of significant interest in paleography. Currently, the application of deep learning techniques in the context of scribe recognition has been hindered by the lack of a sufficiently large, labeled dataset, since the labeling process is incredibly complex and time-consuming. Here, we propose the first successful application of the recent framework of self-supervised learning to the field of digital paleography, wherein we pretrain a convolutional neural network by leveraging large amounts of unlabeled manuscripts. To this end, we build a novel dataset consisting of both labeled and unlabeled manuscripts for copyist identification extracted from the Vatican Apostolic Library. We show that fine-tuning this model to the task of interest significantly outperforms other baselines, including the common setup of initializing the network from general-domain features, or training the model from scratch, also in terms of generalization power. Overall, our results reveal the strong potential of self-supervised techniques in the field of digital paleography, where unlabeled data (i.e., digitized manuscripts) is nowadays available, while labeled data is scarcer.

Deep unsupervised methods towards behavior analysis in ubiquitous sensor data

Behavioral analysis (BA) on ubiquitous sensor data is the task of finding the latent distribution of features for modeling user-specific characteristics. These characteristics, in turn, can be used for a number of tasks including resource management, power efficiency, and smart home applications. In recent years, the employment of topic models for BA has been found to successfully extract the dynamics of the sensed data. Topic modeling is popularly performed on text data for mining inherent topics. The task of finding the latent topics in textual data is done in an unsupervised manner. In this work we propose a novel clustering technique for BA which can find hidden routines in ubiquitous data and also captures the pattern in the routines. Our approach efficiently works on high dimensional data for BA without performing any computationally expensive reduction operations. We evaluate three different techniques namely Latent Dirichlet Allocation (LDA), the Non-negative Matrix Factorization (NMF), and the Probabilistic Latent Semantic Analysis (PLSA) for comparative study. We have analyzed the efficiency of the methods by using performance indices like perplexity and silhouette on three real-world ubiquitous sensor datasets namely, the Intel Lab, Kyoto, and MERL. Through rigorous experiments, we achieve silhouette scores of 0.7049 over the Intel Lab dataset, 0.6547 over the Kyoto dataset, and 0.8312 over the MERL dataset for clustering. In these cases, however, it is di cult to validate the results obtained as the datasets do not contain any ground truth information. Towards that, we investigate a self-supervised method that will be capable of capturing the inherent ground truths that are available in the dataset. We design a self-supervised technique which we apply on datasets containing ground truth and also without. We see that our performance on data without ground truth differs from that with ground truth by approximately 8% (F-score) hence showing the efficacy of self-supervised techniques towards capturing ground truth information.