Self-supervised Method Research Articles

Study on Various Self-supervised Video Representation Learning Methods

In the field of video representation, the self-supervised learning method was efficiently applied to the pre-training domain to downstream tasks using large and many unlabeled datasets. The basic approaches are typically based on a pre-text task method and a contrastive learning method. First, in a pre-text task method, a user defines a new problem and uses it as a proxy for self-supervised learning. Second, contrastive learning is a method of predicting the relationship between instances by assuming that feature values extracted through a certain model will have similar information between instances. According to the recent popularity of unsupervised learning, various self-supervised methods as well as the above methods are used in the field of video representation learning. Effective video representation learning is performed by fusing the multimodality advantages of video and the features of audio-visual information with various deep learning techniques. In this paper, recent representative methods of self-supervised video representation learning are summarized and described. Additionally, we provide a brief overview of how to utilize multimodality (audio-visual) information, which is the strength of the video.

Self-supervised graph representation learning via positive mining

Inspired by the recent success of self-supervised methods applied on images, self-supervised learning on graph structured data has seen rapid growth especially centered on augmentation-based contrastive methods. However, these methods pay little attention to the inherent distinction between images and graphs: while augmentation is well defined on images, it may behave arbitrarily on graphs. That is, without carefully designed augmentation techniques, augmentations on graphs may behave arbitrarily in that the underlying semantics of graphs can drastically change. As a consequence, the performance of existing augmentation-based methods is highly dependent on the choice of augmentation scheme, i.e., hyperparameters associated with augmentations. In this paper, we propose a novel augmentation-free self-supervised learning method for graphs, named AFGRL, which learns graph representations without using any arbitrary augmentation scheme. Specifically, we generate an alternative view of a graph by discovering nodes that share the local structural information and the global semantics within the graph. Extensive experiments toward various node-level tasks (i.e., node classification, clustering, and similarity search) and graph-level tasks (i.e. graph classification) on various real-world datasets demonstrate the superiority of AFGRL.

Self-Supervised Learning for Scene Classification in Remote Sensing: Current State of the Art and Perspectives

Deep learning methods have become an integral part of computer vision and machine learning research by providing significant improvement performed in many tasks such as classification, regression, and detection. These gains have been also observed in the field of remote sensing for Earth observation where most of the state-of-the-art results are now achieved by deep neural networks. However, one downside of these methods is the need for large amounts of annotated data, requiring lots of labor-intensive and expensive human efforts, in particular for specific domains that require expert knowledge such as medical imaging or remote sensing. In order to limit the requirement on data annotations, several self-supervised representation learning methods have been proposed to learn unsupervised image representations that can consequently serve for downstream tasks such as image classification, object detection or semantic segmentation. As a result, self-supervised learning approaches have been considerably adopted in the remote sensing domain within the last few years. In this article, we review the underlying principles developed by various self-supervised methods with a focus on scene classification task. We highlight the main contributions and analyze the experiments, as well as summarize the key conclusions, from each study. We then conduct extensive experiments on two public scene classification datasets to benchmark and evaluate different self-supervised models. Based on comparative results, we investigate the impact of individual augmentations when applied to remote sensing data as well as the use of self-supervised pre-training to boost the classification performance with limited number of labeled samples. We finally underline the current trends and challenges, as well as perspectives of self-supervised scene classification.

Graph Barlow Twins: A self-supervised representation learning framework for graphs

The self-supervised learning (SSL) paradigm is an essential exploration area, which tries to eliminate the need for expensive data labeling. Despite the great success of SSL methods in computer vision and natural language processing, most of them employ contrastive learning objectives that require negative samples, which are hard to define. This becomes even more challenging in the case of graphs and is a bottleneck for achieving robust representations. To overcome such limitations, we propose a framework for self-supervised graph representation learning — Graph Barlow Twins, which utilizes a cross-correlation-based loss function instead of negative samples. Moreover, it does not rely on non-symmetric neural network architectures — in contrast to state-of-the-art self-supervised graph representation learning method BGRL. We show that our method achieves as competitive results as the best self-supervised methods and fully supervised ones while requiring fewer hyperparameters and substantially shorter computation time (ca. 30 times faster than BGRL).

Simultaneous label inference and discriminant projection estimation through adaptive self-taught graphs

Processing structured data has become an interesting topic in recent years. The development of graph-based semi-supervised learning models has attracted much attention from machine learning researchers. In this paper, we present a novel approach for graph-based semi-supervised learning. We provide an effective method for simultaneous label recovery and linear transformation estimation. The targeted linear transformation is to obtain a discriminant subspace. The most important factor in this work to improve the semi-supervised learning is to exploit the data structure and soft labels of the available unlabeled samples. In the iterative optimization scheme used, the prior estimation of the labels increases the monitoring information in an indirect way through an introduced label-graph, avoiding the use of confidence-based hard decisions as used in self-supervised methods. It also enforces label smoothing and projected data smoothing through the use of hybrid graphs. For each smoothing type, the hybrid graph is an adaptive fusion of the two graphs encoding the similarity of the data information and the similarity of the label information. The proposed method leads to an improved discriminant linear transformation. Several experimental results on real image datasets confirm the effectiveness of the proposed method. This work also shows superior performance compared to semi-supervised methods that use simultaneous embedding and inference of labels.

Self-supervised learning in medicine and healthcare.

The development of medical applications of machine learning has required manual annotation of data, often by medical experts. Yet, the availability of large-scale unannotated data provides opportunities for the development of better machine-learning models. In this Review, we highlight self-supervised methods and models for use in medicine and healthcare, and discuss the advantages and limitations of their application to tasks involving electronic health records and datasets of medical images, bioelectrical signals, and sequences and structures of genes and proteins. We also discuss promising applications of self-supervised learning for the development of models leveraging multimodal datasets, and the challenges in collecting unbiased data for their training. Self-supervised learning may accelerate the development of medical artificial intelligence.

Self-Supervised Learning for Time-Series Anomaly Detection in Industrial Internet of Things

Industrial sensors have presently emerged as a very important device for monitoring environmental conditions in the manufacturing system. However, abnormal behavior of these smart sensors may cause some failure or potential risk during system operation, thereby increasing the high availability of the entire manufacturing process. An anomaly detection tool in industrial monitoring system must detect any abnormal behavior in advance. Recently, self-supervised learning demonstrated comparable performance with other methods while eliminating manually labeled processes in training. Moreover, this technique decreases the complexity of the training model in lightweight devices to increase the processing time and detect accurately the health of equipment assets. Therefore, this paper proposes an anomaly detection method using a self-supervised learning framework in a time-series dataset to improve the model performance in terms of high accuracy and lightweight method. With the consideration of time-series data augmentation for generating pseudo-label, a classifier using one-dimension convolutional neural network (1DCNN) is applied to learn the characteristics of normal data. This classification model output will effectively measure the degree of abnormality. The experimental results indicate that our proposed method outperforms classic anomaly detection methods. Furthermore, the model deployment in a real testbed is performed to illustrate the efficiency of the self-supervised learning method for time-series anomaly detection.

Self-Supervised Representation Learning Framework for Remote Physiological Measurement Using Spatiotemporal Augmentation Loss

Recent advances in supervised deep learning methods are enabling remote measurements of photoplethysmography-based physiological signals using facial videos. The performance of these supervised methods, however, are dependent on the availability of large labelled data. Contrastive learning as a self-supervised method has recently achieved state-of-the-art performances in learning representative data features by maximising mutual information between different augmented views. However, existing data augmentation techniques for contrastive learning are not designed to learn physiological signals from videos and often fail when there are complicated noise and subtle and periodic colour/shape variations between video frames. To address these problems, we present a novel self-supervised spatiotemporal learning framework for remote physiological signal representation learning, where there is a lack of labelled training data. Firstly, we propose a landmark-based spatial augmentation that splits the face into several informative parts based on the Shafer’s dichromatic reflection model to characterise subtle skin colour fluctuations. We also formulate a sparsity-based temporal augmentation exploiting Nyquist–Shannon sampling theorem to effectively capture periodic temporal changes by modelling physiological signal features. Furthermore, we introduce a constrained spatiotemporal loss which generates pseudo-labels for augmented video clips. It is used to regulate the training process and handle complicated noise. We evaluated our framework on 3 public datasets and demonstrated superior performances than other self-supervised methods and achieved competitive accuracy compared to the state-of-the-art supervised methods. Code is available at https://github.com/Dylan-H-Wang/SLF-RPM.

Can Semantic Labels Assist Self-Supervised Visual Representation Learning?

Recently, contrastive learning has largely advanced the progress of unsupervised visual representation learning. Pre-trained on ImageNet, some self-supervised algorithms reported higher transfer learning performance compared to fully-supervised methods, seeming to deliver the message that human labels hardly contribute to learning transferrable visual features. In this paper, we defend the usefulness of semantic labels but point out that fully-supervised and self-supervised methods are pursuing different kinds of features. To alleviate this issue, we present a new algorithm named Supervised Contrastive Adjustment in Neighborhood (SCAN) that maximally prevents the semantic guidance from damaging the appearance feature embedding. In a series of downstream tasks, SCAN achieves superior performance compared to previous fully-supervised and self-supervised methods, and sometimes the gain is significant. More importantly, our study reveals that semantic labels are useful in assisting self-supervised methods, opening a new direction for the community.

Task-Level Self-Supervision for Cross-Domain Few-Shot Learning

Learning with limited labeled data is a long-standing problem. Among various solutions, episodic training progres-sively classifies a series of few-shot tasks and thereby is as-sumed to be beneficial for improving the model’s generalization ability. However, recent studies show that it is eveninferior to the baseline model when facing domain shift between base and novel classes. To tackle this problem, we pro-pose a domain-independent task-level self-supervised (TL-SS) method for cross-domain few-shot learning.TL-SS strategy promotes the general idea of label-based instance-levelsupervision to task-level self-supervision by augmenting mul-tiple views of tasks. Two regularizations on task consistencyand correlation metric are introduced to remarkably stabi-lize the training process and endow the generalization ability into the prediction model. We also propose a high-order associated encoder (HAE) being adaptive to various tasks.By utilizing 3D convolution module, HAE is able to generate proper parameters and enables the encoder to flexibly toany unseen tasks. Two modules complement each other andshow great promotion against state-of-the-art methods experimentally. Finally, we design a generalized task-agnostic test,where our intriguing findings highlight the need to re-think the generalization ability of existing few-shot approaches.

Augmentation-Free Self-Supervised Learning on Graphs

Inspired by the recent success of self-supervised methods applied on images, self-supervised learning on graph structured data has seen rapid growth especially centered on augmentation-based contrastive methods. However, we argue that without carefully designed augmentation techniques, augmentations on graphs may behave arbitrarily in that the underlying semantics of graphs can drastically change. As a consequence, the performance of existing augmentation-based methods is highly dependent on the choice of augmentation scheme, i.e., augmentation hyperparameters and combinations of augmentation. In this paper, we propose a novel augmentation-free self-supervised learning framework for graphs, named AFGRL. Specifically, we generate an alternative view of a graph by discovering nodes that share the local structural information and the global semantics with the graph. Extensive experiments towards various node-level tasks, i.e., node classification, clustering, and similarity search on various real-world datasets demonstrate the superiority of AFGRL. The source code for AFGRL is available at https://github.com/Namkyeong/AFGRL.

Barely-Supervised Learning: Semi-supervised Learning with Very Few Labeled Images

This paper tackles the problem of semi-supervised learning when the set of labeled samples is limited to a small number of images per class, typically less than 10, problem that we refer to as barely-supervised learning. We analyze in depth the behavior of a state-of-the-art semi-supervised method, FixMatch, which relies on a weakly-augmented version of an image to obtain supervision signal for a more strongly-augmented version. We show that it frequently fails in barely-supervised scenarios, due to a lack of training signal when no pseudo-label can be predicted with high confidence. We propose a method to leverage self-supervised methods that provides training signal in the absence of confident pseudo-labels. We then propose two methods to refine the pseudo-label selection process which lead to further improvements.The first one relies on a per-sample history of the model predictions, akin to a voting scheme. The second iteratively up-dates class-dependent confidence thresholds to better explore classes that are under-represented in the pseudo-labels. Our experiments show that our approach performs significantly better on STL-10 in the barely-supervised regime,e.g. with 4 or 8 labeled images per class.

A new self-supervised task on graphs: Geodesic distance prediction

The heavy dependence on human-curated labels makes supervised algorithms hit a bottleneck in real applications. Fortunately, a more data-efficient learning paradigm—self-supervised learning (SSL) disrupts this dilemma. By creating pretext tasks, self-supervised algorithms learn from the data itself without external supervision. Until recently, SSL has yielded immense success in the image domain, but its research in graph mining has received relatively little scrutiny. In fact, rich topological links and affiliated attributes promise more potential for self-supervised graph learning. Thus, in this paper, we propose a new pretext task called geodesic distance prediction to guide node representation learning. Specifically, this task demands neural nets to learn to infer the geodesic distance of one node relative to the other. Our underlying hypothesis is that doing well on reasoning about the pairwise distance requires the model to extract relation rules between topological distances and attributes. In this way, the complex correlation between nodes can be measured with a simple distance scale. Experiments demonstrate that our S2GRL achieves competitive or better performance than many state-of-the-art self-supervised methods. A case study on distance inference shows that S2GRL can effectively evaluate the risk of business transactions or make recommendations from a topological view.

Abstract 5047: Self-supervised deep learning to assess breast cancer risk

Abstract Background: Personalized breast cancer (BC) screening adjusts the imaging modality and frequency of exams according to a woman's risk of developing BC. This can lower cost and false positives by reducing unnecessary exams and has the potential to find more cancers at a curable stage. Deep learning (DL) is a class of artificial intelligence algorithms that progressively extracts higher-level representations from raw input. A critical challenge to applying DL for BC risk prediction is that images are needed from exams performed before a possible cancer diagnosis. Large longitudinal datasets with cancer labeling are relatively scarce. Recently, new self-supervised methods have been developed which do not require labeling. Instead, they learn to recognize higher-level features by comparing two augmented images and determining if they are derived from the same original image. Methods: We developed Self-supervised AI for CAncer Risk Assessment (SAICARA), a mammography-based DL model to predict BC risk. We trained SAICARA on mammograms from the Chicago Multiethnic Epidemiologic Cohort (ChiMEC). We used the momentum contrast method in pretraining to train an encoder that produces compact representations of input mammography views. We initialized the encoders with weights obtained from training on the ImageNet dataset. We continued pretraining with 223,415 chest radiographs from the CheXpert database. Finally, we used mammograms from ChiMEC without any requirements on the exam date. We used augmentations from two different mammography views to provide better positive pairs for self-supervised learning. For fine-tuning, we trained with exams from women who were known to be cancer-free with at least 100 days of follow-up, and patients diagnosed with BC at least 30 days following the exam. Optimization was performed using a negative-log likelihood loss function which was discretized by considering quantiles of the event-time distribution. Hyperparameters were tuned using a Bayesian optimization strategy implemented by Weights and Biases. We computed the concordance index and the area under the receiver-operating characteristic curve (AUC) at two years to evaluate the discriminating capacity of the predicted risk of BC. We evaluated our model using 10-fold cross-validation. Results: In the final phase of pretraining, we used 13,194 mammography exams from 2,835 women. For fine-tuning, we used 4,849 exams from 1,418 women who were known to be cancer-free at their last follow-up, and 1,760 exams from 744 women who had exams that were followed by a BC diagnosis. SAICARA achieved a mean concordance index of 0.62 (standard deviation, SD = 0.11) and a mean AUC of 0.61 (SD = 0.09). Conclusion: Self-supervised DL holds promise as a technique for improving the performance of image-based BC risk prediction models. Citation Format: Anna Woodard, Olasubomi J. Omoleye, Rachna Gupta, Fangyuan Zhao, Aarthi Koripelly, Ian Foster, Kyle Chard, Toshio F. Yoshimatsu, Yonglan Zheng, Dezheng Huo, Olufunmilayo I. Olopade. Self-supervised deep learning to assess breast cancer risk [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2022; 2022 Apr 8-13. Philadelphia (PA): AACR; Cancer Res 2022;82(12_Suppl):Abstract nr 5047.

Abstract 5060: A deep learning based pancreatic adenocarcinoma survival prediction model applicable to adenocarcinoma of other organs

Abstract Introduction: To analyze how the histomorphological features of adenocarcinoma lesions are related to the survival of patients, we developed a deep learning based pancreatic adenocarcinoma survival model that predicts death risk through the general adenocarcinoma feature extractor (GAFE). Our pancreatic adenocarcinoma survival model and GAFE was trained with hematoxylin and eosin (H&E) stained whole slide images (WSIs). Also, to measure the generalization performance, we predict the death risk of rectum adenocarcinoma and breast adenocarcinoma using the pancreatic adenocarcinoma survival model. Design: The whole data analyzed in this study was from The Cancer Genome Atlas (TCGA). We used a self-supervised contrastive learning method to train GAFE with randomly chosen 328 adenocarcinoma H&E stained WSIs. These WSIs are consisted of 92 lung (TCGA-LUAD), 100 colon (TCGA-COAD), 97 prostate (TCGA-PRAD) and 39 stomach (TCGA-STAD) adenocarcinoma WSIs. For the pancreatic adenocarcinoma survival model, we used 179 WSIs of TCGA pancreatic adenocarcinoma (TCGA-PAAD) dataset labeled with survival events and periods. There existed 66 uncensored data, and we used 5-fold cross validation. Each WSI was divided into 256*256 pixel-wise patches. Feature vectors were extracted from the patches using the GAFE. Then the attention-based multi-instance learning was applied to train the survival model. The trained model returned a death risk for each patient. We used 122 adenocarcinoma H&E stained WSIs of rectum (TCGA-READ) and 999 of breast (TCGA-BRCA) from TCGA to test the generalization performance for different organs. Result: We used 5-fold cross validation for testing the pancreatic adenocarcinoma survival model. The mean value of C-index from the TCGA-PAAD test datasets was 0.7258 where the highest and the lowest C-index was 0.7784 and 0.6598. Using the model with median performance (C-index: 0.7216), we measured death risk and divided patients into 50% of high-risk and 50% of low-risk groups. From this, the p-value was 0.01986 for the log-rank test.To analyze the performance of the generalization, we ensembled 5 models from the above. We tested our ensembled model on TCGA-READ and TCGA-BRCA. The C-index for TCGA-Read was 0.6941 where the C-index for TCGA-BRCA was 0.5711. Conclusion: We found that adenocarcinoma’s histomorphological features had some correlation with pancreatic adenocarcinoma survival prediction. Moreover, we could see the possibility of transferability of adenocarcinoma histomorphological knowledge and survival analysis to other organs such as rectum adenocarcinoma and breast adenocarcinoma. Citation Format: Joonho Lee, Geongyu Lee, Tae-Yeong Kwak, Sun Woo Kim, Hyeyoon Chang. A deep learning based pancreatic adenocarcinoma survival prediction model applicable to adenocarcinoma of other organs [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2022; 2022 Apr 8-13. Philadelphia (PA): AACR; Cancer Res 2022;82(12_Suppl):Abstract nr 5060.

A Masked Self-Supervised Pretraining Method for Face Parsing

Face Parsing aims to partition the face into different semantic parts, which can be applied into many downstream tasks, e.g., face mask up, face swapping, and face animation. With the popularity of cameras, it is easier to acquire facial images. However, pixel-wise manually labeling is time-consuming and labor-intensive, which motivates us to explore the unlabeled data. In this paper, we present a self-supervised learning method attempting to make full use of the unlabeled facial images for face parsing. In particular, we randomly mask some patches in the central area of facial images, and the model is required to reconstruct the masked patches. This self-supervised pretraining is capable of making the model capture facial feature representations through these unlabeled data. After self-supervised pretraining, the model is fine-tuned on a few labeled data for the face parsing task. Experimental results show that the model achieves better performance for face parsing assisted by the self-supervised pretraining, which greatly decreases the labeling cost. Our approach achieves 74.41 mIoU on the LaPa test set fine-tuned on only 0.2% of the labeled data of the whole training data, surpassing the model that is directly trained by a large margin of +5.02 mIoU. In addition, our approach achieves a new state-of-the-art on the LaPa and CelebAMask-HQ test set.

Comparison of Supervised and Self-Supervised Deep Representations Trained on Histological Images.

Self-supervised methods gain more and more attention, especially in the medical domain, where the number of labeled data is limited. They provide results on par or superior to their fully supervised competitors, yet the difference between information coded by both methods is unclear. This work introduces a novel comparison framework for explaining differences between supervised and self-supervised models using visual characteristics important to the human perceptual system. We apply this framework to models trained for Gleason score and conclude that self-supervised methods are more biased toward contrast and texture transformation than their supervised counterparts. At the same time, supervised methods code more information about the shape.

Self-Supervised Clustering for Leaf Disease Identification

Plant diseases have been one of the most threatening scenarios to farmers. Although most plant diseases can be identified by observing leaves, it often requires human expertise. The recent improvements in computer vision have led to introduce disease classification systems through observing leaf images. Nevertheless, most disease classification systems are specific to diseases and plants, limiting method’s usability. The methods are also costly as they require vast labeled data, which can only be done by experts. This paper introduces a self-supervised leaf disease clustering system that can be used for classifying plant diseases. As self-supervision does not require labeled data, the proposed method can be inexpensive and can be implemented for most types of plants. The method implements a siamese deep convolutional neural network (DCNN) for generating clusterable embeddings from leaf images. The training strategy of the embedding network is conducted using AutoEmbedder approach with randomly augmented image pairs. The self-supervised embedding model training involves three different data pair linkage scenarios: can-link, cannot-link, and may-link pairs. The embeddings are further clustered using k-means algorithm in the final classification stage. The experiment is conducted to individually classify diseases of eight different fruit leaves. The results indicate that the proposed leaf disease identification method performs better than the existing self-supervised clustering systems. The paper indicates that end-to-end siamese networks can outperform well-designed sequentially trained self-supervised methods.

Self-supervised method for 3D human pose estimation with consistent shape and viewpoint factorization

Self-supervised method for 3D human pose estimation with consistent shape and viewpoint factorization

Self-Supervised Keypoint Discovery in Behavioral Videos.

We propose a method for learning the posture and structure of agents from unlabelled behavioral videos. Starting from the observation that behaving agents are generally the main sources of movement in behavioral videos, our method, Behavioral Keypoint Discovery (B-KinD), uses an encoder-decoder architecture with a geometric bottleneck to reconstruct the spatiotemporal difference between video frames. By focusing only on regions of movement, our approach works directly on input videos without requiring manual annotations. Experiments on a variety of agent types (mouse, fly, human, jellyfish, and trees) demonstrate the generality of our approach and reveal that our discovered keypoints represent semantically meaningful body parts, which achieve state-of-the-art performance on keypoint regression among self-supervised methods. Additionally, B-KinD achieve comparable performance to supervised keypoints on downstream tasks, such as behavior classification, suggesting that our method can dramatically reduce model training costs vis-a-vis supervised methods.