Weakly-supervised Learning Research Articles

A Survey of Label-Efficient Deep Learning for 3D Point Clouds.

In the past decade, deep neural networks have achieved significant progress in point cloud learning. However, collecting large-scale precisely-annotated point clouds is extremely laborious and expensive, which hinders the scalability of existing point cloud datasets and poses a bottleneck for efficient exploration of point cloud data in various tasks and applications. Label-efficient learning offers a promising solution by enabling effective deep network training with much-reduced annotation efforts. This paper presents the first comprehensive survey of label-efficient learning of point clouds. We address three critical questions in this emerging research field: i) the importance and urgency of label-efficient learning in point cloud processing, ii) the subfields it encompasses, and iii) the progress achieved in this area. To this end, we propose a taxonomy that organizes label-efficient learning methods based on the data prerequisites provided by different types of labels. We categorize four typical label-efficient learning approaches that significantly reduce point cloud annotation efforts: data augmentation, domain transfer learning, weakly-supervised learning, and pretrained foundation models. For each approach, we outline the problem setup and provide an extensive literature review that showcases relevant progress and challenges. Finally, we share our views on the current research challenges and potential future directions.

HTD-TS3: Weakly Supervised Hyperspectral Target Detection Based on Transformer via Spectral-Spatial Similarity.

As an advanced technique in remote sensing, hyperspectral target detection (HTD) is widely concerned in civilian and military applications. However, the limitation of prior and heterogeneous backgrounds makes HTD models sensitive to data corruption under various interference from the environment. In this article, a novel united HTD framework based on the concept of transformer is proposed to extract [HTD based on transformer via spectral-spatial similarity (HTD-TS3)] under weak supervision, which opens up more flexible ways to study HTD. For the first time, the transformer mechanism is introduced into the HTD task to extract spectral and spatial features in a unified optimization procedure. By modeling long-range dependence among spectra, it realizes spectral-spatial joint inference based on long-range context, which addresses the issues of insufficient utilization of spatial information. To provide samples for weakly supervised learning (WSL), the coarse sample selection and spectral sequence construction in an efficient way are proposed, which makes full use of limited prior information. Finally, an exponential constrained nonlinear function is adopted to acquire pixel-level prediction via combining discriminative spectral-spatial features and coarse spatial information. Experiments on real hyperspectral images (HSIs) captured by different sensors at various scenes verify the effectiveness and efficiency of HTD-TS3.

Weakly-supervised classification of Mohs surgical sections using artificial intelligence

Basal cell carcinoma (BCC) is the most frequently diagnosed form of skin cancer, and its incidence continues to rise, particularly among older individuals. This trend puts a significant strain on healthcare systems, especially in terms of histopathologic diagnostics required for Mohs micrographic surgery (MMS), which is used to treat BCC in sensitive locations to minimize tissue loss. This study aims to address the challenges in BCC detection within MMS whole-slide images (WSIs) by developing and evaluating a deep learning model that bridges weakly-supervised learning with interpretable segmentation-based methods through attention maps. Utilizing datasets from two medical centers, the model demonstrated an average area under the ROC curve (AUC) of 0.958 on internal testing and an AUC of 0.934 on an independent third external dataset despite no fine-tuning or preprocessing for the latter. Attention maps provided insights into the model’s decision-making, highlighting critical regions for slide-level classification. The sensitivity of the attention maps in localizing tumor regions was 0.853 when no filtering was applied and gave 8 revision false positives per slide on average and was reduced to an average of 2 false positives per slide with a sensitivity of 0.873 when detections smaller than 200 micrometers were removed from the attention maps. These findings indicate that the deep learning model is highly effective in detecting BCC in MMS WSIs, with robust performance across different datasets and conditions.The use of attention maps enhances the model’s interpretability, making it a promising tool for aiding dermatopathologists and MMS surgeons.

Exploring better sparsely annotated shadow detection

Sparsely annotated image segmentation has attracted increasing attention due to its low labeling cost. However, existing weakly-supervised shadow detection methods require complex training procedures, and there is still a significant performance gap compared to fully-supervised methods. This paper summarizes two current challenges in sparsely annotated shadow detection, i.e., weak supervision diffusion and poor structure recovery, and attempts to alleviate them. To this end, we propose a one-stage weakly-supervised learning framework to facilitate sparsely annotated shadow detection. Specifically, we first design a simple yet effective semantic affinity module (SAM) that adaptively propagates scribble supervision to unlabeled regions using a gradient diffusion scheme. Then, to better recover shadow structures, we introduce a feature-guided edge-aware loss, which leverages higher-level semantic relations to perceive shadow boundaries, while avoiding interference from ambiguous regions. Finally, we present an intensity-guided structure consistency loss to ensure that the same images with different brightness are predicted to be consistent shadow masks, which can be regarded as a self-consistent mechanism to improve the model’s generalization ability. Experimental results on three benchmark datasets demonstrate that our approach significantly outperforms previous weakly-supervised methods and achieves competitive performance in comparison to recent state-of-the-art fully-supervised methods.

Progressively global–local fusion with explicit guidance for accurate and robust 3d hand pose reconstruction

Parametric and non-parametric methods are two commonly used strategies in current 3D hand pose reconstruction. Parametric methods predict low-dimensional parameters to fit a predefined hand model to the input image. Benefiting from the prior knowledge of hand models, parametric methods guarantee plausible hand poses, whereas the pose estimation accuracy is limited due to nonlinear regression and spatial information loss. Differently, non-parametric methods directly estimate the coordinates of keypoints or mesh vertices from the input image. The reconstructed 3D hand poses show high precision but may be less robust. In this paper, we integrate the advantages of two methods for accurate and robust hand pose reconstruction. Specifically, we disentangle the hand pose reconstruction into global modeling and local refinement, which is performed in a coarse-to-fine manner. Firstly, we utilize global features from the encoder to generate the initial estimation by a parametric method, which aims to provide the prior knowledge of the human hand for subsequent processes. Then, we gradually fuse multi-scale contextual features for local refinement by explicitly integrating global prior information and local visual features. In particular, we introduce a consecutive pixel-aligned feature retrieval module to extract fine-grained information from visual features, thereby achieving pixel-level alignment. Furthermore, we demonstrate that our method can be extended to weakly-supervised learning where only sparse pose annotations are needed, potentially alleviating the burden of meticulous mesh annotation. The effectiveness and robustness of our method are substantiated through both fully- and weakly-supervised experiments, demonstrating superior performance compared to state-of-the-art methods. We plan to release our code at https://github.com/Kun-Gao/P_GLFnet.

CORAL: Collaborative Automatic Labeling System Based on Large Language Models

In the era of big data, data annotation is integral to numerous applications. However, it is widely acknowledged as a laborious and time-consuming process, significantly impeding the scalability and efficiency of data-driven applications. To reduce the human cost, we demonstrate CORAL, a collaborative automatic labeling system driven by large language models (LLMs), which achieves high-quality annotation with the least human effort. Firstly, CORAL employs LLM to automatically annotate vast datasets, generating coarse-grained labels. Subsequently, a weakly-supervised learning module trains small language models (SLMs) using noisy label learning techniques to distill accurate labels from LLM's annotations. It also allows statistical analysis of model outcomes to identify potentially erroneous labels, reducing the human cost of error detection. Furthermore, CORAL supports iterative refinement by LLMs and SLMs using manually corrected labels, thereby ensuring continual enhancement in annotation quality and model performance. A visual interface enables annotation process monitoring and result analysis.

Towards more reliable photovoltaic energy conversion systems: A weakly-supervised learning perspective on anomaly detection

With the increasing popularity of photovoltaic (PV) systems, both academia and industry have been paying growing attention to fault prediction and health management. Although deep learning has achieved remarkable results in this field, the high cost of labeled data acquisition has become a bottleneck for its application. While unsupervised methods can reduce this cost, they fail to effectively utilize the prior knowledge of anomalous samples. To address this issue, this paper innovatively proposes a weakly-supervised anomaly detection network based on feature map conversion and hypersphere transformation (FMC-HT). Firstly, PV electroluminescence (EL) images are input into the feature extraction layer based on feature map conversion, which can reduce redundancy, enhance information density, and achieve efficient feature extraction through feature map conversion. Subsequently, inverse squared norm loss is set to utilize the knowledge of unlabeled sample features and labeled anomalous sample features, and backpropagation is performed separately to train the hypersphere transformation network, ultimately achieving effective distinction between normal and anomalous samples. On real PV EL datasets, this model exhibits impressive performance and remains stable under different prior knowledge and anomaly rates. This study not only provides a new solution for anomaly detection in PV systems but also expands new directions for the application of deep learning in scenarios with limited labeled data.

Anatomy-specific Progression Classification in Chest Radiographs via Weakly Supervised Learning.

Purpose To develop a machine learning approach for classifying disease progression in chest radiographs using weak labels automatically derived from radiology reports. Materials and Methods In this retrospective study, a twin neural network was developed to classify anatomy-specific disease progression into four categories: improved, unchanged, worsened, and new. A two-step weakly supervised learning approach was employed, pretraining the model on 243 008 frontal chest radiographs from 63 877 patients (mean age, 51.7 years ± 17.0 [SD]; 34 813 [55%] female) included in the MIMIC-CXR database and fine-tuning it on the subset with progression labels derived from consecutive studies. Model performance was evaluated for six pathologic observations on test datasets of unseen patients from the MIMIC-CXR database. Area under the receiver operating characteristic (AUC) analysis was used to evaluate classification performance. The algorithm is also capable of generating bounding-box predictions to localize areas of new progression. Recall, precision, and mean average precision were used to evaluate the new progression localization. One-tailed paired t tests were used to assess statistical significance. Results The model outperformed most baselines in progression classification, achieving macro AUC scores of 0.72 ± 0.004 for atelectasis, 0.75 ± 0.007 for consolidation, 0.76 ± 0.017 for edema, 0.81 ± 0.006 for effusion, 0.7 ± 0.032 for pneumonia, and 0.69 ± 0.01 for pneumothorax. For new observation localization, the model achieved mean average precision scores of 0.25 ± 0.03 for atelectasis, 0.34 ± 0.03 for consolidation, 0.33 ± 0.03 for edema, and 0.31 ± 0.03 for pneumothorax. Conclusion Disease progression classification models were developed on a large chest radiograph dataset, which can be used to monitor interval changes and detect new pathologic conditions on chest radiographs. Keywords: Prognosis, Unsupervised Learning, Transfer Learning, Convolutional Neural Network (CNN), Emergency Radiology, Named Entity Recognition Supplemental material is available for this article. © RSNA, 2024 See also commentary by Alves and Venkadesh in this issue.

From image-level to pixel-level labeling: A weakly-supervised learning method for identifying aquaculture ponds using iterative anti-adversarial attacks guided by aquaculture features

Aquaculture mapping is essential for monitoring and managing aquaculture resources. However, accurately geotargeting individual aquaculture ponds from medium-resolution remote sensing imagery remains challenging, and convolutional deep learning methods for identifying aquaculture ponds require labor-intensive pixel-level annotations. This paper presents a novel weakly-supervised learning method to derive pixel-level labels from image-level annotations for aquaculture ponds. Our approach uses iterative anti-adversarial attacks to refine localization results from multi-scale class activation maps (CAMs). The improved method integrates two regularization methods guided by aquaculture features to form a joint loss function for adversarial samples: discriminative water region suppression and non-aquaculture water class suppression. We also propose an aquaculture feature termed CFNDWI to constrain the localization results and generate high-quality pseudo-labels. As a result, the pseudo-labels are used to train semantic segmentation networks for accurately identifying aquaculture ponds. We evaluated the performance of our method using commonly-used backbones on 10 m Sentinel-2 imagery. Our method achieves Intersection over Union (IoU) values of 0.618–0.655 for pseudo-label generation, and IoU values of 0.664–0.708 for semantic segmentation, outperforming state-of-the-art weakly-supervised methods and public datasets. The effectiveness of each module of our method was also testified through ablation experiments. Our method leverages knowledge-driven aquaculture features to guide the data-driven adversarial learning process, addressing the lack of high-quality aquaculture datasets for model training. The code for implementing our method will be accessible at https://github.com/designer1024/WSLM-AQ.

Weakly-supervised Learning of Schrödinger Equation

Weakly-supervised Learning of Schrödinger Equation

Transformer based multiple superpixel-instance learning for weakly supervised segmenting lesions of interstitial lung disease

Automatic and accurate segmentation of lesions from high-resolution computed tomography (HRCT) images play a critical role in diagnosis of interstitial lung disease (ILD). Fully supervised methods require considerable expertise annotations, resulting in time-consuming and labor-intensive cost. Contrastively, weakly supervised segmentation (WSS) algorithms could alleviate greatly this issue of medical image annotation. Multiple-instance learning (MIL), a class of weakly supervised learning (WSL), has proven to be effective in image segmentation tasks. However, most existing MIL methods based on patches or pixels lack meaningful edge information of the region of interests (ROIs) and leave out of considering the dependence relations of inter-instances. In this study, we propose a novel Transformer based multiple superpixel-instance learning (MSIL) for pixel-level lesions segmentation of ILD. This method introduces superpixels into a MIL framework, which solves the problem of inconsistency in lesion boundary segmentation caused by using patches as instances. In addition, the proposed method introduces Transformer into the MIL framework to capture global or long-range dependencies and establish the relationship between superpixel instances based on the multi-head self-attention, which solves the shortcoming that instances are independent of each other in MIL. Experimental results on private HRCT images confirm that our proposed method can achieve state-of-the-art (SOTA) performance compared to other WSS algorithms, MIL-based WSS algorithms, CAMs-based classification algorithms (CA) and fully supervised segmentation algorithms (FSS).

Semi-supervised image manipulation localization with residual enhancement

Images have become a significant medium for information transmission, while image forensics has garnered widespread attention from researchers. Due to the scarcity of finely annotated images in the field of image manipulation detection and localization, existing methods for locating manipulated images can only utilize a limited amount of data for model training. However, deep learning models typically require a large number of finely annotated images to fully leverage their powerful fitting capabilities. In this paper, we propose a semi-supervised image manipulation localization framework, employing semi-supervised learning to use unannotated images for deep learning model training. To achieve this goal, we first design a residual enhancement module that contains an encoding-decoding structure. The regression target of this branch is the pristine regions in the images to generate the reconstructed images. Next, we perform a residual operation between the reconstructed and original images to roughly locate the anomalous regions and refine the features extracted by the encoder. Secondly, we employ weakly-supervised learning by adding an image-level classification head to the final layer of the encoder. The classification head generates a class active map based on image-level classification results, which further guides the model’s feature augmentation. Finally, we formulate a specialized semi-supervised framework tailored for image manipulation detection, enabling the utilization of a large volume of unannotated data for model training. Extensive experimental results highlight the notable efficacy of the proposed approach, and outperforms state-of-the-art approaches.

A systematic comparison of deep learning methods for Gleason grading and scoring

Prostate cancer is the second most frequent cancer in men worldwide after lung cancer. Its diagnosis is based on the identification of the Gleason score that evaluates the abnormality of cells in glands through the analysis of the different Gleason patterns within tissue samples. The recent advancements in computational pathology, a domain aiming at developing algorithms to automatically analyze digitized histopathology images, lead to a large variety and availability of datasets and algorithms for Gleason grading and scoring. However, there is no clear consensus on which methods are best suited for each problem in relation to the characteristics of data and labels. This paper provides a systematic comparison on nine datasets with state-of-the-art training approaches for deep neural networks (including fully-supervised learning, weakly-supervised learning, semi-supervised learning, Additive-MIL, Attention-Based MIL, Dual-Stream MIL, TransMIL and CLAM) applied to Gleason grading and scoring tasks. The nine datasets are collected from pathology institutes and openly accessible repositories. The results show that the best methods for Gleason grading and Gleason scoring tasks are fully supervised learning and CLAM, respectively, guiding researchers to the best practice to adopt depending on the task to solve and the labels that are available.

Improving weakly-supervised object localization using adversarial erasing and pseudo label

Weakly-supervised learning approaches have gained significant attention due to their ability to reduce the effort required for human annotations in training neural networks. This paper investigates a framework for weakly-supervised object localization, which aims to train a neural network capable of predicting both the object class and its location using only images and their image-level class labels. The proposed framework consists of a shared feature extractor, a classifier, and a localizer. The localizer predicts pixel-level class probabilities, while the classifier predicts the object class at the image level. Since image-level class labels are insufficient for training the localizer, weakly-supervised object localization methods often encounter challenges in accurately localizing the entire object region. To address this issue, the proposed method incorporates adversarial erasing and pseudo labels to improve localization accuracy. Specifically, novel losses are designed to utilize adversarially erased foreground features and adversarially erased feature maps, reducing dependence on the most discriminative region. Additionally, the proposed method employs pseudo labels to suppress activation values in the background while increasing them in the foreground. The proposed method is applied to two backbone networks (MobileNetV1 and InceptionV3) and is evaluated on three publicly available datasets (ILSVRC-2012, CUB-200-2011, and PASCAL VOC 2012). The experimental results demonstrate that the proposed method outperforms previous state-of-the-art methods across all evaluated metrics.

Semi-supervised Active Learning for Video Action Detection

In this work, we focus on label efficient learning for video action detection. We develop a novel semi-supervised active learning approach which utilizes both labeled as well as un- labeled data along with informative sample selection for ac- tion detection. Video action detection requires spatio-temporal localization along with classification, which poses several challenges for both active learning (informative sample se- lection) as well as semi-supervised learning (pseudo label generation). First, we propose NoiseAug, a simple augmenta- tion strategy which effectively selects informative samples for video action detection. Next, we propose fft-attention, a novel technique based on high-pass filtering which enables effective utilization of pseudo label for SSL in video action detection by emphasizing on relevant activity region within a video. We evaluate the proposed approach on three different bench- mark datasets, UCF-101-24, JHMDB-21, and Youtube-VOS. First, we demonstrate its effectiveness on video action detec- tion where the proposed approach outperforms prior works in semi-supervised and weakly-supervised learning along with several baseline approaches in both UCF101-24 and JHMDB- 21. Next, we also show its effectiveness on Youtube-VOS for video object segmentation demonstrating its generalization capability for other dense prediction tasks in videos.

ZOOM: Learning Video Mirror Detection with Extremely-Weak Supervision

Mirror detection is an active research topic in computer vision. However, all existing mirror detectors learn mirror representations from large-scale pixel-wise datasets, which are tedious and expensive to obtain. Although weakly-supervised learning has been widely explored in related topics, we note that popular weak supervision signals (e.g., bounding boxes, scribbles, points) still require some efforts from the user to locate the target objects, with a strong assumption that the images to annotate always contain the target objects. Such an assumption may result in the over-segmentation of mirrors. Our key idea of this work is that the existence of mirrors over a time period may serve as a weak supervision to train a mirror detector, for two reasons. First, if a network can predict the existence of mirrors, it can essentially locate the mirrors. Second, we observe that the reflected contents of a mirror tend to be similar to those in adjacent frames, but exhibit considerable contrast to regions in far-away frames (e.g., non-mirror frames). To this end, in this paper, we propose ZOOM, the first method to learn robust mirror representations from extremely-weak annotations of per-frame ZerO-One Mirror indicators in videos. The key insight of ZOOM is to model the similarity and contrast (between mirror and non-mirror regions) in temporal variations to locate and segment the mirrors. To this end, we propose a novel fusion strategy to leverage temporal consistency information for mirror localization, and a novel temporal similarity-contrast modeling module for mirror segmentation. We construct a new video mirror dataset for training and evaluation. Experimental results under new and standard metrics show that ZOOM performs favorably against existing fully-supervised mirror detection methods.

Data-Efficient Graph Learning

My research strives to develop fundamental graph-centric learning algorithms to reduce the need for human supervision in low-resource scenarios. The focus is on achieving effective and reliable data-efficient learning on graphs, which can be summarized into three facets: (1) graph weakly-supervised learning; (2) graph few-shot learning; and (3) graph self-supervised learning.

Weakly-supervised auto-encoder via energy regularization and soft multi-label learning on k labeled samples

Image classification is a hot topic in computer vision tasks. As a simple unsupervised network model, auto-encoder can learn and apply features to classification. However, due to the lack of prior knowledge of auto-encoder, there are significant limitations in their performance in practical applications. On the contrary, although precise labels can guide supervised learning to learn accurately, it is difficult to obtain strong supervised information on many tasks because of the high cost of data annotation. In this paper, we propose a weakly-supervised auto-encoder via energy regularization and soft multi-label learning on k labeled samples (WSAE), where weakly-supervised learning not only reduces the workload of manual labeling but also introduces supervised information, which improves the performance of auto-encoder to a large extent. Specifically, we first explore a lower bound value of training data that needs to be labeled in weakly supervised learning. Then, we discriminate the visual consistency of k labeled samples compared to unlabeled samples with soft multi-label to efficiently increase the difference of features between classes and reduce the variation of features within classes for more discriminative feature representations and higher classification accuracy. In addition, an energy regularization is introduced to the model to fit the probability distribution of the data so that the smaller the energy, the more concentrated the probability distribution and the more ordered the system. The model with energy regularization can represent the data better, thereby improving the feature extraction ability and increasing the classification accuracy of the network. Extensive experiments are conducted on some publicly available datasets and demonstrate that our method outperforms the state-of-the-art methods.

Moderately supervised learning: definition, framework and generality

AbstractLearning with supervision has achieved remarkable success in numerous artificial intelligence (AI) applications. In the current literature, by referring to the properties of the labels prepared for the training dataset, learning with supervision is categorized as supervised learning (SL) and weakly supervised learning (WSL). SL concerns the situation where the training dataset is assigned with ideal (complete, exact and accurate) labels, while WSL concerns the situation where the training dataset is assigned with non-ideal (incomplete, inexact or inaccurate) labels. However, various solutions for SL tasks under the era of deep learning have shown that the given labels are not always easy to learn, and the transformation from the given labels to easy-to-learn targets can significantly affect the performance of the final SL solutions. Without considering the properties of the transformation from the given labels to easy-to-learn targets, the definition of SL conceals some details that can be critical to building the appropriate solutions for specific SL tasks. Thus, for practitioners in various AI application fields, it is desirable to reveal these details systematically. This article attempts to achieve this goal by expanding the categorization of SL and investigating the sub-type that plays the central role in SL. More specifically, taking into consideration the properties of the transformation from the given labels to easy-to-learn targets, we firstly categorize SL into three narrower sub-types. Then we focus on the moderately supervised learning (MSL) sub-type that concerns the situation where the given labels are ideal, but due to the simplicity in annotation, careful designs are required to transform the given labels into easy-to-learn targets. From the perspectives of the definition, framework and generality, we conceptualize MSL to present a complete fundamental basis to systematically analyse MSL tasks. At meantime, revealing the relation between the conceptualization of MSL and the mathematicians’ vision, this article as well establishes a tutorial for AI application practitioners to refer to viewing a problem to be solved from the mathematicians’ vision.

Weakly Supervised Solar Panel Mapping via Uncertainty Adjusted Label Transition in Aerial Images.

This paper proposes a novel uncertainty-adjusted label transition (UALT) method for weakly supervised solar panel mapping (WS-SPM) in aerial Images. In weakly supervised learning (WSL), the noisy nature of pseudo labels (PLs) often leads to poor model performance. To address this problem, we formulate the task as a label-noise learning problem and build a statistically consistent mapping model by estimating the instance-dependent transition matrix (IDTM). We propose to estimate the IDTM with a parameterized label transition network describing the relationship between the latent clean labels and noisy PLs. A trace regularizer is employed to impose constraints on the form of IDTM for its stability. To further reduce the estimation difficulty of IDTM, we incorporate uncertainty estimation to first improve the accuracy of noisy dataset distillation and then mitigate the negative impacts of falsely distilled examples with an uncertainty-adjusted re-weighting strategy. Extensive experiments and ablation studies on two challenging aerial data sets support the validity of the proposed UALT.