Weak Annotations Research Articles

Spectral Analysis for Semantic Segmentation with Applications on Feature Truncation and Weak Annotation

Spectral Analysis for Semantic Segmentation with Applications on Feature Truncation and Weak Annotation

Deep Reinforcement Learning for Small Bowel Path Tracking using Different Types of Annotations.

Small bowel path tracking is a challenging problem considering its many folds and contact along its course. For the same reason, it is very costly to achieve the ground-truth (GT) path of the small bowel in 3D. In this work, we propose to train a deep reinforcement learning tracker using datasets with different types of annotations. Specifically, we utilize CT scans that have only GT small bowel segmentation as well as ones with the GT path. It is enabled by designing a unique environment that is compatible for both, including a reward definable even without the GT path. The performed experiments proved the validity of the proposed method. The proposed method holds a high degree of usability in this problem by being able to utilize the scans with weak annotations, and thus by possibly reducing the required annotation cost.

Vessel-CAPTCHA: An efficient learning framework for vessel annotation and segmentation.

Deep learning techniques for 3D brain vessel image segmentation have not been as successful as in the segmentation of other organs and tissues. This can be explained by two factors. First, deep learning techniques tend to show poor performances at the segmentation of relatively small objects compared to the size of the full image. Second, due to the complexity of vascular trees and the small size of vessels, it is challenging to obtain the amount of annotated training data typically needed by deep learning methods. To address these problems, we propose a novel annotation-efficient deep learning vessel segmentation framework. The framework avoids pixel-wise annotations, only requiring weak patch-level labels to discriminate between vessel and non-vessel 2D patches in the training set, in a setup similar to the CAPTCHAs used to differentiate humans from bots in web applications. The user-provided weak annotations are used for two tasks: (1) to synthesize pixel-wise pseudo-labels for vessels and background in each patch, which are used to train a segmentation network, and (2) to train a classifier network. The classifier network allows to generate additional weak patch labels, further reducing the annotation burden, and it acts as a second opinion for poor quality images. We use this framework for the segmentation of the cerebrovascular tree in Time-of-Flight angiography (TOF) and Susceptibility-Weighted Images (SWI). The results show that the framework achieves state-of-the-art accuracy, while reducing the annotation time by ∼77% w.r.t. learning-based segmentation methods using pixel-wise labels for training.

AAWS-Net: Anatomy-aware weakly-supervised learning network for breast mass segmentation.

Accurate segmentation of breast masses is an essential step in computer aided diagnosis of breast cancer. The scarcity of annotated training data greatly hinders the model’s generalization ability, especially for the deep learning based methods. However, high-quality image-level annotations are time-consuming and cumbersome in medical image analysis scenarios. In addition, a large amount of weak annotations is under-utilized which comprise common anatomy features. To this end, inspired by teacher-student networks, we propose an Anatomy-Aware Weakly-Supervised learning Network (AAWS-Net) for extracting useful information from mammograms with weak annotations for efficient and accurate breast mass segmentation. Specifically, we adopt a weakly-supervised learning strategy in the Teacher to extract anatomy structure from mammograms with weak annotations by reconstructing the original image. Besides, knowledge distillation is used to suggest morphological differences between benign and malignant masses. Moreover, the prior knowledge learned from the Teacher is introduced to the Student in an end-to-end way, which improves the ability of the student network to locate and segment masses. Experiments on CBIS-DDSM have shown that our method yields promising performance compared with state-of-the-art alternative models for breast mass segmentation in terms of segmentation accuracy and IoU.

Constrained Domain Adaptation for Image Segmentation.

Domain Adaption tasks have recently attracted substantial attention in computer vision as they improve the transferability of deep network models from a source to a target domain with different characteristics. A large body of state-of-the-art domain-adaptation methods was developed for image classification purposes, which may be inadequate for segmentation tasks. We propose to adapt segmentation networks with a constrained formulation, which embeds domain-invariant prior knowledge about the segmentation regions. Such knowledge may take the form of anatomical information, for instance, structure size or shape, which can be known a priori or learned from the source samples via an auxiliary task. Our general formulation imposes inequality constraints on the network predictions of unlabeled or weakly labeled target samples, thereby matching implicitly the prediction statistics of the target and source domains, with permitted uncertainty of prior knowledge. Furthermore, our inequality constraints easily integrate weak annotations of the target data, such as image-level tags. We address the ensuing constrained optimization problem with differentiable penalties, fully suited for conventional stochastic gradient descent approaches. Unlike common two-step adversarial training, our formulation is based on a single segmentation network, which simplifies adaptation, while improving training quality. Comparison with state-of-the-art adaptation methods reveals considerably better performance of our model on two challenging tasks. Particularly, it consistently yields a performance gain of 1-4% Dice across architectures and datasets. Our results also show robustness to imprecision in the prior knowledge. The versatility of our novel approach can be readily used in various segmentation problems, with code available publicly.

Baggage Image Retrieval with Attention-Based Network for Security Checks

Baggage image search is the task of finding the same baggage across several cameras, which can improve the efficiency of security check. Since existing state-of-the-art image retrieval requires a significant amount of training data, we aim to investigate the few-shot situation, utilizing only a few samples for each baggage. In this paper, the framework we introduced is called AttentionNet, exploring the weak region-wise annotations as attention clues to improve the retrieval performance. Specifically, we integrate the semantic and recognition tasks using shared convolutional neural networks by multi-task learning. Semantic attention mechanism is used to explicitly re-weight the feature map for emphasizing the foreground. To prevent overfitting in few-shot training, we adopt a variant of the triplet loss to perform deep metric learning with an online hard triplet mining strategy. Once trained, AttentionNet identifies a probe image by computing its embedding’s cosine distance with images in the gallery. Experiments show that our method achieves state-of-the-art accuracy in image search. For instance, we obtain 82.9% Rank-1 score on the MVB dataset, dramatically outperforming the currently existing methods.

Semi-automated weak annotation for deep neural network skin thickness measurement.

Correctly calculating skin stiffness with ultrasound shear wave elastography techniques requires an accurate measurement of skin thickness. We developed and compared two algorithms, a thresholding method and a deep learning method, to measure skin thickness on ultrasound images. Here, we also present a framework for weakly annotating an unlabeled dataset in a time-effective manner to train the deep neural network. Segmentation labels for training were proposed using the thresholding method and validated with visual inspection by a human expert reader. We reduced decision ambiguity by only inspecting segmentations at the center A-line. This weak annotation approach facilitated validation of over 1000 segmentation labels in 2 hours. A lightweight deep neural network that segments entire 2D images was designed and trained on this weakly-labeled dataset. Averaged over six folds of cross-validation, segmentation accuracy was 57% for the thresholding method and 78% for the neural network. In particular, the network was better at finding the distal skin margin, which is the primary challenge for skin segmentation. Both algorithms have been made publicly available to aid future applications in skin characterization and elastography.

Combining weakly and strongly supervised learning improves strong supervision in Gleason pattern classification

BackgroundOne challenge to train deep convolutional neural network (CNNs) models with whole slide images (WSIs) is providing the required large number of costly, manually annotated image regions. Strategies to alleviate the scarcity of annotated data include: using transfer learning, data augmentation and training the models with less expensive image-level annotations (weakly-supervised learning). However, it is not clear how to combine the use of transfer learning in a CNN model when different data sources are available for training or how to leverage from the combination of large amounts of weakly annotated images with a set of local region annotations. This paper aims to evaluate CNN training strategies based on transfer learning to leverage the combination of weak and strong annotations in heterogeneous data sources. The trade-off between classification performance and annotation effort is explored by evaluating a CNN that learns from strong labels (region annotations) and is later fine-tuned on a dataset with less expensive weak (image-level) labels.ResultsAs expected, the model performance on strongly annotated data steadily increases as the percentage of strong annotations that are used increases, reaching a performance comparable to pathologists (kappa = 0.691 pm 0.02). Nevertheless, the performance sharply decreases when applied for the WSI classification scenario with kappa = 0.307 pm 0.133. Moreover, it only provides a lower performance regardless of the number of annotations used. The model performance increases when fine-tuning the model for the task of Gleason scoring with the weak WSI labels kappa = 0.528 pm 0.05.ConclusionCombining weak and strong supervision improves strong supervision in classification of Gleason patterns using tissue microarrays (TMA) and WSI regions. Our results contribute very good strategies for training CNN models combining few annotated data and heterogeneous data sources. The performance increases in the controlled TMA scenario with the number of annotations used to train the model. Nevertheless, the performance is hindered when the trained TMA model is applied directly to the more challenging WSI classification problem. This demonstrates that a good pre-trained model for prostate cancer TMA image classification may lead to the best downstream model if fine-tuned on the WSI target dataset. We have made available the source code repository for reproducing the experiments in the paper: https://github.com/ilmaro8/Digital_Pathology_Transfer_Learning

Towards closing the gap in weakly supervised semantic segmentation with DCNNs: Combining local and global models

Generating training sets for Deep Convolutional Neural Networks (DCNNs) is a bottleneck for modern real-world applications. This is a demanding task for applications where annotating training data is costly, such as in semantic segmentation. In the literature, there is still a gap between the performance achieved by a network trained on full and on weak annotations. In this paper, we establish a simple and natural strategy to measure this gap and to identify the components necessary to reduce it. On scribbles, we establish new state-of-the-art results: we obtain a mIoU of 75.6% without, and 75.7% with CRF post-processing. We reduce the gap by 64.2% whereas the current state-of-the-art reduces it only by 57.5%. Thanks to a formal reformulation of the weak supervision problem, a systematic study of the different components involved, and an original experimental strategy, we unravel a counter-intuitive mechanism analog to the philosophy of ensemble learning. This strategy is simple and amenable to generalizations to other weakly-supervised scenarios: averaging poor local predicted annotations with a generic naive baseline and reusing them for training a DCNN yields new state-of-the-art results. We show that our strategy accommodates effortlessly other pixel-level weak annotations such as bounding boxes and remains competitive.

Weakly supervised individual ganglion cell segmentation from adaptive optics OCT images for glaucomatous damage assessment.

Cell-level quantitative features of retinal ganglion cells (GCs) are potentially important biomarkers for improved diagnosis and treatment monitoring of neurodegenerative diseases such as glaucoma, Parkinson's disease, and Alzheimer's disease. Yet, due to limited resolution, individual GCs cannot be visualized by commonly used ophthalmic imaging systems, including optical coherence tomography (OCT), and assessment is limited to gross layer thickness analysis. Adaptive optics OCT (AO-OCT) enables in vivo imaging of individual retinal GCs. We present an automated segmentation of GC layer (GCL) somas from AO-OCT volumes based on weakly supervised deep learning (named WeakGCSeg), which effectively utilizes weak annotations in the training process. Experimental results show that WeakGCSeg is on par with or superior to human experts and is superior to other state-of-the-art networks. The automated quantitative features of individual GCLs show an increase in structure-function correlation in glaucoma subjects compared to using thickness measures from OCT images. Our results suggest that by automatic quantification of GC morphology, WeakGCSeg can potentially alleviate a major bottleneck in using AO-OCT for vision research.

Boundarymix: Generating pseudo-training images for improving segmentation with scribble annotations

Weakly-supervised semantic segmentation, as a promising solution to alleviate the burden of collecting per-pixel annotations, aims to train a segmentation model from partial weak annotations. Scribble on the object is one of the commonly used weak annotations and has shown to be sufficient for learning a decent segmentation model. Despite being effective, scribble-based weakly-supervised learning methods often lead to imprecise segmentation on object boundaries. This is mainly because the scribble annotations usually locate inside the objects and the dataset lacks annotations close to the semantic boundaries. To alleviate this issue, this paper proposes a simple-but-effective solution, i.e., BoundaryMix, which generates pseudo training image-annotation pairs from the original images to supplement the missing semantic boundaries. Specifically, given a prediction of segmentation, we cut off the regions around the estimated boundaries, which are error-prone and replace them with the contents from another image, which in effect creates new samples with less ambiguity around semantic boundaries. With training on scribbles and the on-the-fly generated pseudo annotations, the network acquires better prediction capability around the boundary region and thus improves the overall segmentation performance. By conducting experiments on PASCAL VOC 2012 dataset and POTSDAM dataset with only scribble annotations, we demonstrate the excellent performance of the proposed method and the almost closed gap between scribble-supervised and fully-supervised image segmentation.

A novel multiple instance learning framework for COVID-19 severity assessment via data augmentation and self-supervised learning.

How to fast and accurately assess the severity level of COVID-19 is an essential problem, when millions of people are suffering from the pandemic around the world. Currently, the chest CT is regarded as a popular and informative imaging tool for COVID-19 diagnosis. However, we observe that there are two issues – weak annotation and insufficient data that may obstruct automatic COVID-19 severity assessment with CT images. To address these challenges, we propose a novel three-component method, i.e., 1) a deep multiple instance learning component with instance-level attention to jointly classify the bag and also weigh the instances, 2) a bag-level data augmentation component to generate virtual bags by reorganizing high confidential instances, and 3) a self-supervised pretext component to aid the learning process. We have systematically evaluated our method on the CT images of 229 COVID-19 cases, including 50 severe and 179 non-severe cases. Our method could obtain an average accuracy of 95.8%, with 93.6% sensitivity and 96.4% specificity, which outperformed previous works.

State of the Art Cell Detection in Bone Marrow Whole Slide Images

Context: Diseases of the hematopoietic system such as leukemia is diagnosed using bone marrow samples. The cell type distribution plays a major role but requires manual analysis of different cell types in microscopy images. Aims: Automated analysis of bone marrow samples requires detection and classification of different cell types. In this work, we propose and compare algorithms for cell localization, which is a key component in automated bone marrow analysis. Settings and Design: We research fully supervised detection architectures but also propose and evaluate several techniques utilizing weak annotations in a segmentation network. We further incorporate typical cell-like artifacts into our analysis. Whole slide microscopy images are acquired from the human bone marrow samples and annotated by expert hematologists. Subjects and Methods: We adapt and evaluate state-of-the-art detection networks. We further propose to utilize the popular U-Net for cell detection by applying suitable preprocessing steps to the annotations. Statistical Analysis Used: Evaluations are performed on a held-out dataset using multiple metrics based on the two different matching algorithms. Results: The results show that the detection of cells in hematopoietic images using state-of-the-art detection networks yields very accurate results. U-Net-based methods are able to slightly improve detection results using adequate preprocessing - despite artifacts and weak annotations. Conclusions: In this work, we propose, U-Net-based cell detection methods and compare with state-of-the-art detection methods for the localization of hematopoietic cells in high-resolution bone marrow images. We show that even with weak annotations and cell-like artifacts, cells can be localized with high precision.

Texture feature extraction and optimization of facial expression based on weakly supervised clustering

In order to improve the recognition rate of weak annotation data in facial expression recognition task, this paper proposes a multi-scale and multi-region vector triangle texture feature extraction scheme based on weakly supervised clustering algorithm. According to the information gain rate of extracted features, combined with threshold selection and random dropout strategy, the best selection of vector triangle texture feature scale is explored, and the feature space is optimized under the premise of sufficient feature space information, the reduction of feature space is realized and the information redundancy is reduced. For the positive and negative expression units, the facial expression images in the data set are divided into two categories. The positive and negative facial expressions are taken to form the same kind of samples, the positive and negative facial expressions are taken to form the positive and negative samples, and the annotation labels are taken to form the weak annotation labels. The experimental results show that the best recognition rate of the proposed scheme is 84.1%, which is 5.8% higher than the unoptimized texture feature scheme.

A Weakly-Supervised Semantic Segmentation Approach Based on the Centroid Loss: Application to Quality Control and Inspection

It is generally accepted that one of the critical parts of current vision algorithms based on deep learning and convolutional neural networks is the annotation of a sufficient number of images to achieve competitive performance. This is particularly difficult for semantic segmentation tasks since the annotation must be ideally generated at the pixel level. Weakly-supervised semantic segmentation aims at reducing this cost by employing simpler annotations that, hence, are easier, cheaper and quicker to produce. In this paper, we propose and assess a new weakly-supervised semantic segmentation approach making use of a novel loss function whose goal is to counteract the effects of weak annotations. To this end, this loss function comprises several terms based on partial cross-entropy losses, being one of them the Centroid Loss. This term induces a clustering of the image pixels in the object classes under consideration, whose aim is to improve the training of the segmentation network by guiding the optimization. The performance of the approach is evaluated against datasets from two different industry-related case studies: while one involves the detection of instances of a number of different object classes in the context of a quality control application, the other stems from the visual inspection domain and deals with the localization of images areas whose pixels correspond to scene surface points affected by a specific sort of defect. The detection results that are reported for both cases show that, despite the differences among them and the particular challenges, the use of weak annotations do not prevent from achieving a competitive performance level for both.

Self-Supervised Attention Mechanism for Pediatric Bone Age Assessment With Efficient Weak Annotation.

Pediatric bone age assessment (BAA) is a common clinical practice to investigate endocrinology, genetic and growth disorders of children. Different specific bone parts are extracted as anatomical Regions of Interest (RoIs) during this task, since their morphological characters have important biological identification in skeletal maturity. Following this clinical prior knowledge, recently developed deep learning methods address BAA with an RoI-based attention mechanism, which segments or detects the discriminative RoIs for meticulous analysis. Great strides have been made, however, these methods strictly require large and precise RoIs annotations, which limits the real-world clinical value. To overcome the severe requirements on RoIs annotations, in this paper, we propose a novel self-supervised learning mechanism to effectively discover the informative RoIs without the need of extra knowledge and precise annotation-only image-level weak annotation is all we take. Our model, termed PEAR-Net for Part Extracting and Age Recognition Network, consists of one Part Extracting (PE) agent for discriminative RoIs discovering and one Age Recognition (AR) agent for age assessment. Without precise supervision, the PE agent is designed to discover and extract RoIs fully automatically. Then the proposed RoIs are fed into AR agent for feature learning and age recognition. Furthermore, we utilize the self-consistency of RoIs to optimize PE agent to understand the part relation and select the most useful RoIs. With this self-supervised design, the PE agent and AR agent can reinforce each other mutually. To the best of our knowledge, this is the first end-to-end bone age assessment method which can discover RoIs automatically with only image-level annotation. We conduct extensive experiments on the public RSNA 2017 dataset and achieve state-of-the-art performance with MAE 3.99 months. Project is available at http://imcc.ustc.edu.cn/project/ssambaa/.

A Vision-Based Two-Stage Framework for Inferring Physical Properties of the Terrain

The friction and stiffness properties of the terrain are very important pieces of information for mobile robots in motion control, dynamics parameter adjustment, trajectory planning, etc. Inferring the friction and stiffness properties in advance can improve the safety, adaptability and reliability, and reduce the energy consumption of the robot. This paper proposes a vision-based two-stage framework for pre-estimating physical properties of the terrain. We established a field terrain image dataset with weak annotations. A semantic segmentation network that can segment terrains at the pixel level was designed. Given that the same terrain also has different physical properties, we designed two kinds of image features, and we use a decision-making model to realize the mapping from terrain to physical properties. We trained and tested the network comprehensively, and experimented with the complete framework for estimating physical properties. The experimental results show that our framework has good performance.

Discretely-constrained deep network for weakly supervised segmentation

An efficient strategy for weakly-supervised segmentation is to impose constraints or regularization priors on target regions. Recent efforts have focused on incorporating such constraints in the training of convolutional neural networks (CNN), however this has so far been done within a continuous optimization framework. Yet, various segmentation constraints and regularization priors can be modeled and optimized more efficiently in a discrete formulation. This paper proposes a method, based on the alternating direction method of multipliers (ADMM) algorithm, to train a CNN with discrete constraints and regularization priors. This method is applied to the segmentation of medical images with weak annotations, where both size constraints and boundary length regularization are enforced. Experiments on two benchmark datasets for medical image segmentation show our method to provide significant improvements compared to existing approaches in terms of segmentation accuracy, constraint satisfaction and convergence speed.

Detecting helicobacter pylori in whole slide images via weakly supervised multi-task learning

Due to the difficulty to accurately define the morphologies of Helicobacter Pylori (H. pylori) and the complexity of dealing with the whole slide images (WSIs), no computer-aided solution has currently been presented for detecting H. pylori infection in WSIs. We present the first image semantic segmentation solution for the computer-aided detection of H. pylori in WSIs. The solution only requires polygon annotations as weak supervision, which roughly, instead of pixel-level accurately, label the H. pylori infected areas in WSIs. We propose a new weakly supervised multi-task learning framework (WSMLF) that aims to improve the segmentation performance by more effectively exploiting the weak supervision. To make more effective usage of the weak supervision, we extract multiple inaccurate targets representing different modes of the true target from the available weak annotations. For improvement of the segmentation performance, we design a weakly supervised multi-task learning algorithm that can automatically learn from the weighted summarization of the extracted multiple inaccurate targets. These two advances constitute the resulting technique WSMLF. Introducing the proposed WSMLF to several common deep image semantic segmentation approaches for the detection of H. pylori in WSIs, we observe that WSMLF can enable these approaches to achieve more reasonable segmentation results, which eventually improve the detection performance of H. pylori by at most 6%. WSMLF provides new thoughts for more effectively employing weak supervision to achieve more effective results for image semantic segmentation.

Unsupervised Eyeglasses Removal in the Wild.

Eyeglasses removal is challenging in removing different kinds of eyeglasses, e.g., rimless glasses, full-rim glasses, and sunglasses, and recovering appropriate eyes. Due to the significant visual variants, the conventional methods lack scalability. Most existing works focus on the frontal face images in the controlled environment, such as the laboratory, and need to design specific systems for different eyeglass types. To address the limitation, we propose a unified eyeglass removal model called the eyeglasses removal generative adversarial network (ERGAN), which could handle different types of glasses in the wild. The proposed method does not depend on the dense annotation of eyeglasses location but benefits from the large-scale face images with weak annotations. Specifically, we study the two relevant tasks simultaneously, that is, removing eyeglasses and wearing eyeglasses. Given two face images with and without eyeglasses, the proposed model learns to swap the eye area in two faces. The generation mechanism focuses on the eye area and invades the difficulty of generating a new face. In the experiment, we show the proposed method achieves a competitive removal quality in terms of realism and diversity. Furthermore, we evaluate ERGAN on several subsequent tasks, such as face verification and facial expression recognition. The experiment shows that our method could serve as a preprocessing method for these tasks.