Image-level Annotations Research Articles

Patch-based weakly supervised semantic segmentation network for crack detection

Obtaining spatial and topological information for cracks in construction materials is important for the evaluation of service performance in infrastructure engineering. Manually extracting crack information from an image is a tedious process. Recently, popular deep learning-based semantic segmentation technologies have been employed to alleviate this problem. However, existing semantic segmentation methods for crack detection are fully supervised, i.e., these methods require manual annotation of data to obtain pixel-level labels for training, which is time-consuming. To solve this problem, this paper proposes a patch-based weakly supervised semantic segmentation network for crack detection. The proposed method uses image-level annotation as the supervision condition and fully considers the local similarity of the crack topology in the image. The use of patches cropped from the image as the input in this method can reduce the image complexity significantly without losing the spatial location information of the crack. A discriminative localization technique is used to extract rough location information of the crack from a trained classification network, which is then refined by a conditional random field to obtain a synthetic label. These synthetic labels can replace the manually annotated pixel-level labels for the training of the segmentation network. Thereafter, a neighborhood fusion strategy is used to merge the patches into the final output. Two datasets are employed to train and evaluate the proposed method. The results indicate that this method can achieve a performance comparable to that of fully supervised methods (an MIoU of 0.821 and F-score of 0.8 were obtained for the fully supervised method compared with an MIoU of 0.782 and F-score of 0.741 with the proposed method), while reducing the annotation workload by approximately 80%.

WSODPB: Weakly supervised object detection with PCSNet and box regression module

Weakly supervised object detection (WSOD) task uses only image-level annotations to train object detection task. WSOD does not require time-consuming instance-level annotations, which attracts more and more attention. Previous weakly supervised object detection methods iteratively update detectors and pseudo-labels or use rules-based methods, which could not generate complete and accurate proposals. We utilize the features extracted by the convolutional layers to optimize the proposals generated by rules-based methods, and solve the above problem through combining the two different features. Then, a box regression module is added to the weakly supervised object detection network, which supervised by a proposal completeness scoring network (PCSNet). The box regression module modifies proposals to obtain higher intersection-over-Union (IoU) with ground truth. PCSNet scores the proposal output from the box regression network and utilizes the score to improve the box regression module. In addition, we take advantage of the random proposal scoring (RPS) algorithm for generating more accurate pseudo labels to train the box regression module. The results show that our methods have made significant improvements on PASCAL VOC 2007 and 2012 datasets.

Spatial division networks for weakly supervised detection

With only global image-level annotations, weakly supervised learning of deep convolutional neural networks has shown enough capacity in classification and localization but lack of ability to present the detection explicitly. In this work, we propose a novel spatial division network, which is applied to detect bounding boxes only with weak supervision. The essence of our model is two innovative differentiable modules, determination network and parameterized division, which perform the spatial division in feature maps of classification networks. After training, the learned parameters of the spatial division would correspond to a set of predicted bounding box coordinates. To demonstrate the effectiveness of our model for multi-label classification and weakly supervised detection, we conduct extensive experiments on the multi-MNIST dataset. Experimental results show our spatial division networks can (1) help improve the accuracy of multi-label classification, (2) implement in an end-to-end way only with the image-level annotations, and (3) output accurate bounding box coordinate, thereby achieving multi-digits detection.

Weakly Supervised Learning for Raindrop Removal on a Single Image

In this paper, we address a problem of view-disturbing raindrop removal on a single image. In existing methods to tackle this problem, machine learning based ones seem promising but require elaborate pairwise images, i.e., the raindrop-degraded image and the corresponding clean image of the same scene, for training. To overcome this drawback, we propose a weakly supervised learning based model in the absence of pairwise training examples, which needs only a collection of images with image-level annotations indicating the presence/absence of raindrops for training. Specifically, we train a raindrop detector for highlighting regions of raindrops in a multi-task learning manner. Then, we propose an attention-based generative network for raindrop removal and introduce a weighted preservation loss to retain the non-raindrop details. Specially, our model can be mixedly trained with pairwise and unpaired samples, which enables us to conveniently adapt the model to a new domain. Experiments verify the effect of the proposed method. Especially, using only weakly-supervised learning, our method can achieve comparable results with state-of-the-art strongly-supervised learning methods.

Automatic Weakly Supervised Object Detection From High Spatial Resolution Remote Sensing Images via Dynamic Curriculum Learning

In this article, we focus on tackling the problem of weakly supervised object detection from high spatial resolution remote sensing images, which aims to learn detectors with only image-level annotations, i.e., without object location information during the training stage. Although promising results have been achieved, most approaches often fail to provide high-quality initial samples and thus are difficult to obtain optimal object detectors. To address this challenge, a dynamic curriculum learning strategy is proposed to progressively learn the object detectors by feeding training images with increasing difficulty that matches current detection ability. To this end, an entropy-based criterion is firstly designed to evaluate the difficulty for localizing objects in images. Then, an initial curriculum that ranks training images in ascending order of difficulty is generated, in which easy images are selected to provide reliable instances for learning object detectors. With the gained stronger detection ability, the subsequent order in the curriculum for retraining detectors is accordingly adjusted by promoting difficult images as easy ones. In such way, the detectors can be well prepared by training on easy images for learning from more difficult ones and thus gradually improve their detection ability more effectively. Moreover, an effective instance-aware focal loss function for detector learning is developed to alleviate the influence of positive instances of bad quality and meanwhile enhance the discriminative information of class-specific hard negative instances. Comprehensive experiments and comparisons with state-of-the-art methods on two publicly available data sets demonstrate the superiority of our proposed method.

Progressive Contextual Instance Refinement for Weakly Supervised Object Detection in Remote Sensing Images

Weakly supervised learning has been attracting much attention due to its broad applications, which only requires image-level annotations to indicate whether there exist objects in the images. Currently, most of the existing weakly supervised object detection (WSOD) methods are inclined to seek only one top-scoring object instance per image from noisy proposals to train the corresponding object detector. However, more than one same-class instances often exist in the large-scale, cluttered remote sensing images. Thus, selecting only one top-scoring proposal usually results in highlighting the most representative part of an object rather than the whole object, which may cause learning a suboptimal object detector by losing much important information. To address this problem, a novel end-to-end progressive contextual instance refinement (PCIR) method is proposed to perform WSOD. Specifically, a dual-contextual instance refinement (DCIR) strategy is designed to divert the focus of the detection network from the local distinct part to the whole object and further to other potential instances by leveraging both local and global context information. Benefiting from DCIR, a progressive proposal self-pruning (PPSP) strategy is further developed to mitigate the influence of the complex background by dynamically rejecting the negative training proposals. Comprehensive experiments on the challenging NWPU VHR-10.v2 and DIOR data sets clearly demonstrate that the proposed method can significantly boost the detection accuracy compared with the state of the arts.

Object Instance Mining for Weakly Supervised Object Detection

Weakly supervised object detection (WSOD) using only image-level annotations has attracted growing attention over the past few years. Existing approaches using multiple instance learning easily fall into local optima, because such mechanism tends to learn from the most discriminative object in an image for each category. Therefore, these methods suffer from missing object instances which degrade the performance of WSOD. To address this problem, this paper introduces an end-to-end object instance mining (OIM) framework for weakly supervised object detection. OIM attempts to detect all possible object instances existing in each image by introducing information propagation on the spatial and appearance graphs, without any additional annotations. During the iterative learning process, the less discriminative object instances from the same class can be gradually detected and utilized for training. In addition, we design an object instance reweighted loss to learn larger portion of each object instance to further improve the performance. The experimental results on two publicly available databases, VOC 2007 and 2012, demonstrate the efficacy of proposed approach.

Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation.

The medical imaging literature has witnessed remarkable progress in high-performing segmentation models based on convolutional neural networks. Despite the new performance highs, the recent advanced segmentation models still require large, representative, and high quality annotated datasets. However, rarely do we have a perfect training dataset, particularly in the field of medical imaging, where data and annotations are both expensive to acquire. Recently, a large body of research has studied the problem of medical image segmentation with imperfect datasets, tackling two major dataset limitations: scarce annotations where only limited annotated data is available for training, and weak annotations where the training data has only sparse annotations, noisy annotations, or image-level annotations. In this article, we provide a detailed review of the solutions above, summarizing both the technical novelties and empirical results. We further compare the benefits and requirements of the surveyed methodologies and provide our recommended solutions. We hope this survey article increases the community awareness of the techniques that are available to handle imperfect medical image segmentation datasets.

A semi-automated annotation algorithm based on weakly supervised learning for medical images

For medical image recognition, deep learning requires a massive training set, while annotation work is a tedious and time-consuming process because of the high technical threshold. Furthermore, it is difficult to guarantee annotation accuracy due to the knowledge, skills, and status of the annotator. In this research, we propose a semi-automated annotation model based on weakly supervised learning. Moreover, a target-level annotation method is proposed based on weakly supervised learning that is guided by machine learning. The machine learning method is used to screen the regions of interest (RoIs), whose semantic feature vectors are extracted by the deep learning method. Then, the machine learning method is used to cluster them, and the RoIs are finally classified and labeled by a distance comparison. Therefore, this model achieves target-level semi-automated annotation by only using image-level annotations. We applied this method to ultrasound imaging of thyroid papillary carcinoma. The experiments demonstrate the potential of this new methodology to reduce the workload for pathologists and increase the objectivity of diagnoses. We find that 89.8% of papillary thyroid carcinoma regions can be detected automatically, while 82.6% of benign and normal tissue can be excluded without the use of any additional immunohistochemical markers or human intervention.

Improvising Weakly Supervised Object Detection WSOD using Deep Learning Technique

Object detection is closely related with video and image analysis. Under computer vision technology, object detection model training with image-level labels only is challenging research area.Researchers have not yet discovered accurate model for Weakly Supervised Object Detection (WSOD). WSOD is used for detecting and localizing the objects under the supervision of image level annotations only.The proposed work usesself-paced approach which is applied on region proposal network of Faster R-CNN architecture which gives better solution from previous weakly-supervised object detectors and it can be applied for computer visionapplications in near future.

A progressive learning framework based on single-instance annotation for weakly supervised object detection

Fully-supervised object detection (FSOD) and weakly-supervised object detection (WSOD) are two extremes in the field of object detection. The former relies entirely on detailed bounding-box annotations while the later discards them completely. To balance these two extremes, we propose to make use of the so-called single-instance annotations, i.e., all images that contain only a single object are labeled with the corresponding bounding-boxes. By using such instance annotations of the simplest images, we propose a progressive learning framework that integrates image-level learning, single-instance learning, and multi-instance learning into an end-to-end network. Specifically, our framework is composed of three parallel streams that share a proposal feature extractor. The first stream is supervised by image-level annotations, which provides global information of all training data for the shared feature extractor. The second stream is supervised by single-instance annotations to bridge the features learning gap between the image level and instance level. To further learn from complex images, we propose an overlap-based instance mining algorithm to mine pseudo multi-instance annotations from the detection results of the second stream, and use them to supervise the third stream. Our method achieves a trade-off between the detection accuracy and annotation cost. Extensive experiments demonstrate the effectiveness of our proposed method on the PASCAL VOC and MS-COCO dataset, implying that a few single-instance annotations can improve the detection performance of WSOD significantly (more than 10%) and reduce the average annotation cost of FSOD greatly (more than 5 times).

Multiple Linear Discriminant Models for Extracting Salient Characteristic Patterns in Capsule Endoscopy Images for Multi-Disease Detection.

Background: Computer-aided disease detection schemes from wireless capsule endoscopy (WCE) videos have received great attention by the researchers for reducing physicians’ burden due to the time-consuming and risky manual review process. While single disease classification schemes are greatly dealt by the researchers in the past, developing a unified scheme which is capable of detecting multiple gastrointestinal (GI) diseases is very challenging due to the highly irregular behavior of diseased images in terms of color patterns. Method: In this paper, a computer-aided method is developed to detect multiple GI diseases from WCE videos utilizing linear discriminant analysis (LDA) based region of interest (ROI) separation scheme followed by a probabilistic model fitting approach. Commonly in training phase, as pixel-labeled images are available in small number, only the image-level annotations are used for detecting diseases in WCE images, whereas pixel-level knowledge, although a major source for learning the disease characteristics, is left unused. In view of learning the characteristic disease patterns from pixel-labeled images, a set of LDA models are trained which are later used to extract the salient ROI from WCE images both in training and testing stages. The intensity patterns of ROI are then modeled by a suitable probability distribution and the fitted parameters of the distribution are utilized as features in a supervised cascaded classification scheme. Results: For the purpose of validation of the proposed multi-disease detection scheme, a set of pixel-labeled images of bleeding, ulcer and tumor are used to extract the LDA models and then, a large WCE dataset is used for training and testing. A high level of accuracy is achieved even with a small number of pixel-labeled images. Conclusion: Therefore, the proposed scheme is expected to help physicians in reviewing a large number of WCE images to diagnose different GI diseases.

Convolutional Neural Network Based Weakly Supervised Learning for Aircraft Detection From Remote Sensing Image

Object detection methods based on Convolutional Neural Networks (CNNs) require a large number of images with annotation information to train. In aircraft detection from remote sensing images (RSIs), aircraft targets are usually small and the cost of manual annotation is very high. In this article, we tackle the problem of weakly supervised aircraft detection from RSIs, which aims to learn detectors with only image-level annotations, i.e., without bounding-box labeled data during the training stage. Based on the fact that the feature maps learned from the CNN network are localizable, we propose a simple yet efficient aircraft detection algorithm called Weakly Supervised Learning in AlexNet (AlexNet-WSL). In AlexNet-WSL, we utilize the AlexNet CNN as backbone network, but replace the last two fully connected layers with a Global Average Pooling (GAP) and two convolutional layers. Based on the class activation maps, we generate heat maps via reverse weighting for locating the target object. Unlike object detection methods that require object location data for training, our proposal only needs image-level labelled data. We furthermore build a set of remote sensing aircraft images, the Weakly Supervised Aircraft Detection Dataset (WSADD) for algorithm benchmarking. The experimental results on the WSADD show that AlexNet-WSL effectively detects the aircraft and achieves a detection effect equivalent to the Faster R-CNN method and the YOLOv3 method, which both require bounding-box labelled data for training, with a lower false alarm rate and a shorter training time.

Weakly Supervised Object Detection Using Complementary Learning and Instance Clustering

Supervised object detection schemes use fully annotated training data, which is fairly expensive to constitute. Whereas, weakly supervised object detection (WSOD) uses only image-level annotations for training which are much simpler to acquire. WSOD is a challenging task since it aims to learn object localization and detection with image-level labels. In line with this assertion, in this paper, we present an end-to-end framework for WSOD based on discriminative feature learning. We use the objectness technique to get initial proposals from the images. Afterwards, two complementary networks are trained in parallel to obtain discriminative image features, which are channel-wise concatenated with the features of the third network. We name this classification network designed for discriminative feature learning as fused complementary network. This network learns the proposals enclosing whole object instances by complementary features which ultimately learns to predict the high probabilities for whole objects than proposals containing only object parts. Clustering is then hierarchically performed on the region proposals. Our clustering method, named instance clustering, first performs inter-class clustering followed by iterative intra-class clustering using intersection-over-union metric to obtain spatially adjacent cluster members corresponding to each object instance. In each intra-class clustering iteration, the high scoring proposal is set as centroid from each intra-class cluster. Experiments are conducted on PASCAL VOC2007 and PASCAL VOC2012 datasets. Both qualitative and quantitative results have shown improved WSOD performance on these benchmarks.

Simultaneous Segmentation and Classification of Mass Region From Mammograms Using a Mixed-Supervision Guided Deep Model

Automatic diagnosis based on medical imaging necessitates both lesion segmentation and disease classification. Lesion segmentation requires pixel-level annotations while disease classification only requires image-level annotations. The two tasks are usually studied separately despite the latter problem relies on the former. Motivated by the close correlation between them, we propose a mixed-supervision guided method and a residual-aided classification U-Net model (ResCU-Net) for joint segmentation and benign-malignant classification. By coupling the strong supervision in the form of segmentation mask and weak supervision in the form of benign-malignant label through a simple annotation procedure, our method efficiently segments tumor regions while simultaneously predicting a discriminative map for identifying the benign-malignant types of tumors. Our network, ResCU-Net, extends U-Net by incorporating the residual module and the SegNet architecture to exploit multilevel information for achieving improved tissue identification. With experiments on a public mammogram database of INbreast, we validate the effectiveness of our method and achieve consistent improvements over state-of-the-art models.

Instance search based on weakly supervised feature learning

Instance search has been conventionally addressed as an image retrieval issue. In the existing solutions, traditional hand-crafted features and global deep features have been widely adopted. Unfortunately, since the features are not directly derived from the exact area of an instance in an image, satisfactory performance from most of them is undesirable. In this paper, a compact instance level feature representation is proposed. The scheme basically consists of two convolutional neural network (CNN) pipelines. One is designed for localizing potential instances from an image, while another is trained to learn object-aware weights to produce distinctive features. The sensitivity to the unknown categories, the distinctiveness to different instances, and most importantly, the capability of localizing an instance in an image are all carefully considered in the feature design. Moreover, both pipelines only require image level annotations, which makes the framework feasible for large-scale image collections with variety of instances. To the best of our knowledge, this is the first piece of work that builds the instance level representation based on weakly supervised object detection.

Machine vision-based automatic disease symptom detection of onion downy mildew

The effective crop management is major issue in recent agriculture because the cultivation area per farmer is increasing consistently while the aging-related reductions in the labor force. To manage crop cultivation effectively, it needs automatic monitoring in farmland. This paper presents an image-based field monitoring system for automatically crop monitoring and consists of constructing field monitoring system for periodic capturing of onion field images, training the deep neural network model for detecting the disease symptom, and evaluating performance of the developed system. The field monitoring system was composed of a PTZ camera, a motor system, wireless transceiver, and image logging module. The deep learning model was trained based on weakly supervised learning method that can classify and localize objects only with image-level annotation. It is effective to recognize crop disease symptom which has ambiguous boundary. The model was trained using captured onion images using the filed monitoring system, and 6 classes including the disease symptom were classified. The detected disease symptom was localized from background through thresholding of the class activation map. The 60% of maximum value in class activation map was determined as an Optimal threshold for disease symptom localization. Identification performance of disease symptom was evaluated using mAP metric by IoU. The results show that the mAP at IoU criteria 0.5, which should have over 50% overlap, was the highest in all models from 74.1 to 87.2. The results showed that the developed field monitoring system could automatically detect onion disease symptoms in real-time.

Weakly- and Semi-Supervised Fast Region-Based CNN for Object Detection

Learning an effective object detector with little supervision is an essential but challenging problem in computer vision applications. In this paper, we consider the problem of learning a deep convolutional neural network (CNN) based object detector using weakly-supervised and semi-supervised information in the framework of fast region-based CNN (Fast R-CNN). The target is to obtain an object detector as accurate as the fully-supervised Fast R-CNN, but it requires less image annotation effort. To solve this problem, we use weakly-supervised training images (i.e., only the image-level annotation is given) and a few proportions of fully-supervised training images (i.e., the bounding box level annotation is given), that is a weakly- and semi-supervised (WASS) object detection setting. The proposed solution is termed as WASS R-CNN, in which there are two main components. At first, a weakly-supervised R-CNN is firstly trained; after that semi-supervised data are used for finetuning the weakly-supervised detector. We perform object detection experiments on the PASCAL VOC 2007 dataset. The proposed WASS R-CNN achieves more than 85% of a fully-supervised Fast R-CNN’s performance (measured using mean average precision) with only 10% of fully-supervised annotations together with weak supervision for all training images. The results show that the proposed learning framework can significantly reduce the labeling efforts for obtaining reliable object detectors.

Learning Object Scale With Click Supervision for Object Detection

Weakly-supervised object detection has recently attracted increasing attention since it only requires image-level annotations. However, the performance obtained by existing methods is still far from being satisfactory compared with fully-supervised object detection methods. To achieve a good trade-off between annotation cost and object detection performance, we propose a simple yet effective method which incorporates CNN visualization with click supervision to generate the pseudo ground-truths (i.e., bounding boxes). These pseudo ground-truths can be used to train a fully-supervised detector. To estimate the object scale, we firstly adopt a proposal selection algorithm to preserve high-quality proposals, and then generate Class Activation Maps (CAMs) for these preserved proposals by the proposed CNN visualization algorithm called Spatial Attention CAM. Finally, we fuse these CAMs together to generate pseudo ground-truths and train a fully-supervised object detector with these ground-truths. Experimental results on the PASCAL VOC 2007 and VOC 2012 datasets show that the proposed method can obtain much higher accuracy for estimating the object scale, compared with the state-of-the-art image-level based methods and the center-click based method.

Adaptively Denoising Proposal Collection for Weakly Supervised Object Localization

In this paper, we address the problem of weakly supervised object localization (WSL), which trains a detection network on the dataset with only image-level annotations. The proposed approach is built on the observation that the proposal set from the training dataset is a collection of background, object parts, and objects. Several strategies are taken to adaptively eliminate the noisy proposals and generate pseudo object-level annotations for the weakly labeled dataset. A multiple instance learning (MIL) algorithm enhanced by mask-out strategy is adopted to collect the class-specific object proposals, which are then utilized to adapt a pre-trained classification network to a detection network. In addition, the detection results from the detection network are re-weighted by jointly considering the detection scores and the overlap ratio of proposals in a proposal subset optimization framework. The optimal proposals work as object-level labels that enable a pseudo-strongly supervised dataset for training the detection network. Consequently, we establish a fully adaptive detection network. Extensive evaluations on the PASCAL VOC 2007 and 2012 datasets demonstrate a significant improvement compared with the state-of-the-art methods.