Supervised Image Segmentation Research Articles

Unsupervised Scene Image Text Segmentation Based on Improved CycleGAN

Scene image text segmentation is an important task in computer vision, but the complexity and diversity of backgrounds make it challenging. All supervised image segmentation tasks require paired semantic label data to ensure the accuracy of segmentation, but semantic labels are often difficult to obtain. To solve this problem, we propose an unsupervised scene image text segmentation model based on the image style transfer model cyclic uniform Generation Adversarial network (CycleGAN), which is trained by partial unpaired label data. Text segmentation is achieved by converting a complex background to a simple background. Since the images generated by CycleGAN cannot retain the details of the text content, we also introduced the Atrous spatial Pyramid pool module (ASPP) to obtain the features of the text from multiple scales. The resulting image quality is improved. The proposed method is verified by experiments on a synthetic data set, the IIIT 5k word data set and the MACT data set, which effectively segments the text and preserves the details of the text content.

Weakly supervised image segmentation beyond tight bounding box annotations

Weakly supervised image segmentation approaches in the literature usually achieve high segmentation performance using tight bounding box supervision and decrease the performance greatly when supervised by loose bounding boxes. However, compared with loose bounding box, it is much more difficult to acquire tight bounding box due to its strict requirements on the precise locations of the four sides of the box. To resolve this issue, this study investigates whether it is possible to maintain good segmentation performance when loose bounding boxes are used as supervision. For this purpose, this work extends our previous parallel transformation based multiple instance learning (MIL) for tight bounding box supervision by integrating an MIL strategy based on polar transformation to assist image segmentation. The proposed polar transformation based MIL formulation works for both tight and loose bounding boxes, in which a positive bag is defined as pixels in a polar line of a bounding box with one endpoint located inside the object enclosed by the box and the other endpoint located at one of the four sides of the box. Moreover, a weighted smooth maximum approximation is introduced to incorporate the observation that pixels closer to the origin of the polar transformation are more likely to belong to the object in the box. The proposed approach was evaluated on two public datasets using dice coefficient when bounding boxes at different precision levels were considered in the experiments. The results demonstrate that the proposed approach achieves state-of-the-art performance for bounding boxes at all precision levels and is robust to mild and moderate errors in the loose bounding box annotations. The codes are available at https://github.com/wangjuan313/wsis-beyond-tightBB.

Combining Color and Spatial Image Features for Unsupervised Image Segmentation with Mixture Modelling and Spectral Clustering

The demand for accurate and reliable unsupervised image segmentation methods is high. Regardless of whether we are faced with a problem for which we do not have a usable training dataset, or whether it is not possible to obtain one, we still need to be able to extract the desired information from images. In such cases, we are usually gently pushed towards the best possible clustering method, as it is often more robust than simple traditional image processing methods. We investigate the usefulness of combining two clustering methods for unsupervised image segmentation. We use the mixture models to extract the color and spatial image features based on the obtained output segments. Then we construct a similarity matrix (adjacency matrix) based on these features to perform spectral clustering. In between, we propose a label noise correction using Markov random fields. We investigate the usefulness of our method on many hand-crafted images of different objects with different shapes, colorization, and noise. Compared to other clustering methods, our proposal performs better, with 10% higher accuracy. Compared to state-of-the-art supervised image segmentation methods based on deep convolutional neural networks, our proposal proves to be competitive.

Semi-supervised medical image segmentation using adversarial consistency learning and dynamic convolution network.

Popular semi-supervised medical image segmentation networks often suffer from error supervision from unlabeled data since they usually use consistency learning under different data perturbations to regularize model training. These networks ignore the relationship between labeled and unlabeled data, and only compute single pixel-level consistency leading to uncertain prediction results. Besides, these networks often require a large number of parameters since their backbone networks are designed depending on supervised image segmentation tasks. Moreover, these networks often face a high over-fitting risk since a small number of training samples are popular for semi-supervised image segmentation. To address the above problems, in this paper, we propose a novel adversarial self-ensembling network using dynamic convolution (ASE-Net) for semi-supervised medical image segmentation. First, we use an adversarial consistency training strategy (ACTS) that employs two discriminators based on consistency learning to obtain prior relationships between labeled and unlabeled data. The ACTS can simultaneously compute pixel-level and image-level consistency of unlabeled data under different data perturbations to improve the prediction quality of labels. Second, we design a dynamic convolution-based bidirectional attention component (DyBAC) that can be embedded in any segmentation network, aiming at adaptively adjusting the weights of ASE-Net based on the structural information of input samples. This component effectively improves the feature representation ability of ASE-Net and reduces the overfitting risk of the network. The proposed ASE-Net has been extensively tested on three publicly available datasets, and experiments indicate that ASE-Net is superior to state-of-the-art networks, and reduces computational costs and memory overhead. The code is available at: https://github.com/SUST-reynole/ASE-Net.

Cross-Image Region Mining With Region Prototypical Network for Weakly Supervised Segmentation

Weakly supervised image segmentation trained with image-level labels usually suffers from inaccurate coverage of object areas during the generation of the pseudo groundtruth. This is because the object activation maps are trained with the classification objective and lack the ability to generalize. To improve the generality of the object activation maps, we propose a region prototypical network ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">RPNet</b> ) to explore the cross-image object diversity of the training set. Similar object parts across images are identified via region feature comparison. Object confidence is propagated between regions to discover new object areas while background regions are suppressed. Experiments show that the proposed method generates more complete and accurate pseudo object masks while achieving state-of-the-art performance on PASCAL VOC 2012 and MS COCO. In addition, we investigate the robustness of the proposed method on reduced training sets. The code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/liuweide01/RPNet-Weakly-Supervised-Segmentation</uri> .

CDRNet: accurate cup-to-disc ratio measurement with tight bounding box supervision in fundus photography using deep learning

The cup-to-disc ratio (CDR) is one of the most significant indicator for glaucoma diagnosis. Different from the use of costly fully supervised learning formulation with pixel-wise annotations in the literature, this study investigates the feasibility of accurate CDR measurement in fundus images using only tight bounding box supervision. For this purpose, we develop a two-task network named as CDRNet for accurate CDR measurement, one for weakly supervised image segmentation, and the other for bounding-box regression. The weakly supervised image segmentation task is implemented based on generalized multiple instance learning formulation and smooth maximum approximation, and the bounding-box regression task outputs class-specific bounding box prediction in a single scale at the original image resolution. To get accurate bounding box prediction, a class-specific bounding-box normalizer and an expected intersection-over-union are proposed. In the experiments, the proposed approach was evaluated by a testing set with 1200 images using CDR error and $F_1$ score for CDR measurement and dice coefficient for image segmentation. A grader study was conducted to compare the performance of the proposed approach with those of individual graders. The experimental results indicate that the proposed approach outperforms the state-of-the-art performance obtained from the fully supervised image segmentation (FSIS) approach using pixel-wise annotation for CDR measurement. Its performance is also better than those of individual graders. In addition, the proposed approach gets performance close to the state-of-the-art obtained from FSIS and the performance of individual graders for optic cup and disc segmentation. The codes are available at \url{https://github.com/wangjuan313/CDRNet}.

A lightweight weakly supervised learning segmentation algorithm for imbalanced image based on rotation density peaks

Image segmentation has been an important technique in the field of image processing. Fine-level manual annotations are very limited and difficult for a large collection of imbalanced images, where each image contains quite different small objects. However, we find that these imbalanced images have similar decision graphs obtained by a lightweight and simple clustering algorithm Density Peaks (DPeaks). Hence, in this paper, we propose a weakly supervised image segmentation algorithm. It trains a decision curve from decision graphs of a few sample imbalanced images by SVM and Support Vector Regression (SVR), which is effective for identifying the sparse region of an imbalanced image. Besides, RangeTree is applied to accelerate RDP for large images due to the high complexity of DPeaks Clustering. Experiments prove that the proposed algorithm works well on imbalanced image data sets, not only it can recognize main things, but also has the ability to identify some relatively small objects.

A Chan-Vese model based on the Markov chain for unsupervised medical image segmentation

The accurate segmentation of medical images is crucial to medical care and research; however, many efficient supervised image segmentation methods require sufficient pixel level labels. Such requirement is difficult to meet in practice and even impossible in some cases, e.g., rare Pathoma images. Inspired by traditional unsupervised methods, we propose a novel Chan-Vese model based on the Markov chain for unsupervised medical image segmentation. It combines local information brought by superpixels with the global difference between the target tissue and the background. Based on the Chan-Vese model, we utilize weight maps generated by the Markov chain to model and solve the segmentation problem iteratively using the min-cut algorithm at the superpixel level. Our method exploits abundant boundary and local region information in segmentation and thus can handle images with intensity inhomogeneity and object sparsity. In our method, users gain the power of fine-tuning parameters to achieve satisfactory results for each segmentation. By contrast, the result from deep learning based methods is rigid. The performance of our method is assessed by using four Computerized Tomography (CT) datasets. Experimental results show that the proposed method outperforms traditional unsupervised segmentation techniques.

Unsupervised Object Segmentation Based on Bi-Partitioning Image Model Integrated with Classification

The development of convolutional neural networks for deep learning has significantly contributed to image classification and segmentation areas. For high performance in supervised image segmentation, we need many ground-truth data. However, high costs are required to make these data, so unsupervised manners are actively being studied. The Mumford–Shah and Chan–Vese models are well-known unsupervised image segmentation models. However, the Mumford–Shah model and the Chan–Vese model cannot separate the foreground and background of the image because they are based on pixel intensities. In this paper, we propose a weakly supervised model for image segmentation based on the segmentation models (Mumford–Shah model and Chan–Vese model) and classification. The segmentation model (i.e., Mumford–Shah model or Chan–Vese model) is to find a base image mask for classification, and the classification network uses the mask from the segmentation models. With the classifcation network, the output mask of the segmentation model changes in the direction of increasing the performance of the classification network. In addition, the mask can distinguish the foreground and background of images naturally. Our experiment shows that our segmentation model, integrated with a classifier, can segment the input image to the foreground and the background only with the image’s class label, which is the image-level label.

CCT: Conditional Co-Training for Truly Unsupervised Remote Sensing Image Segmentation in Coastal Areas

As the fastest growing trend in big data analysis, deep learning technology has proven to be both an unprecedented breakthrough and a powerful tool in many fields, particularly for image segmentation tasks. Nevertheless, most achievements depend on high-quality pre-labeled training samples, which are labor-intensive and time-consuming. Furthermore, different from conventional natural images, coastal remote sensing ones generally carry far more complicated and considerable land cover information, making it difficult to produce pre-labeled references for supervised image segmentation. In our research, motivated by this observation, we take an in-depth investigation on the utilization of neural networks for unsupervised learning and propose a novel method, namely conditional co-training (CCT), specifically for truly unsupervised remote sensing image segmentation in coastal areas. In our idea, a multi-model framework consisting of two parallel data streams, which are superpixel-based over-segmentation and pixel-level semantic segmentation, is proposed to simultaneously perform the pixel-level classification. The former processes the input image into multiple over-segments, providing self-constrained guidance for model training. Meanwhile, with this guidance, the latter continuously processes the input image into multi-channel response maps until the model converges. Incentivized by multiple conditional constraints, our framework learns to extract high-level semantic knowledge and produce full-resolution segmentation maps without pre-labeled ground truths. Compared to the black-box solutions in conventional supervised learning manners, this method is of stronger explainability and transparency for its specific architecture and mechanism. The experimental results on two representative real-world coastal remote sensing datasets of image segmentation and the comparison with other state-of-the-art truly unsupervised methods validate the plausible performance and excellent efficiency of our proposed CCT.

The Influence of Seasonality on the Multi-Spectral Image Segmentation for Identification of Abandoned Land

Areas of agricultural land in Lithuania have decreased from 2005 to 2021 by up to 2.4%. Agricultural lands that are no longer used for their main purpose are very likely to become abandoned and the emergence of such lands can cause a variety of social, economic, and environmental problems. Therefore, it is very important to constantly monitor changes of abandoned agricultural lands. The purpose of the research is to analyse the influence of seasonality on image segmentation for the identification of abandoned land areas. Multi-spectral Sentinel-2 images from different periods (April, July, and September) and three supervised image segmentation methods (Spectral Angle Mapping (SAM), Maximum_Likelihood (ML), and Minimum distance (MD)) were used with the same parameters in this research. Studies had found that the most appropriate time to segment abandoned lands was in September, according to the SAM and ML algorithms. During this period, the intensity of the green colour was the highest and the colour brightness of abandoned lands differed from the colour intensity of other lands.

Deep learning for discovering pathological continuum of crypts and evaluating therapeutic effects: An implication for in vivo preclinical study.

Applying deep learning to the field of preclinical in vivo studies is a new and exciting prospect with the potential to unlock decades worth of underutilized data. As a proof of concept, we performed a feasibility study on a colitis model treated with Sulfasalazine, a drug used in therapeutic care of inflammatory bowel disease. We aimed to evaluate the colonic mucosa improvement associated with the recovery response of the crypts, a complex histologic structure reflecting tissue homeostasis and repair in response to inflammation. Our approach requires robust image segmentation of objects of interest from whole slide images, a composite low dimensional representation of the typical or novel morphological variants of the segmented objects, and exploration of image features of significance towards biology and treatment efficacy. Both interpretable features (eg. counts, area, distance and angle) as well as statistical texture features calculated using Gray Level Co-Occurance Matrices (GLCMs), are shown to have significance in analysis. Ultimately, this analytic framework of supervised image segmentation, unsupervised learning, and feature analysis can be generally applied to preclinical data. We hope our report will inspire more efforts to utilize deep learning in preclinical in vivo studies and ultimately make the field more innovative and efficient.

Texture Segmentation Benchmark.

The Prague texture segmentation data-generator and benchmark (mosaic.utia.cas.cz) is a web-based service designed to mutually compare and rank (recently nearly 200) different static and dynamic texture and image segmenters, to find optimal parametrization of a segmenter and support the development of new segmentation and classification methods. The benchmark verifies segmenter performance characteristics on potentially unlimited monospectral, multispectral, satellite, and bidirectional texture function (BTF) data using an extensive set of over forty prevalent criteria. It also enables us to test for noise robustness and scale, rotation, or illumination invariance. It can be used in other applications, such as feature selection, image compression, query by pictorial example, etc. The benchmark's functionalities are demonstrated in evaluating several examples of leading previously published unsupervised and supervised image segmentation algorithms. However, they are used to illustrate the benchmark functionality and not review the recent image segmentation state-of-the-art.

Semi Supervised Image Segmentation Based on Markov Random Field and Kernel K-means

Semi supervised image segmentation uses a small amount of supervised information to improve the performance of image segmentation. In the first part of this paper, we construct a submodular regularization term based on Markov random field (MRF). The regularization term is combined with kernel k-means (KKM), and a semi supervised kernel k-means algorithm (SKKM) is designed based on graph cut technique. In the second part of this paper, we combine SKKM with smooth regularizer to get MRF & SKKM. The experimental results show that, compared with the original SKKM, the segmentation results of MRF & SKKM achieve better results in common evaluation indicators; moreover, compared with the binary image of segmentation results, MRF & SKKM has higher and smoother edge fiting.

Incorporating minimal user input into deep learning based image segmentation.

Computer-assisted image segmentation techniques could help clinicians to perform the border delineation task faster with lower inter-observer variability. Recently, convolutional neural networks (CNNs) are widely used for automatic image segmentation. In this study, we used a technique to involve observer inputs for supervising CNNs to improve the accuracy of the segmentation performance. We added a set of sparse surface points as an additional input to supervise the CNNs for more accurate image segmentation. We tested our technique by applying minimal interactions to supervise the networks for segmentation of the prostate on magnetic resonance images. We used U-Net and a new network architecture that was based on U-Net (dual-input path [DIP] U-Net), and showed that our supervising technique could significantly increase the segmentation accuracy of both networks as compared to fully automatic segmentation using U-Net. We also showed DIP U-Net outperformed U-Net for supervised image segmentation. We compared our results to the measured inter-expert observer difference in manual segmentation. This comparison suggests that applying about 15 to 20 selected surface points can achieve a performance comparable to manual segmentation.

AutoSegNet: An Automated Neural Network for Image Segmentation

Neural Architecture Search (NAS) has drawn significant attention as a tool for automatically constructing deep neural networks. The generated neural networks are mainly applied for image classification, and natural language processing. However, there are increasing demands for image segmentation in various areas, such as medical image processing, satellite image object location, and autopilot technology. We propose a NAS method called Automated Segmentation Network (AutoSegNet), targeting industrial and medical image segmentation. The search architectures are constructed by stacking the downsampling layer, the bridge layer, and the upsampling layer, which are explored by a recurrent neural network. Compared with other related methods for image segmentation, the proposed method has a small search space but can explore most of the-state-of-the-art supervised image segmentation models. We perform verification on two datasets, and the results show that AutoSegNet achieves superior segmentation results with clear and continuous segmented edges, as well as better image details.

Supervised image segmentation based on superpixel and improved normalised cuts

Effective and efficient image segmentation is an important task in computer vision. As the full-automatic image segmentation is usually difficult to segment the natural image, it is an excellent solution to use interactive schemes. Here, to overcome the defects of SSNCut in its low quality and speed, the authors proposed a new interactive image segmentation method based on superpixels, must-link, cannot-link constraints and improved normalised cuts. The main contribution of their work is as follows: first, the similarity between two superpixel regions is calculated using Bhattacharyya distance. Second, they adaptively modify the weights of must-link and cannot-link constraints. Compared to SSNCut, their method greatly improves the accuracy of segmentation. Comparative experiments on open datasets show that the proposed method can get better results compared with SSNCut, GrabCut in one cut, interactive segmentation using binary partition tree, interactive graph cut, seed region growing, and simple interactive object extraction.

Effective image clustering based on human mental search

Image segmentation is one of the fundamental techniques in image analysis. One group of segmentation techniques is based on clustering principles, where association of image pixels is based on a similarity criterion. Conventional clustering algorithms, such as k-means, can be used for this purpose but have several drawbacks including dependence on initialisation conditions and a higher likelihood of converging to local rather than global optima.In this paper, we propose a clustering-based image segmentation method that is based on the human mental search (HMS) algorithm. HMS is a recent metaheuristic algorithm based on the manner of searching in the space of online auctions. In HMS, each candidate solution is called a bid, and the algorithm comprises three major stages: mental search, which explores the vicinity of a solution using Levy flight to find better solutions; grouping which places a set of candidate solutions into a group using a clustering algorithm; and moving bids toward promising solution areas. In our image clustering application, bids encode the cluster centres and we evaluate three different objective functions.In an extensive set of experiments, we compare the efficacy of our proposed approach with several state-of-the-art metaheuristic algorithms including a genetic algorithm, differential evolution, particle swarm optimisation, artificial bee colony algorithm, and harmony search. We assess the techniques based on a variety of metrics including the objective functions, a cluster validity index, as well as unsupervised and supervised image segmentation criteria. Moreover, we perform some tests in higher dimensions, and conduct a statistical analysis to compare our proposed method to its competitors. The obtained results clearly show that the proposed algorithm represents a highly effective approach to image clustering that outperforms other state-of-the-art techniques.

A New Synthetic Aperture Radar (SAR) Imaging Method Combining Match Filter Imaging and Image Edge Enhancement.

In general, synthetic aperture radar (SAR) imaging and image processing are two sequential steps in SAR image processing. Due to the large size of SAR images, most image processing algorithms require image segmentation before processing. However, the existence of speckle noise in SAR images, as well as poor contrast and the uneven distribution of gray values in the same target, make SAR images difficult to segment. In order to facilitate the subsequent processing of SAR images, this paper proposes a new method that combines the back-projection algorithm (BPA) and a first-order gradient operator to enhance the edges of SAR images to overcome image segmentation problems. For complex-valued signals, the gradient operator was applied directly to the imaging process. The experimental results of simulated images and real images validate our proposed method. For the simulated scene, the supervised image segmentation evaluation indexes of our method have more than 1.18%, 11.2% and 11.72% improvement on probabilistic Rand index (PRI), variability index (VI), and global consistency error (GCE). The proposed imaging method will make SAR image segmentation and related applications easier.

How Reliable Are the Properties of Coronal Mass Ejections Measured from a Single Viewpoint?

We present an analysis of widths and kinematic properties of coronal mass ejections (CMEs) obtained via a supervised image segmentation algorithm, the CORonal SEgmentation Technique (CORSET), on simultaneous observations from the two COR2 telescopes on the Solar Terrestrial Relations Observatory (STEREO) mission, from 2007 May to 2014 September. The sample of 460 events with measurements from two vantage points offers the opportunity to test the accuracy and constraints of single-viewpoint properties that underlie the bulk of CME research to date. In addition, we examine the dependence of the properties on the morphology of the events. The main findings are as follows. (1) The radial speeds derived from different perspectives are in good agreement with a relatively low intrinsic uncertainty of 39%. (2) Projection effects are more important for determination of CME width rather than for speed. (3) The expansion speeds depend on CME morphology, with loop-type CMEs expanding twice as fast as flux-rope CMEs, possibly underpinning the more explosive nature. (4) Triangulations of CME speed and propagation direction are optimal from viewpoints separated by 60??90?; e.g., between the Lagrangian points L1 and L5 (or L4). (5) The projected speeds are underestimated, on average, by at least 20% when compared to their deprojected (triangulated) values. We also discuss in detail the lessons learned from the application of the CORSET algorithm to event tracking. Our findings should hopefully be a useful guide in the use of (semi)automated algorithms for extraction of CME physical parameters and in the interpretation of single-viewpoint observations (likely to be the norm after the end of the STEREO mission).