Interactive Segmentation Method Research Articles

Surface-GCN: Learning interaction experience for organ segmentation in 3D medical images.

Accurate segmentation of organs has a great significance for clinical diagnosis, but it is still hard work due to the obscure imaging boundaries caused by tissue adhesion on medical images. Based on the image continuity in medical image volumes, segmentation on these slices could be inferred from adjacent slices with a clear organ boundary. Radiologists can delineate a clear organ boundary by observing adjacent slices. Inspired by the radiologists' delineating procedure, we design an organ segmentation model based on boundary information of adjacent slices and a human-machine interactive learning strategy to introduce clinicalexperience. We propose an interactive organ segmentation method for medical image volume based on Graph Convolution Network (GCN) called Surface-GCN. First, we propose a Surface Feature Extraction Network (SFE-Net) to capture surface features of a target organ, and supervise it by a Mini-batch Adaptive Surface Matching (MBASM) module. Then, to predict organ boundaries precisely, we design an automatic segmentation module based on a Surface Convolution Unit (SCU), which propagates information on organ surfaces to refine the generated boundaries. In addition, an interactive segmentation module is proposed to learn radiologists' experience of interactive corrections on organ surfaces to reduce interactionclicks. We evaluate the proposed method on one prostate MR image dataset and two abdominal multi-organ CT datasets. The experimental results show that our method outperforms other state-of-the-artmethods. For prostate segmentation, the proposed method conducts a DSC score of 94.49% on PROMISE12 test dataset. For abdominal multi-organ segmentation, the proposed method achieves DSC scores of 95, 91, 95, and 88% for the left kidney, gallbladder, spleen, and esophagus, respectively. For interactive segmentation, the proposed method reduces 5-10 interaction clicks to reach the same accuracy. To overcome the medical organ segmentation challenge, we propose a Graph Convolutional Network called Surface-GCN by imitating radiologist interactions and learning clinical experience. On single and multiple organ segmentation tasks, the proposed method could obtain more accurate segmentation boundaries compared with other state-of-the-artmethods.

Land-Cover Classification With High-Resolution Remote Sensing Images Using Interactive Segmentation

Deep convolutional neural network (CNN) has been increasingly applied in interpretation of remote sensing image such as automatically mapping land cover. Although the automatic CNN method achieves relatively high accuracy, there are still many misclassified areas. Considering that it is still far from practical application, this paper proposes a semi-automatic auxiliary scheme for land cover classification whose core idea is to use an interactive segmentation network. To infer the rough positions and categories of objects, a CNN is relied on to classify images in a patch-wise manner. Then an interactive segmentation method is proposed by accepting user-clicks on the inside and outside of object to guide the model for the segmentation task in the patches. This model also introduces different interactive modules to better integrate features of different scales. In addition, we create a large-scale sample library containing five common land cover categories which covers Jiangsu Province, China, and includes both aerial and satellite imagery. On our sample, we gave a thorough evaluation of most recent deep learning-based methods. The experimental results shown by our interactive segmentation also far outperform the recent semantic segmentation method, which provides a reference for semi-automatic land cover mapping.

An efficient interactive segmentation framework for medical images without pre-training.

Accurate and efficient medical image segmentation plays an important role in subsequent clinical applications such as diagnosis and surgical planning. This paper proposes an efficient interactive framework based on a graph convolutional network (GCN) for medical imagesegmentation. The initial segmentation results showed that a set of boundary control points can be generated for further interactive segmentation. We presented an adaptive interactive manner that allows the user to click on the boundary for fast interaction or drag the erroneous predicted control points for accurate correction. Furthermore, we proposed an interactive segmentation network (referred to as IVIF-GCN) to learn user experience in the interactive process by transforming interactive cues into annotations. In IVIF-GCN, a module of information fusion of image features and vertex position features (IVIF) is proposed to learn the location relationship between the current vertex and the neighboring vertices. Finally, the locations of control points around the interaction point is predicted and updatedautomatically. The proposed method achieves mean Dice of 96.6% and 91.3% on PROMISE12 and our in-house nasopharyngeal carcinoma (NPC) test sets, respectively. The experimental results showed that the proposed method outperforms the state-of-the-art segmentationmethods. The proposed interactive medical image segmentation method can efficiently improve segmentation results for clinical applications in the absence of training data. The GUI tool based on our method is available at https://github.com/Tian-lab/IGMedSeg.

Object Detection Based on the GrabCut Method for Automatic Mask Generation.

The Mask R-CNN-based object detection method is typically very time-consuming and laborious since it involves obtaining the required target object masks during training. Therefore, in order to automatically generate the image mask, we propose a GrabCut-based automated mask generation method for object detection. The proposed method consists of two stages. The first stage is based on GrabCut's interactive image segmentation method to generate the mask. The second stage is based on the object detection network of Mask R-CNN, which uses the mask from the previous stage together with the original input image and the associated label information for training. The Mask R-CNN model then automatically detects the relevant objects during testing. During experimentation with three objects from the Berkeley Instance Recognition Dataset, this method achieved a mean of average precision (mAP) value of over 95% for segmentation. The proposed method is simple and highly efficient in obtaining the mask of a segmented target object.

The Dahu graph-cut for interactive segmentation on 2D/3D images

Interactive image segmentation is an important application in computer vision for selecting objects of interest in images. Several interactive segmentation methods are based on distance transform algorithms. However, the most known distance transform, geodesic distance, is sensitive to noise in the image and to seed placement. Recently, the Dahu pseudo-distance, a continuous version of the minimum barrier distance (MBD), is proved to be more powerful than the geodesic distance in noisy and blurred images. This paper presents a method for combining the Dahu pseudo-distance with edge information in a graph-cut optimization framework and leveraging each’s complementary strengths. Our method works efficiently on both 2D/3D images and videos. Results show that our method achieves better performance than other distance-based and graph-cut methods, thereby reducing the user’s efforts.

Rethinking interactive image segmentation: Feature space annotation

Despite the progress of interactive image segmentation methods, high-quality pixel-level annotation is still time-consuming and laborious — a bottleneck for several deep learning applications. We take a step back to propose interactive and simultaneous segment annotation from multiple images guided by feature space projection. This strategy is in stark contrast to existing interactive segmentation methodologies, which perform annotation in the image domain. We show that feature space annotation achieves competitive results with state-of-the-art methods in foreground segmentation datasets: iCoSeg, DAVIS, and Rooftop. Moreover, in the semantic segmentation context, it achieves 91.5% accuracy in the Cityscapes dataset, being 74.75 times faster than the original annotation procedure. Further, our contribution sheds light on a novel direction for interactive image annotation that can be integrated with existing methodologies. The supplementary material presents video demonstrations. Code available at https://github.com/LIDS-UNICAMP/rethinking-interactive-image-segmentation.

Visual and structural feature combination in an interactive machine learning system for medical image segmentation

Currently, Convolutional Neural Networks achieve good performance in automatic image segmentation situations; however, they have not demonstrated sufficiently accurate and robust results in the case of more general and interactive systems. Also, they have been designed specifically for visual features and cannot integrate enough anatomical knowledge inside the learned models they produce. To address these problems, we propose a novel machine-learning-based framework for interactive medical image segmentation. The proposed method incorporates local anatomical knowledge learning capabilities into a bounding box-based segmentation pipeline. Region specific voxel classifiers can be learned and combined to make the model adaptive to different anatomical structures or image modalities. In addition, a spatial relationship learning mechanism is integrated to capture and use additional topological (anatomical) information. New learning procedures have been defined to integrate both types of information (visual features to characterize each substructure and spatial relationships for a relative positioning between the substructures) in a unified model. During incremental and interactive segmentation, local substructures are localized one by one, enabling partial image segmentation. Bounding box positioning within the entire image is performed automatically using previously learned spatial relationships or by the user when necessary. Inside each bounding box, atlas-based methods or CNNs that are dedicated to each substructure can be applied to automatically obtain each local segmentation. Experimental results show that (1) the proposed model is robust for segmenting objects with a small amount of training images; (2) the accuracy is similar to other methods but allows partial segmentation without requiring a global registration; and (3) the proposed method leads to accurate results with fewer user interactions and less user time than traditional interactive segmentation methods due to its spatial relationship learning capabilities.

Human‐in‐the‐Loop Segmentation of Earth Surface Imagery

AbstractSegmentation, or the classification of pixels (grid cells) in imagery, is ubiquitously applied in the natural sciences. Manual methods are often prohibitively time‐consuming, especially those images consisting of small objects and/or significant spatial heterogeneity of colors or textures. Labeling complicated regions of transition that in Earth surface imagery are represented by collections of mixed‐pixels, ‐textures, and ‐spectral signatures, can be especially error‐prone because it is difficult to reliably unmix, identify and delineate consistently. However, the success of supervised machine learning (ML) approaches is entirely dependent on good label data. We describe a fast, semi‐automated, method for interactive segmentation of N‐dimensional (x, y, N) images into two‐dimensional (x, y) label images. It uses human‐in‐the‐loop ML to achieve consensus between the labeler and a model in an iterative workflow. The technique is reproducible; the sequence of decisions made by human labeler and ML algorithms can be encoded to file, so the entire process can be played back and new outputs generated with alternative decisions and/or algorithms. We illustrate the scientific potential of segmentation of imagery of diverse settings and image types using six case studies from river, estuarine, and open coast environments. These photographic and non‐photographic imagery consist of 1‐ and 3‐bands on regular and irregular grids ranging from centimeters to tens of meters. We demonstrate high levels of agreement in label images generated by several labelers on the same imagery, and make suggestions to achieve consensus and measure uncertainty, ideal for widespread application in training supervised ML for image segmentation.

DeepRecS: From RECIST Diameters to Precise Liver Tumor Segmentation.

Liver tumor segmentation (LiTS) is of primary importance in diagnosis and treatment of hepatocellular carcinoma. Known automated LiTS methods could not yield satisfactory results for clinical use since they were hard to model flexible tumor shapes and locations. In clinical practice, radiologists usually estimate tumor shape and size by a Response Evaluation Criteria in Solid Tumor (RECIST) mark. Inspired by this, in this paper, we explore a deep learning (DL) based interactive LiTS method, which incorporates guidance from user-provided RECIST marks. Our method takes a three-step framework to predict liver tumor boundaries. Under this architecture, we develop a RECIST mark propagation network (RMP-Net) to estimate RECIST-like marks in off-RECIST slices. We also devise a context-guided boundary-sensitive network (CGBS-Net) to distill tumors' contextual and boundary information from corresponding RECIST(-like) marks, and then predict tumor maps. To further refine the segmentation results, we process the tumor maps using a 3D conditional random field (CRF) algorithm and a morphology hole-filling operation. Verified on two clinical contrast-enhanced abdomen computed tomography (CT) image datasets, our proposed approach can produce promising segmentation results, and outperforms the state-of-the-art interactive segmentation methods.

3D Magic Wand: 하모닉 필드를 이용한 메쉬 분할 기법

In this paper we present a new method for interactive segmentation of a triangle mesh by using the concavity-sensitive harmonic field and anisotropic geodesic. The proposed method only requires a single vertex in a desired feature region, while most of existing methods need explicit information on segmentation boundary. From the user-clicked vertex, a candidate region which contains the desired feature region is defined and concavity-senstive harmonic field is constructed on the region by using appropriate boundary constraints. An initial isoline is chosen from the uniformly sampled isolines on the harmonic field and optimal points on the initial isoline are determined as interpolation points. Final segmentation boundary is then constructed by computing anisotropic geodesics passing through the interpolation points. In experimental results, we demonstrate the effectiveness of the proposed method by selecting several features in various 3D models.

Seed extraction using superpixel-based SLIC for interactive image segmentation

For complex images, interactive image segmentation can achieve better segmentation performance than other approaches. However, many classical interactive segmentation methods are sensitive to the locations and number of seed pixels. We introduce a seed extraction method to address this issue. First, we propose a robust seed measure to select and mark the seed pixels that cover the whole image. We use the center pixel of each superpixel as a seed. Then, each center pixel is labeled by our seed measure. Second, we propose a relabeling method for addressing isolated seeds. The extracted seeds can be used to locate the initial object area and can serve as the inputs of other interactive segmentation methods to reduce their dependence on user interaction. Extensive experiments on the Berkeley, GrabCut, and MSRA1000 datasets demonstrate the effectiveness of our seed extraction method.

SaltISCG: Interactive Salt Segmentation Method Based on CNN and Graph Cut

Salt body extraction plays an important role in the analysis of salt structures and the exploration of oil and gas. Seismic attributes and edge detection algorithms, which require manual effort, are the conventional methods of extracting salt boundaries from seismic images. Convolutional neural networks (CNNs) have become the state-of-the-art automatic segmentation method for seismic interpretation. However, the fully automatic results of the extraction of salt boundaries may still need to be modified to become accurate and robust enough for practical production. We present a novel deep-learning-based interactive segmentation method for extracting salt boundaries. To incorporate the interaction points into our method, we transform positive and negative points into two Euclidean distance maps (EDMs), which are combined with seismic images to train our CNN model. The model is composed of a U-net and a pyramid pooling module, and it is trained on the TGS Salt Identification Challenge dataset. Then, we use a graph cut algorithm to refine the likelihood maps predicted by our CNN model and subsequently update the salt boundaries. Some field examples show that the proposed method outperforms fully automatic CNN methods with a higher matching degree of the ground truth.

T-MIS: Transparency Adaptation in Medical Image Segmentation

We often locate ourselves in a trade-off situation between what is predicted and understanding why the predictive modeling made such a prediction. This high-risk medical segmentation task is no different where we try to interpret how well has the model learned from the image features irrespective of its accuracy. We propose image-specific fine-tuning to make a deep learning model adaptive to specific medical imaging tasks. Experimental results reveal that: a) proposed model is more robust to segment previously unseen objects (negative test dataset) than state-of-the-art CNNs; b) image-specific fine-tuning with the proposed heuristics significantly enhances segmentation accuracy; and c) our model leads to accurate results with fewer user interactions and less user time than conventional interactive segmentation methods. The model successfully classified ’no polyp’ or ’no instruments’ in the image irrespective of the absence of negative data in training samples from Kvasir-seg and Kvasir-Instrument datasets.

Automatic Inside Point Localization with Deep Reinforcement Learning for Interactive Object Segmentation.

In the task of interactive image segmentation, the Inside-Outside Guidance (IOG) algorithm has demonstrated superior segmentation performance leveraging Inside-Outside Guidance information. Nevertheless, we observe that the inconsistent input between training and testing when selecting the inside point will result in significant performance degradation. In this paper, a deep reinforcement learning framework, named Inside Point Localization Network (IPL-Net), is proposed to infer the suitable position for the inside point to help the IOG algorithm. Concretely, when a user first clicks two outside points at the symmetrical corner locations of the target object, our proposed system automatically generates the sequence of movement to localize the inside point. We then perform the IOG interactive segmentation method for precisely segmenting the target object of interest. The inside point localization problem is difficult to define as a supervised learning framework because it is expensive to collect image and their corresponding inside points. Therefore, we formulate this problem as Markov Decision Process (MDP) and then optimize it with Dueling Double Deep Q-Network (D3QN). We train our network on the PASCAL dataset and demonstrate that the network achieves excellent performance.

Integration of the saliency-based seeds generation and random walks with restart for image segmentation

In the task of interactive image segmentation, the user initially indicates some pixels as target seeds and background seeds and propagates the seeds’ labels to the rest of the image. We propose an equidistant seed generation method for interactive image segmentation that combines saliency detection and random walk (RW) with restart with optimal restarting probability. Our method includes two stages: first, we utilize the saliency detection method to obtain the initial target region and generate the initial seeds. Then we relabel the seeds using a Gaussian mixture model with limited user interaction to obtain accurate seeds. The relabeled seeds can limit the range of object; therefore, in the second stage, we can select an optimal restarting probability by calculating the area of the RW that exceeds the limited range of the object seeds. Our method can generate effective seeds and thus reduce the dependence on user interaction. Furthermore, using the optimal restarting probability, we can obtain higher segmentation accuracy. Extensive experiments on the Berkeley segmentation dataset, GrabCut dataset, and MSRA1000 dataset demonstrate that our method can reduce the dependence on user interaction and achieve better performance than the latest interactive image segmentation methods, such as RW, RW with restart, normalized RW, sub-Markov RW, and label propagation through complex networks.

WDTISeg: One-Stage Interactive Segmentation for Breast Ultrasound Image Using Weighted Distance Transform and Shape-Aware Compound Loss

Accurate tumor segmentation is important for aided diagnosis using breast ultrasound. Interactive segmentation methods can obtain highly accurate results by continuously optimizing the segmentation result via user interactions. However, traditional interactive segmentation methods usually require a large number of interactions to make the result meet the requirements due to the performance limitations of the underlying model. With greater ability in extracting image information, convolutional neural network (CNN)-based interactive segmentation methods have been shown to effectively reduce the number of user interactions. In this paper, we proposed a one-stage interactive segmentation framework (interactive segmentation using weighted distance transform, WDTISeg) for breast ultrasound image using weighted distance transform and shape-aware compound loss. First, we used a pre-trained CNN to attain an initial automatic segmentation, based on which the user provided interaction points of mis-segmented areas. Then, we combined Euclidean distance transform and geodesic distance transform to convert interaction points into weighted distance maps to transfer segmentation guidance information to the model. The same CNN accepted the input image, the initial segmentation, and weighted distance maps as a concatenation input and provided a refined result, without another additional segmentation network. In addition, a shape-aware compound loss function using prior knowledge was designed to reduce the number of user interactions. In the testing phase on 200 cases, our method achieved a dice of 82.86 ± 16.22 (%) for automatic segmentation task and a dice of 94.45 ± 3.26 (%) for interactive segmentation task after 8 interactions. The results of comparative experiments proved that our method could obtain higher accuracy with fewer simple interactions than other interactive segmentation methods.

Automatic extraction and measurement of individual trees from mobile laser scanning point clouds of forests.

Background and AimsIn addition to terrestrial laser scanning (TLS), mobile laser scanning (MLS) is increasingly arousing interest as a technique which provides valuable 3-D data for various applications in forest research. Using mobile platforms, the 3-D recording of large forest areas is carried out within a short space of time. Vegetation structure is described by millions of 3-D points which show an accuracy in the millimetre range and offer a powerful basis for automated vegetation modelling. The successful extraction of single trees from the point cloud is essential for further evaluations and modelling at the individual-tree level, such as volume determination, quantitative structure modelling or local neighbourhood analyses. However, high-precision automated tree segmentation is challenging, and has so far mostly been performed using elaborate interactive segmentation methods.MethodsHere, we present a novel segmentation algorithm to automatically segment trees in MLS point clouds, applying distance adaptivity as a function of trajectory. In addition, tree parameters are determined simultaneously. In our validation study, we used a total of 825 trees from ten sample plots to compare the data of trees segmented from MLS data with manual inventory parameters and parameters derived from semi-automatic TLS segmentation.Key ResultsThe tree detection rate reached 96 % on average for trees with distances up to 45 m from the trajectory. Trees were almost completely segmented up to a distance of about 30 m from the MLS trajectory. The accuracy of tree parameters was similar for MLS-segmented and TLS-segmented trees.ConclusionsBesides plot characteristics, the detection rate of trees in MLS data strongly depends on the distance to the travelled track. The algorithm presented here facilitates the acquisition of important tree parameters from MLS data, as an area-wide automated derivation can be accomplished in a very short time.

MIDeepSeg: Minimally interactive segmentation of unseen objects from medical images using deep learning.

Segmentation of organs or lesions from medical images plays an essential role in many clinical applications such as diagnosis and treatment planning. Though Convolutional Neural Networks (CNN) have achieved the state-of-the-art performance for automatic segmentation, they are often limited by the lack of clinically acceptable accuracy and robustness in complex cases. Therefore, interactive segmentation is a practical alternative to these methods. However, traditional interactive segmentation methods require a large number of user interactions, and recently proposed CNN-based interactive segmentation methods are limited by poor performance on previously unseen objects. To solve these problems, we propose a novel deep learning-based interactive segmentation method that not only has high efficiency due to only requiring clicks as user inputs but also generalizes well to a range of previously unseen objects. Specifically, we first encode user-provided interior margin points via our proposed exponentialized geodesic distance that enables a CNN to achieve a good initial segmentation result of both previously seen and unseen objects, then we use a novel information fusion method that combines the initial segmentation with only a few additional user clicks to efficiently obtain a refined segmentation. We validated our proposed framework through extensive experiments on 2D and 3D medical image segmentation tasks with a wide range of previously unseen objects that were not present in the training set. Experimental results showed that our proposed framework 1) achieves accurate results with fewer user interactions and less time compared with state-of-the-art interactive frameworks and 2) generalizes well to previously unseen objects.

Boundary-Aware Supervoxel-Level Iteratively Refined Interactive 3D Image Segmentation With Multi-Agent Reinforcement Learning.

Interactive segmentation has recently been explored to effectively and efficiently harvest high-quality segmentation masks by iteratively incorporating user hints. While iterative in nature, most existing interactive segmentation methods tend to ignore the dynamics of successive interactions and take each interaction independently. We here propose to model iterative interactive image segmentation with a Markov decision process (MDP) and solve it with reinforcement learning (RL) where each voxel is treated as an agent. Considering the large exploration space for voxel-wise prediction and the dependence among neighboring voxels for the segmentation tasks, multi-agent reinforcement learning is adopted, where the voxel-level policy is shared among agents. Considering that boundary voxels are more important for segmentation, we further introduce a boundary-aware reward, which consists of a global reward in the form of relative cross-entropy gain, to update the policy in a constrained direction, and a boundary reward in the form of relative weight, to emphasize the correctness of boundary predictions. To combine the advantages of different types of interactions, i. e., simple and efficient for point-clicking, and stable and robust for scribbles, we propose a supervoxel-clicking based interaction design. Experimental results on four benchmark datasets have shown that the proposed method significantly outperforms the state-of-the-arts, with the advantage of fewer interactions, higher accuracy, and enhanced robustness.

Interactive prostate MR image segmentation based on ConvLSTMs and GGNN

Accurate segmentation of the prostate on magnetic resonance (MR) images plays an important role for prostate cancer diagnosis and treatment. Although many automated prostate segmentation methods have been proposed, the performance still faces several challenges, which includes large variability in prostate shape, unclear boundary, and complex intensity distribution. Therefore, the results obtained from the automated methods should be further refined by users to get a more accurate and reliable segmentation. In this paper, we propose an end-to-end interactive segmentation method to refine the automated results. A convolutional long short term memory (convLSTM) module and a gated graph neural network (GGNN) are presented in the proposed method for prostate segmentation in both automated and interactive manners. A boundary loss is proposed to train our model. We evaluated the proposed method on two public available datasets and one in–house dataset. Experimental results show that the proposed convLSTM module could obtain a DSC of 91.78% on the test dataset, which outperforms eight state-of-the-art methods. A further 1.5% improvements can be obtained by user interactions based on the GGNN. The segmentation time including user interactions and inference time was 2.3 min on average for segmenting one volume.