Interactive Image Segmentation Method Research Articles

One-Click-Based Perception for Interactive Image Segmentation.

Existing deep learning-based interactive image segmentation methods have significantly reduced the user's interaction burden with simple click interactions. However, they still require excessive numbers of clicks to continuously correct the segmentation for satisfactory results. This article explores how to harvest accurate segmentation of interested targets while minimizing the user interaction cost. To achieve the above goal, we propose a one-click-based interactive segmentation approach in this work. For this particularly challenging problem in the interactive segmentation task, we build a top-down framework dividing the original problem into a one-click-based coarse localization followed by a fine segmentation. A two-stage interactive object localization network is first designed, which aims to completely enclose the target of interest based on the supervision of object integrity (OI). Click centrality (CC) is also utilized to overcome the overlapping problem between objects. This coarse localization helps to reduce the search space and increase the focus of the click at a higher resolution. A principled multilayer segmentation network is then designed by a progressive layer-by-layer structure, which aims to accurately perceive the target with extremely limited prior guidance. A diffusion module is also designed to enhance the information flow between layers. Besides, the proposed model can be naturally extended to multiobject segmentation task. Our method achieves the state-of-the-art performance under one-click interaction on several benchmarks.

A Comparative Study of Various Interactive Image Segmentation Models

The goal of image segmentation is to divide an image into regions that are homogeneous with respect to certain characteristics, such as color, intensity, or texture. Interactive image segmentation (IIS) methods are usually evaluated in terms of segmentation performance vs. number of clicks (NoC). However, the automatic evaluation depends on a clicking procedure and its relation to the procedure used for training. In this work we compare qualitatively and quantitatively three state-of-the-art IIS methods that report the best performances but have not been compared against each other.

An interactive image segmentation method for the anatomical structures of the main olfactory bulb with micro-level resolution.

The main olfactory bulb is the key element of the olfactory pathway of rodents. To precisely dissect the neural pathway in the main olfactory bulb (MOB), it is necessary to construct the three-dimensional morphologies of the anatomical structures within it with micro-level resolution. However, the construction remains challenging due to the complicated shape of the anatomical structures in the main olfactory bulb and the high resolution of micro-optical images. To address these issues, we propose an interactive volume image segmentation method with micro-level resolution in the horizontal and axial direction. Firstly, we obtain the initial location of the anatomical structures by manual annotation and design a patch-based neural network to learn the complex texture feature of the anatomical structures. Then we randomly sample some patches to predict by the trained network and perform an annotation reconstruction based on intensity calculation to get the final location results of the anatomical structures. Our experiments were conducted using Nissl-stained brain images acquired by the Micro-optical sectioning tomography (MOST) system. Our method achieved a mean dice similarity coefficient (DSC) of 81.8% and obtain the best segmentation performance. At the same time, the experiment shows the three-dimensional morphology reconstruction results of the anatomical structures in the main olfactory bulb are smooth and consistent with their natural shapes, which addresses the possibility of constructing three-dimensional morphologies of the anatomical structures in the whole brain.

An Interactive Image Segmentation Method Based on Multi-Level Semantic Fusion.

Understanding and analyzing 2D/3D sensor data is crucial for a wide range of machine learning-based applications, including object detection, scene segmentation, and salient object detection. In this context, interactive object segmentation is a vital task in image editing and medical diagnosis, involving the accurate separation of the target object from its background based on user annotation information. However, existing interactive object segmentation methods struggle to effectively leverage such information to guide object-segmentation models. To address these challenges, this paper proposes an interactive image-segmentation technique for static images based on multi-level semantic fusion. Our method utilizes user-guidance information both inside and outside the target object to segment it from the static image, making it applicable to both 2D and 3D sensor data. The proposed method introduces a cross-stage feature aggregation module, enabling the effective propagation of multi-scale features from previous stages to the current stage. This mechanism prevents the loss of semantic information caused by multiple upsampling and downsampling of the network, allowing the current stage to make better use of semantic information from the previous stage. Additionally, we incorporate a feature channel attention mechanism to address the issue of rough network segmentation edges. This mechanism captures richer feature details from the feature channel level, leading to finer segmentation edges. In the experimental evaluation conducted on the PASCAL Visual Object Classes (VOC) 2012 dataset, our proposed interactive image segmentation method based on multi-level semantic fusion demonstrates an intersection over union (IOU) accuracy approximately 2.1% higher than the currently popular interactive image segmentation method in static images. The comparative analysis highlights the improved performance and effectiveness of our method. Furthermore, our method exhibits potential applications in various fields, including medical imaging and robotics. Its compatibility with other machine learning methods for visual semantic analysis allows for integration into existing workflows. These aspects emphasize the significance of our contributions in advancing interactive image-segmentation techniques and their practical utility in real-world applications.

Sequential interactive image segmentation

Interactive image segmentation (IIS) is an important technique for obtaining pixel-level annotations. In many cases, target objects share similar semantics. However, IIS methods neglect this connection and in particular the cues provided by representations of previously segmented objects, previous user interaction, and previous prediction masks, which can all provide suitable priors for the current annotation. In this paper, we formulate a sequential interactive image segmentation (SIIS) task for minimizing user interaction when segmenting sequences of related images, and we provide a practical approach to this task using two pertinent designs. The first is a novel interaction mode. When annotating a new sample, our method can automatically propose an initial click proposal based on previous annotation. This dramatically helps to reduce the interaction burden on the user. The second is an online optimization strategy, with the goal of providing semantic information when annotating specific targets, optimizing the model with dense supervision from previously labeled samples. Experiments demonstrate the effectiveness of regarding SIIS as a particular task, and our methods for addressing it.

A Novel Interactive Image Segmentation Algorithm Based on Maximization of Submodular Function

In this paper, an efficient interactive image segmentation method based on maximization of submodular function under user’s scribble constraint is presented. The problem of interactive image segmentation is formulated as the maximum entropy rate under user’s constraints. The objective function is submodular, and we solve the constrained submodular function maximization by incorporating a new data structure and some aggregating rules into the greedy algorithm. The main steps of our algorithm are as follows. First, the pixels scribbled by the user are clustered separately as target foreground and background clusters. Second, in the process of greedy algorithm, unscribbled pixels are aggregated to the corresponding target cluster according to the proposed aggregating rules. Finally, the segmentation result is presented by the two target clusters. The experiments and comparisons on three standard benchmarks show that our method has good performance. Our method is straightforward and efficient, and the time complexity of our method is between linear and polynomial. Furthermore, we analyze the influence of different scribbles, and propose some optimal scribble strategies.

An efficient interactive segmentation framework for medical images without pre-training.

Accurate and efficient medical image segmentation plays an important role in subsequent clinical applications such as diagnosis and surgical planning. This paper proposes an efficient interactive framework based on a graph convolutional network (GCN) for medical imagesegmentation. The initial segmentation results showed that a set of boundary control points can be generated for further interactive segmentation. We presented an adaptive interactive manner that allows the user to click on the boundary for fast interaction or drag the erroneous predicted control points for accurate correction. Furthermore, we proposed an interactive segmentation network (referred to as IVIF-GCN) to learn user experience in the interactive process by transforming interactive cues into annotations. In IVIF-GCN, a module of information fusion of image features and vertex position features (IVIF) is proposed to learn the location relationship between the current vertex and the neighboring vertices. Finally, the locations of control points around the interaction point is predicted and updatedautomatically. The proposed method achieves mean Dice of 96.6% and 91.3% on PROMISE12 and our in-house nasopharyngeal carcinoma (NPC) test sets, respectively. The experimental results showed that the proposed method outperforms the state-of-the-art segmentationmethods. The proposed interactive medical image segmentation method can efficiently improve segmentation results for clinical applications in the absence of training data. The GUI tool based on our method is available at https://github.com/Tian-lab/IGMedSeg.

Object Detection Based on the GrabCut Method for Automatic Mask Generation.

The Mask R-CNN-based object detection method is typically very time-consuming and laborious since it involves obtaining the required target object masks during training. Therefore, in order to automatically generate the image mask, we propose a GrabCut-based automated mask generation method for object detection. The proposed method consists of two stages. The first stage is based on GrabCut's interactive image segmentation method to generate the mask. The second stage is based on the object detection network of Mask R-CNN, which uses the mask from the previous stage together with the original input image and the associated label information for training. The Mask R-CNN model then automatically detects the relevant objects during testing. During experimentation with three objects from the Berkeley Instance Recognition Dataset, this method achieved a mean of average precision (mAP) value of over 95% for segmentation. The proposed method is simple and highly efficient in obtaining the mask of a segmented target object.

Rethinking interactive image segmentation: Feature space annotation

Despite the progress of interactive image segmentation methods, high-quality pixel-level annotation is still time-consuming and laborious — a bottleneck for several deep learning applications. We take a step back to propose interactive and simultaneous segment annotation from multiple images guided by feature space projection. This strategy is in stark contrast to existing interactive segmentation methodologies, which perform annotation in the image domain. We show that feature space annotation achieves competitive results with state-of-the-art methods in foreground segmentation datasets: iCoSeg, DAVIS, and Rooftop. Moreover, in the semantic segmentation context, it achieves 91.5% accuracy in the Cityscapes dataset, being 74.75 times faster than the original annotation procedure. Further, our contribution sheds light on a novel direction for interactive image annotation that can be integrated with existing methodologies. The supplementary material presents video demonstrations. Code available at https://github.com/LIDS-UNICAMP/rethinking-interactive-image-segmentation.

Integration of the saliency-based seeds generation and random walks with restart for image segmentation

In the task of interactive image segmentation, the user initially indicates some pixels as target seeds and background seeds and propagates the seeds’ labels to the rest of the image. We propose an equidistant seed generation method for interactive image segmentation that combines saliency detection and random walk (RW) with restart with optimal restarting probability. Our method includes two stages: first, we utilize the saliency detection method to obtain the initial target region and generate the initial seeds. Then we relabel the seeds using a Gaussian mixture model with limited user interaction to obtain accurate seeds. The relabeled seeds can limit the range of object; therefore, in the second stage, we can select an optimal restarting probability by calculating the area of the RW that exceeds the limited range of the object seeds. Our method can generate effective seeds and thus reduce the dependence on user interaction. Furthermore, using the optimal restarting probability, we can obtain higher segmentation accuracy. Extensive experiments on the Berkeley segmentation dataset, GrabCut dataset, and MSRA1000 dataset demonstrate that our method can reduce the dependence on user interaction and achieve better performance than the latest interactive image segmentation methods, such as RW, RW with restart, normalized RW, sub-Markov RW, and label propagation through complex networks.

Error-tolerant label prior for interactive image segmentation

User interactions are generally utilized to impose the hard constraint and estimate the label prior probability in the interactive image segmentation methods. The conventional interactive approaches cannot work well when the user inputs contain incorrect marks. The existing error tolerance methods mainly focus on the interaction itself and try to eliminate the influence of incorrect hard constraint, which however ignore the label prior estimation. This paper mainly focuses on solving the label prior estimation problem when error marks appear. The prior probability is generally defined as the matching degree between pixels and the cluster centers of marked pixels. Incorrect interaction leads to the formation of incorrect clusters. Therefore, a reliability learning model is constructed in this paper by 1) assigning smaller weighting factors to incorrect clusters, 2) assigning larger weighting factors to correct clusters with higher matching degree. Accurate label prior probability can be obtained by the weighted averaging. Then referring to the existing methods, an error-tolerant segmentation model is designed as a ratio energy function, which can both overcome the hard constraint and the label prior limitation with error marks. Experiments on the challenging Berkeley segmentation data set and Microsoft GrabCut database demonstrate the effectiveness of the proposed method.

Saliency-guided automatic detection and segmentation of tumor in breast ultrasound images

The development of computer aided diagnosis (CAD) systems for the task of tumor detection and segmentation in breast ultrasound (BUS) images presents one of the most active research fields. In this paper, a saliency-guided approach for fast and automatic tumor segmentation in BUS images is proposed. We explore the ability of the concept of saliency to emulate radiologist expertise for tumor lesion detection in BUS images. For that, we compute BUS image saliency to generate effective tumor seeds and background seeds. Then, a graph-based interactive image segmentation method is applied automatically using the generated seeds to extract tumor region. For more accurate segmentation, we propose a novel saliency score measure to apply a post-processing step for result refinement. Our results have been compared with other approaches in challenged datasets and demonstrated the effectiveness of our saliency-guided approach for tumor detection and segmentation using different evaluation metrics.

Data visualization and data fusion on the visual performance of illustration

The influence of COVID-19 causes a certain impact on data visualization and data fusion on the visual performance of illustration. Based on the development of illustration, this paper discusses the relationship between illustration, text information and media. This paper studies the feasibility of the combination of illustration and information visualization. In this paper, the interactive image segmentation and gridding methods are proposed. Then, the background theory and significance of flow field design are described, and the flow field generation method based on heat source diffusion is proposed. In this paper, the shadow of the topology of the convective field through the interaction input of the flow field design is analyzed, and then compared with the related work. In the visualization of flow field, based on the weighted distance field formed by the diffusion of heat source, a visualization method of stratified flow field line is proposed. Finally, the visualization method of stratified flow field is explained and its effect is demonstrated. Experimental data show that the information visualization method proposed in this paper can improve the efficiency and accuracy of illustration information extraction.

Weld pool image acquisition and contour extraction based on arc spectrum and camera quantum efficiency

Based on the arc spectral radiation characteristics of ZL201A aluminum ally in CMT + P process and camera’s quantum efficiency, we select 520 nm band-stop as the working band for weld pool imaging acquisition systems and obtain high signal-to-noise ratio aluminum alloy weld pool images at the base level of weld current. Through analyzing the characteristics of captured aluminum alloy weld pool image, a semi-supervised and adaptive contour extraction method is designed, called Maximal Similarity based Region Merging and Adaptive Matting (MSRM-AM). Firstly, an interactive image segmentation method based on super-pixel is used to obtain the initial contour, and then the initial contour is finely segmented to get the final segmentation result. Comparing with other contour extraction algorithms, the proposed algorithm has higher accuracy and reliability, and has verified through abundant experiments.

Supervised image segmentation based on superpixel and improved normalised cuts

Effective and efficient image segmentation is an important task in computer vision. As the full-automatic image segmentation is usually difficult to segment the natural image, it is an excellent solution to use interactive schemes. Here, to overcome the defects of SSNCut in its low quality and speed, the authors proposed a new interactive image segmentation method based on superpixels, must-link, cannot-link constraints and improved normalised cuts. The main contribution of their work is as follows: first, the similarity between two superpixel regions is calculated using Bhattacharyya distance. Second, they adaptively modify the weights of must-link and cannot-link constraints. Compared to SSNCut, their method greatly improves the accuracy of segmentation. Comparative experiments on open datasets show that the proposed method can get better results compared with SSNCut, GrabCut in one cut, interactive segmentation using binary partition tree, interactive graph cut, seed region growing, and simple interactive object extraction.

Interactive image segmentation using geodesic appearance overlap graph cut

Image segmentation is a fundamental step in many applications such as image editing, medical image analysis and processing. It is quite common to use graph cuts for image segmentation in recent years. Tang, Gorelick, Veksler, and Boykov proposed a new image segmentation model which uses the appearance overlap on unnormalized histograms and graph cut framework. Their model is highly effective for interactive segmentation but is prone to isolated points. To avoid that problem, we propose an effective interactive image segmentation method, that is appropriately incorporating geodesic distance information, appearance overlap information, and edge information together into the well-known graph-cut framework. Rather than a simple union of these information, the respective strengths of each information term can be tuned adaptively in our method. We utilize the user’s scribbles to obtain the estimated foreground/background color models via fast kernel density estimation, and then get the appearance overlap intricateness according to the inferred color models. By taking comprehensive advantage of the geodesic distance and the global appearance overlap color clues, our method requires less user effort and achieves higher accuracy of segmentation than the latest interactive segmentation techniques, such as Geodesic Graph Cut, GrabCut in One Cut, Semi-Supervised Normalized Cuts, and Convexity Shape Prior for Binary Segmentation, as shown in our experiments.

A fully convolutional two-stream fusion network for interactive image segmentation

In this paper, we propose a novel fully convolutional two-stream fusion network (FCTSFN) for interactiveimage segmentation. The proposed network includes two sub-networks: a two-stream late fusion network (TSLFN) that predicts the foreground at a reduced resolution, and a multi-scale refining network (MSRN) that refines the foreground at full resolution. The TSLFN includes two distinct deep streams followed by a fusion network. The intuition is that, since user interactions are more direct information on foreground/background than the image itself, the two-stream structure of the TSLFN reduces the number of layers between the pure user interaction features and the network output, allowing the user interactions to have a more direct impact on the segmentation result. The MSRN fuses the features from different layers of TSLFN with different scales, in order to seek the local to global information on the foreground to refine the segmentation result at full resolution. We conduct comprehensive experiments on four benchmark datasets. The results show that the proposed network achieves competitive performance compared to current state-of-the-art interactive image segmentation methods. 1

Global graph diffusion for interactive object extraction

Many interactive image segmentation methods design energy functions based on the local relationship of neighboring pixels, which is insufficient at capturing the abundant information of an image, and these methods are susceptible to many problems, such as sensitivity to seeds and under-segmentation of boundaries. To solve these problems, this paper explores utilizing the global relationship for interactive image segmentation. To effectively obtain the global information of the image, we first propose a robust affinity diffusion (RAD) method to propagate the local affinity graph. Compared with the existing diffusion-based approaches, the advantage of RAD is that it can converge to an effective limit value, which makes the diffusion process more computationally efficient and easier to control. The segmentation model is then constructed based on this convergent global graph. To efficiently utilize the global information, the energy function is designed by the multiplication of the global affinity matrix and a prior probability vector. The use of global information can significantly improve the segmentation performance. Experiments on challenging data sets demonstrate that RAD can obtain better results than state-of-the-art methods.

Diffusive likelihood for interactive image segmentation

The performance of conventional interactive image segmentation methods is strongly affected by seed quantity and position, and it is difficult for them to maintain global data coherence due to the bias that is caused by limited interactions. Furthermore, the pixel-level relationships in these methods are too local to capture long-range connectivity cues, which often causes them to obtain under-segmented results. To solve these problems, this paper proposes an interactive segmentation method that is based on likelihood diffusion and perceptual learning. The diffusive likelihood strategy is proposed for accurately estimating the prior label probability from limited user inputs. Superpixel-level grouping cues are utilized to enforce continuity during the segmentation process. The geometrical adjacency and long-range grouping cues are fused in the proposed framework to ensure that the segmentation results maintain proximity and continuity. The final results can be obtained by applying a joint optimization technique to solve a pair of sub-module functions. Experiments on the Berkeley segmentation data set and the Microsoft GrabCut database demonstrate that the proposed method outperforms state-of-the-art methods.

Interactive image segmentation with a regression based ensemble learning paradigm

To achieve fine segmentation of complex natural images, people often resort to an interactive segmentation paradigm, since fully automatic methods often fail to obtain a result consistent with the ground truth. However, when the foreground and background share some similar areas in color, the fine segmentation result of conventional interactive methods usually relies on the increase of manual labels. This paper presents a novel interactive image segmentation method via a regression-based ensemble model with semi-supervised learning. The task is formulated as a non-linear problem integrating two complementary spline regressors and strengthening the robustness of each regressor via semi-supervised learning. First, two spline regressors with a complementary nature are constructed based on multivariate adaptive regression splines (MARS) and smooth thin plate spline regression (TPSR). Then, a regressor boosting method based on a clustering hypothesis and semi-supervised learning is proposed to assist the training of MARS and TPSR by using the region segmentation information contained in unlabeled pixels. Next, a support vector regression (SVR) based decision fusion model is adopted to integrate the results of MARS and TPSR. Finally, the GraphCut is introduced and combined with the SVR ensemble results to achieve image segmentation. Extensive experimental results on benchmark datasets of BSDS500 and Pascal VOC have demonstrated the effectiveness of our method, and the comparison with experiment results has validated that the proposed method is comparable with the state-of-the-art methods for interactive natural image segmentation.