Interactive Segmentation Algorithm Research Articles

Clustering-based interactive image segmentation

Objectives. The task of color image segmentation without the use of preliminary training is considered. It arises, for example, when it is necessary to perform image segmentation with semantic and color properties unknown in advance immediately after their acquisition, or when the set of images intended for segmentation is too small, as well as when performing preliminary "exploratory" analysis of images. In such cases, powerful neural network and other segmentation tools that require deep learning can not be used.Methods. An algorithm for interactive image segmentation is proposed, based on the analysis of the colors of areas selected interactively. First, in interactive mode, the image areas belonging to the objects are selected very approximately, and then regions belonging to the background are chosen. In the next step, the set of colors of the selected object areas and the set of colors of the selected background areas are clustered separately by one of the clustering algorithms, for example, k-means, fuzzy c-means, or the multi-level clustering algorithm proposed by the author. After this, non-informative elements are removed from the set of cluster centers describing the objects and the set of clusters presenting the background. The modified sets of object and background cluster centers are used for image segmentation.Results . The constructed algorithm allows selection of the required objects in color images if the colors of the objects and the background are different. Interactive selection of object areas and background areas does not require accuracy or much effort and usually takes several tens of seconds. For selection, rectangular areas that lie entirely inside the object images, and rectangular areas that belong completely to the background can be used. Below an example of interactive regions selection and color image segmentation is shown.Conclusion. The experiments performed showed the effectiveness of the proposed approach to segmenting color images. It can be used in cases where the semantic and color properties of images are not known in advance, and in cases where the use of more powerful deep learning methods, including neural networks, is too expensive or impossible.

Cartoon copyright recognition method based on character personality action

Aiming at the problem of cartoon piracy and plagiarism, this paper proposes a method of cartoon copyright recognition based on character personality actions. This method can be used to compare the original cartoon actions with the action characteristics of pirated or copied cartoons to identify whether there is piracy or plagiarism. Firstly, an image preprocessing scheme for character extraction is designed. GrabCut interactive image segmentation algorithm was used to obtain cartoon characters, and then binarization and morphological processing were performed on the results. Secondly, a feature extraction scheme based on character profile, moving character and character pose is designed. By extracting the perimeter and area of the character contour, the length-to-width ratio of the smallest rectangle and the inclination angle of the contour, the character contour features are obtained. The three-dimensional coordinates are established by the central point position of the cartoon character in the two-dimensional image and the change of the character's zoom in and out, and the character's motion angle characteristics are calculated. By skeletonizing a character to obtain the pose characteristics, and using deburring operation to remove redundant branches, then extract the skeleton joint angle information. Finally, feature fusion is performed on the extracted features. The experimental results show that the proposed method breaks the limitation of the conventional single feature based recognition, and can better extract the character features including contour feature, motion feature and pose feature through multi-feature based extraction, so as to protect the cartoon copyright.

Abstract 2595: Accelerated and precise tumor segmentation in NSCLC: A comparative analysis of automated ClickSeg and manual annotation for radiomics

Abstract Background: Radiomics models utilizing artificial intelligence are being explored as a potential biomarker in the field of oncology. Radiomics analysis requires segmentations of radiographic imaging. However, manual segmentation is a labor-intensive process that is time consuming, and acts as a major rate-limiting step. Thus the development of automated segmentation tools presents an opportunity for innovation in regards to efficiency and precision. Our study aims to explore the feasibility of autosegmentation in comparison with manual segmentation. Methods: A cohort of 105 stage III-IV non-small cell lung cancer patients receiving immunotherapy was examined, with a total of 168 lesions. We conducted a comprehensive comparative analysis of manual versus automated segmentation in time efficiency and segmentation quality, utilizing CT scans from the patients. Manual segmentation was performed by three physicians annotating in consensus using LIFEx software v7.3.0 (IMIV/CEA, Orsay, France). For a subset of randomly selected patients, annotation time was recorded as a reference point for the automated segmentation. In addition, automated segmentation was performed on all 168 lesions using ClickSeg from PictureHealth, a click-based interactive segmentation algorithm optimized for NSCLC lesions. Segmentation quality was evaluated using the Dice similarity coefficient to measure the concordance between ClickSeg outputs and consensus expert annotations. Results: Average manual annotation time of the tumor region was 15min 34sec, and ranged from 1min 2sec to 53min 12sec.. Average prediction time for the automated segmentation model was 1.94 seconds, and ranged from 0.71 seconds to 4.39 seconds. The median Dice coefficient, reflecting segmentation accuracy, was 0.72, highlighting the robust performance of the automated method. Conclusion: Our study demonstrates the efficient and accurate nature of automated segmentation supporting its potential integration into routine radiomic analysis procedures, streamlining the assessment of treatment response and ultimately improving patient care. Citation Format: Seyoung Lee, Kai Zhang, Jeeyeon Lee, Peter Haseok Kim, Amogh Hiremath, Salie Lee, Monica Yadav, Maria J. Chuchuca, Taegyu Um, Myungwoo Nam, Liam Il-Young Chung, Hye Sung Kim, Jisang Yu, Trie Arni Djunadi, Leeseul Kim, Youjin Oh, Sungmi Yoon, Zunairah Shah, Yuchan Kim, Ilene Hong, Grace Kang, Jessica Jang, Amy Cho, Soowon Lee, Cecilia Nam, Timothy Hong, Yuri S. Velichko, Anant Madabhushi, Nathaniel Braman, Young Kwang Chae. Accelerated and precise tumor segmentation in NSCLC: A comparative analysis of automated ClickSeg and manual annotation for radiomics [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 2595.

Interactive segmentation of medical images using deep learning.

Medical image segmentation algorithms based on deep learning have achieved good segmentation results in recent years, but they require a large amount of labeled data. When performing pixel-level labeling on medical images, labeling a target requires marking ten or even hundreds of points along its edge, which requires a lot of time and labor costs. To reduce the labeling cost, we utilize a click-based interactive segmentation method to generate high-quality segmentation labels. However, in current interactive segmentation algorithms, only the interaction information clicked by the user and the image features are fused as the input of the backbone network (so-called early fusion). The early fusion method has the problem that the interactive information is much sparse at this time. Furthermore, the interactive segmentation algorithms do not take into account the boundary problem, resulting in poor model performance. So we propose early fusion and late fusion strategy to prevent the interaction information from being diluted prematurely and make better use of the interaction information. At the same time, we propose a decoupled head structure, by extracting the image boundary information, and combining the boundary loss function to establish the boundary constraint term, so that the network can pay more attention to the boundary information and further improve the performance of the network. Finally, we conduct experiments on three medical datasets (Chaos, VerSe and Uterine Myoma MRI) to verify the effectiveness of our network. The experimental results show that our network greatly improved compared with the baseline, and NoC@80(the number of interactive clicks over 80% of the IoU threshold) improved by 0.1, 0.1, and 0.2. In particular, we have achieved a NoC@80 score of 1.69 on Chaos. According to statistics, manual annotation takes 25 min to label a case(Uterine Myoma MRI). Annotating a medical image with our method can be done in only 2 or 3 clicks, which can save more than 50% of the cost.

Parallel Processing of Image Databases for Accelerated Morphological Analysis

Nuclear forensics relies on different signatures to identify the source of nuclear material. Such signatures include crystalline structure, chemical composition, and particle morphology. One way to quantify morphology in electron microscope imagery is through image segmentation, where pixels are assigned to several partitions (or groups) that correspond to particles, grains, and other objects of interest within the image. Once pixels are assigned to segments, it is relatively straightforward to quantify other quantities of interest, such as grain size, circularity, etc. However, the range and diversity of microscope images make it difficult to obtain an accurate segmentation automatically. The accuracy of segmentation can be improved through supervised learning, but this requires many images to be manually segmented. Another way to improve the accuracy is to use interactive segmentation. Interactive segmentation requires a human to provide image-specific user input to improve performance. However, the amount of user input (effort) is generally far less than is required for supervised learning. In this paper, we investigate several parallelization strategies to automatically explore the user input parameter space of interactive segmentation algorithms across a large number of images. Specifically, we investigate four different parallelization mechanisms in a high-performance computing (HPC) environment and use the Amdahl fraction to evaluate efficiency on multiple processor cores across multiple nodes. Ultimately, the parallelization strategy that was most efficient utilized the message passing interface integrated with the segmentation and quantification code. This strategy had an Amdahl fraction of 0.985 and a performance of about 0.251 s/image. These results indicate that the parameter space of interactive segmentation algorithms can be efficiently explored using HPC. This opens the door to future work where user input is reduced and where interactive image segmentation algorithms are automatically applied to large image sets.

Geodesic Models With Convexity Shape Prior.

The minimal geodesic models established upon the eikonal equation framework are capable of finding suitable solutions in various image segmentation scenarios. Existing geodesic-based segmentation approaches usually exploit image features in conjunction with geometric regularization terms, such as euclidean curve length or curvature-penalized length, for computing geodesic curves. In this paper, we take into account a more complicated problem: finding curvature-penalized geodesic paths with a convexity shape prior. We establish new geodesic models relying on the strategy of orientation-lifting, by which a planar curve can be mapped to an high-dimensional orientation-dependent space. The convexity shape prior serves as a constraint for the construction of local geodesic metrics encoding a particular curvature constraint. Then the geodesic distances and the corresponding closed geodesic paths in the orientation-lifted space can be efficiently computed through state-of-the-art Hamiltonian fast marching method. In addition, we apply the proposed geodesic models to the active contours, leading to efficient interactive image segmentation algorithms that preserve the advantages of convexity shape prior and curvature penalization.

A Novel Interactive Image Segmentation Algorithm Based on Maximization of Submodular Function

In this paper, an efficient interactive image segmentation method based on maximization of submodular function under user’s scribble constraint is presented. The problem of interactive image segmentation is formulated as the maximum entropy rate under user’s constraints. The objective function is submodular, and we solve the constrained submodular function maximization by incorporating a new data structure and some aggregating rules into the greedy algorithm. The main steps of our algorithm are as follows. First, the pixels scribbled by the user are clustered separately as target foreground and background clusters. Second, in the process of greedy algorithm, unscribbled pixels are aggregated to the corresponding target cluster according to the proposed aggregating rules. Finally, the segmentation result is presented by the two target clusters. The experiments and comparisons on three standard benchmarks show that our method has good performance. Our method is straightforward and efficient, and the time complexity of our method is between linear and polynomial. Furthermore, we analyze the influence of different scribbles, and propose some optimal scribble strategies.

Design and Implementation of Print Advertisement Based on Computer Image Interactive Virtualization

This study aimed to further enhance and enrich users’ visual experience, provide users with a more infectious image sense, and further promote the development of multidimensional, virtualized, and interactive graphic advertising design applications. This study presents a design idea of print advertisement based on computer image interactive virtual technology. Based on the in-depth analysis of the multidimensional expression of plane image interactive virtualization, this idea carries out interactive virtual design with the help of interactive virtual image segmentation algorithm and plane advertising image element optimization technology. Through the test of image interactive edge extraction function on MATLAB platform, the simulation results show that the edge width of single side extraction results with different gray levels of the same width and different widths of the same gray level are 9 pixels, the gray values are 5a0, 110, and 200, respectively, and the edge width corresponds to 3, 9, and 15 pixels, respectively. The results show that the system can effectively enhance the image definition and obtain better print advertising design effect.

ScribbleNet: Efficient interactive annotation of urban city scenes for semantic segmentation

Annotation is a crucial first step in the semantic segmentation of urban images that facilitates the development of autonomous navigation systems. However, annotating complex urban images is time-consuming and challenging. It requires significant human effort making it expensive and error-prone. To reduce human effort during annotation, multiple images need to be annotated in a short time-span. In this paper, we introduce ScribbleNet, an interactive image segmentation algorithm to address this issue. Our approach provides users with a pre-segmented image that iteratively improves the segmentation using scribble as an annotation input. This method is based on conditional inference and exploits the learnt correlations in a deep neural network (DNN). ScribbleNet can: (1) work with urban city scenes captured in unseen environments, (2) annotate new classes not present in the training set, and (3) correct several labels at once. We compare this method with other interactive segmentation approaches on multiple datasets such as CityScapes, BDD, Mapillary Vistas, KITTI, and IDD. ScribbleNet reduces the annotation time of an image by up to 14.7 × over manual annotation and up to 5.4× over the current approaches. The algorithm is integrated into the publicly available LabelMe image annotation tool and will be released as an open-source software.

Effects of Different Superpixel Algorithms on Interactive Segmentations

Semi-automated segmentation or more commonly known as interactive image segmentation is an algorithm that extracts a region of interest (ROI) from an image based on the input information from the user. The said algorithm will be repetitively fed with such input information until required region of interest is successfully segmented. To accelerate this segmentation procedure as well as enhancing the result, pre-processing steps can be applied. The application of superpixel is an example of such pre-processing step. Superpixel can be defined as a collection of pixels that share common features such as texture and colours. Though employed as pre-processing step in many interactive segmentation algorithms, to date, no study has been conducted to assess the effects of such incorporations on the segmentation algorithms. Thus, this study aims to address this issue. In this study, five different types of superpixels ranging from watershed, density, graph, clustering and energy optimization categories are evaluated. The superpixels generated by these five algorithms will be used on two interactive image segmentation algorithms: i) Maximal Similarity based Region Merging (MSRM) and ii) Graph-Based Manifold Ranking (GBMR) with single and multiple strokes on various images from the Berkeley image dataset. The result of testing had shown that MSRM achieved better result compared to GBMR in both single and multiple input strokes using SEEDS superpixel algorithm. This study summary concluded that at different superpixel algorithms produced different results and that it is not possible to single out one particular superpixel algorithm that can work well for all the interactive segmentation algorithms. As such, the key to achieving a decent segmentation result lies in choosing the right superpixel algorithms for a given interactive segmentation algorithm.

Robust High-Throughput Phenotyping with Deep Segmentation Enabled by a Web-Based Annotator.

The abilities of plant biologists and breeders to characterize the genetic basis of physiological traits are limited by their abilities to obtain quantitative data representing precise details of trait variation, and particularly to collect this data at a high-throughput scale with low cost. Although deep learning methods have demonstrated unprecedented potential to automate plant phenotyping, these methods commonly rely on large training sets that can be time-consuming to generate. Intelligent algorithms have therefore been proposed to enhance the productivity of these annotations and reduce human efforts. We propose a high-throughput phenotyping system which features a Graphical User Interface (GUI) and a novel interactive segmentation algorithm: Semantic-Guided Interactive Object Segmentation (SGIOS). By providing a user-friendly interface and intelligent assistance with annotation, this system offers potential to streamline and accelerate the generation of training sets, reducing the effort required by the user. Our evaluation shows that our proposed SGIOS model requires fewer user inputs compared to the state-of-art models for interactive segmentation. As a case study of the use of the GUI applied for genetic discovery in plants, we present an example of results from a preliminary genome-wide association study (GWAS) of in planta regeneration in Populus trichocarpa (poplar). We further demonstrate that the inclusion of a semantic prior map with SGIOS can accelerate the training process for future GWAS, using a sample of a dataset extracted from a poplar GWAS of in vitro regeneration. The capabilities of our phenotyping system surpass those of unassisted humans to rapidly and precisely phenotype our traits of interest. The scalability of this system enables large-scale phenomic screens that would otherwise be time-prohibitive, thereby providing increased power for GWAS, mutant screens, and other studies relying on large sample sizes to characterize the genetic basis of trait variation. Our user-friendly system can be used by researchers lacking a computational background, thus helping to democratize the use of deep segmentation as a tool for plant phenotyping.

Superpixel Sizes using Topology Preserved Regular Superpixel Algorithm and their Impacts on Interactive Segmentation

Interactive Image Segmentation is a type of semi-automated segmentation that uses user input to extract the object of interest. It is possible to speed up and improve the end result of segmentation by using pre-processing steps. The use of superpixels is an example of a pre-processing step. A superpixel is a collection of pixels with similar properties such as texture and colour. Previous research was conducted to assess the impact of the number of superpixels (based on SEEDS superpixel aglorithms) required to achieve the best segmentation results. The study, however, only examined one type of input (strokes) and a small number of images. As a result, the goal of this study is to extend previous work by performing interactive segmentation with input strokes and a combination of bounding box and strokes on images from Grabcut image data sets generated by Topology preserved regular superpixel (TPRS). Based on our findings, an image with 1000 to 2500 superpixels and a combination of bounding box and strokes will help the interactive segmentation algorithm produce a good segmentation result. Finally, the size of the superpixels would influence the final segmentation results as well as the input type.

Residual refinement for interactive skin lesion segmentation

BackgroundImage segmentation is a difficult and classic problem. It has a wide range of applications, one of which is skin lesion segmentation. Numerous researchers have made great efforts to tackle the problem, yet there is still no universal method in various application domains.ResultsWe propose a novel approach that combines a deep convolutional neural network with a grabcut-like user interaction to tackle the interactive skin lesion segmentation problem. Slightly deviating from grabcut user interaction, our method uses boxes and clicks. In addition, contrary to existing interactive segmentation algorithms that combine the initial segmentation task with the following refinement task, we explicitly separate these tasks by designing individual sub-networks. One network is SBox-Net, and the other is Click-Net. SBox-Net is a full-fledged segmentation network that is built upon a pre-trained, state-of-the-art segmentation model, while Click-Net is a simple yet powerful network that combines feature maps extracted from SBox-Net and user clicks to residually refine the mistakes made by SBox-Net. Extensive experiments on two public datasets, PH2 and ISIC, confirm the effectiveness of our approach.ConclusionsWe present an interactive two-stage pipeline method for skin lesion segmentation, which was demonstrated to be effective in comprehensive experiments.

Interactive Liver Segmentation Algorithm Based on Geodesic Distance and V-Net

Interactive Liver Segmentation Algorithm Based on Geodesic Distance and V-Net

FAST INTERACTIVE REGIONAL PATTERN MERGING FOR GENERIC TISSUE SEGMENTATION IN HISTOPATHOLOGY IMAGES

The image segmentation of histopathological tissue images has always been a challenge due to the overlapping of tissue color distributions, the complexity of extracellular texture and the large image size. In this paper, we introduce a new region-merging algorithm, namely, the Regional Pattern Merging (RPM) for interactive color image segmentation and annotation, by efficiently retrieving and applying the user’s prior knowledge of stroke-based interaction. Low-level color/texture features of each region are used to compose a regional pattern adapted to differentiating a foreground object from the background scene. This iterative region-merging is based on a modified Region Adjacency Graph (RAG) model built from initial segmented results of the mean shift to speed up the merging process. The foreground region of interest (ROI) is segmented by the reduction of the background region and discrimination of uncertain regions. We then compare our method against state-of-the-art interactive image segmentation algorithms in both natural images and histological images. Taking into account the homogeneity of both color and texture, the resulting semi-supervised classification and interactive segmentation capture histological structures more completely than other intensity or color-based methods. Experimental results show that the merging of the RAG model runs in a linear time according to the number of graph edges, which is essentially faster than both traditional graph-based and region-based methods.

Real-time Quantitative Visual Inspection using Extended Reality


 In this study, we propose a technique for quantitative visual inspection that can quantify structural damage using extended reality (XR). The XR headset can display and overlay graphical information on the physical space and process the data from the built-in camera and depth sensor. Also, the device permits accessing and analyzing image and video stream in real-time and utilizing 3D meshes of the environment and camera pose information. By leveraging these features for the XR headset, we build a workflow and graphic interface to capture the images, segment damage regions, and evaluate the physical size of damage. A deep learning-based interactive segmentation algorithm called f-BRS was deployed to precisely segment damage regions through the XR headset. A ray-casting algorithm is implemented to obtain 3D locations corresponding to the pixel locations of the damage region on the image. The size of the damage region is computed from the 3D locations of its boundary. The performance of the proposed method is demonstrated through a field experiment at an in-service bridge where spalling damage is present at its abutment. The experiment shows that the proposed method provides sub-centimeter accuracy for the size estimation.

Adaptive fusion with multi-scale features for interactive image segmentation

Multi-scale features are usually utilized to improve the performance of interactive image segmentation, however, they have varying leverages over the result of segmentation, for example, thinner segmentation results could be achieved by pixel-level features, but sensitive to image noise, and superpixel-level features could provide the semantic perception of the object, but easily lead to over-segmentations. Therefore, we propose an interactive image segmentation algorithm by adaptive fusion with multi-scale features (AFMSF). It intends on combining the multi-scale information adaptively for the segmentation via learning the influence coefficients of multi-scale features. First, multi-scale superpixel layers are generated by controlling the size of superpixels. Based on features of this multi-scale information, the similarity matrices and label priors with pixel-superpixel levels are then obtained. A fusion with diffusion strategy is designed to build the energy function by combining these multi-cues. Finally, the influence coefficient of each scale and the labeling are updated with each other until convergence. The algorithm we proposed is robust to diverse circumstances of objects, the experimental results on public interactive image segmentation datasets Graz, LHI, and MSRC validate the superior performance of the proposed method.

Global Manifold Learning for Interactive Image Segmentation

This paper presents an interactive image segmen-tation algorithm, in which the segmentation problem is formulated as a global manifold learning process. Based on the principle that the label of each element depends on the influence of all the elements in the image, we extend the conventional local neighborhood or the long range regional relationships to the global relationships over the whole image. A probabilistic framework is established to measure the global effect of each element on all other elements. Based on two different manifold learning styles, the semi-global manifold learning (SGML) and the fully-global manifold learning (FGML) algorithms are proposed to capture the global geometry structure of the data manifold. SGML learns the intrinsic effects of each element separately, equivalent to a matrix diffusion process on an affinity graph. FGML learns the intrinsic effects of all elements together, equivalent to a label pair diffusion process on a higher-order tensor product graph. The global manifold learning helps to overcome the low contrast, weak boundary and texture problems. Extensive experiments on three public interactive segmentation datasets demonstrate the superior performance of the proposed algorithms both in accuracy and efficiency.

Feasibility of Automatic Seed Generation Applied to Cardiac MRI Image Analysis

We present a method of using interactive image segmentation algorithms to reduce specific image segmentation problems to the task of finding small sets of pixels identifying the regions of interest. To this end, we empirically show the feasibility of automatically generating seeds for GrowCut, a popular interactive image segmentation algorithm. The principal contribution of our paper is the proposal of a method for automating the seed generation method for the task of whole-heart segmentation of MRI scans, which achieves competitive unsupervised results (0.76 Dice on the MMWHS dataset). Moreover, we show that segmentation performance is robust to seeds with imperfect precision, suggesting that GrowCut-like algorithms can be applied to medical imaging tasks with little modeling effort.

SwipeCut: Interactive Segmentation via Seed Grouping

Interactive image segmentation algorithms rely on the user to provide annotations as the guidance. When the task of interactive segmentation is performed on a small touchscreen device, the requirement of providing precise annotations could be cumbersome to the user. We design a new interaction mechanism that actively queries seeds for guiding the user to label. Our method enforces sparsity and diversity criteria on the selection of query seeds, and at each round of interaction, the user is only presented with a small number of informative query seeds that are far apart from each other. Therefore, the user merely has to swipe through on the ROI-relevant query seeds for checking which ones of the query seeds are inside the region of interest (ROI). This kind of interaction should be easy since those gestures are commonly used on a touchscreen. As a result, we are able to derive a user-friendly interaction mechanism for annotation on small touchscreen devices. The performance of our algorithm is evaluated on six publicly available datasets. The evaluation results show that our algorithm achieves high segmentation accuracy, with short computational time and less user feedback.