Interactive Segmentation Research Articles

Oral leukoplakia evaluation through clinical photography: classification, interactive segmentation, and automated binarization before going on Artificial Intelligence algorithms

Oral leukoplakia evaluation through clinical photography: classification, interactive segmentation, and automated binarization before going on Artificial Intelligence algorithms

Coarse Mask Guided Interactive Object Segmentation.

Interactive object segmentation aims to produce object masks with user interactions, such as clicks, bounding boxes, and scribbles. Click point is the most popular interactive cue for its efficiency, and related deep learning methods have attracted lots of interest in recent years. Most works encode click points as gaussian maps and concatenate them with images as the model's input. However, the spatial and semantic information of gaussian maps would be noised through multiple convolution layers and won't be fully exploited by top layers for mask prediction. To pass click information to top layers exactly and efficiently, we propose a coarse mask guided model (CMG) which predicts coarse masks with a coarse module to guide the object mask prediction. Specifically, the coarse module encodes user clicks as query features and enriches their semantic information with backbone features through transformer layers, coarse masks are generated based on the enriched query feature and fed into CMG's decoder. Benefiting from the efficiency of transformer, CMG's coarse module and decoder module are lightweight and computationally efficient, making the interaction process more smooth. Experiments on several segmentation benchmarks demonstrate the effectiveness of our method, and we get new state-of-the-art results compared with previous works.

Combing experimental methods and molecular simulations to study self-healing behaviors of polyurethane elastomers containing multiple hydrogen bond networks and flexible blocks.

The preparation of polymers with high self-healing ability is conducive to environmental protection and resource conservation. In the present work, two kinds of polyurethane (PU) elastomers were prepared: the one containing flexible end blocks (polypropylene glycol) and the other containing flexible end blocks and 2-ureido-4[1H]-pyrimidinone (UPy) groups that can form reversible quadruple hydrogen bonds. Both of the two PU elastomers have self-healing ability. At low temperatures the PU without UPy groups exhibits stronger self-healing ability, while at high temperatures the PU with UPy groups has better self-healing function. The difference can be attributed to the combined effect of segmental mobility and reversible network strength. Based on molecular simulations, we further observed that the self-healing behaviors are affected by four factors: healing temperature, reversible interaction strength, reversible interaction site density and segment diffusion ability.

Interactive Image Segmentation Using Superpixels and Deep Metric Learning for Tool Condition Monitoring

Interactive Image Segmentation Using Superpixels and Deep Metric Learning for Tool Condition Monitoring

Interactive Image Segmentation Technique Based on Improved Residual Network

Interactive Image Segmentation Technique Based on Improved Residual Network

Clicking Matters: Towards Interactive Human Parsing

In this work, we focus on Interactive Human Parsing (IHP), which aims to segment a human image into multiple human body parts with guidance from users' interactions. This new task inherits the class-aware property of human parsing, which cannot be well solved by traditional interactive image segmentation approaches that are generally class-agnostic. To tackle this new task, we first exploit user clicks to identify different human parts in the given image. These clicks are subsequently transformed into semantic-aware localization maps, which are concatenated with the RGB image to form the input of the segmentation network and generate the initial parsing result. To enable the network to better perceive user's purpose during the correction process, we investigate several principal ways for the refinement, and reveal that random-sampling-based click augmentation is the best way for promoting the correction effectiveness. Furthermore, we also propose a semantic-perceiving loss (SP-loss) to augment the training, which can effectively exploit the semantic relationships of clicks for better optimization. To the best knowledge, this work is the first attempt to tackle the human parsing task under the interactive setting. Our IHP solution achieves 85\% mIoU on the benchmark LIP, 80\% mIoU on PASCAL-Person-Part and CIHP, 75\% mIoU on Helen with only 1.95, 3.02, 2.84 and 1.09 clicks per class respectively. These results demonstrate that we can simply acquire high-quality human parsing masks with only a few human effort. We hope this work can motivate more researchers to develop data-efficient solutions to IHP in the future.

A Click-Based Interactive Segmentation Network for Point Clouds

A Click-Based Interactive Segmentation Network for Point Clouds

A model for nano-scale spherical surface coverage and protein corona formation by amyloidogenic peptides

The conformation of three amyloidogenic peptides; amyloid beta 1−40 (Aβ1−40), alpha synuclein (α-syn), and beta2-microglobulin (β2m) are closely associated with the process of causing neurodegenerative diseases. Use of a peptide adsorbed gold nano-particle system allowed us to investigate the interactive segment of each peptide responsible for peptide-peptide networking, which is crucial to initiate the formation of an oligomer and leads to fibrillogenesis. The adsorption orientation of the amyloidogenic peptides on the nano-gold colloid spherical surface was explained by simulating how much area of the metal surface was covered by the peptides, i.e., coverage ratio, 2. The empirically extracted 2 was explained by the summation of the 1st layer and the 2nd layer with a spikingout orientation of the prolate. Of note, the involvement of the 2nd layer was peptide type dependent. The nanosize dependence of 2 was linearly correlated with available spacing between adjacent peptides, Sd, which were approximated as prolates, For Aβ1−40 and α-syn, the 2nd layer was more included as Sd increased. In contrast, β2m was found to gyrate over the gold surface as Sd increased, creating a partially positive (δ+) region and repelling the extra β2m from the surface. Thus, as Sd decreased, it prohibited the gyration of β > 2m resulting in less δ+ region, and more β2m monomers were adsorbed with δ+ segment as the 2nd layer than the case where β > 2m gyrates more under relatively larger Sd. Based on the trend found in an experimentally extracted 2 as a function of Sd, optimized charge distribution of Aβ1−40 and α-syn were concluded to be negative partial charge (δ−) region that was covered by the positive partial charge (δ+) region. However, β > 2m may have a relatively large and/or distributed δ+ region with a small portion of δ− at the one end of a prolate. The characterization made in this work confirmed current understandings on the formation of the protein corona over nano-particles

Spiking Neural P System with weight model of majority voting technique for reliable interactive image segmentation

Spiking Neural P System with weight model of majority voting technique for reliable interactive image segmentation

Level Set Interactive Segmentation on Perforated Road Image using Region Based Active Countours Method with im’4

Level Set Interactive Segmentation on Perforated Road Image using Region Based Active Countours Method with im’4

An efficient interactive segmentation framework for medical images without pre-training.

Accurate and efficient medical image segmentation plays an important role in subsequent clinical applications such as diagnosis and surgical planning. This paper proposes an efficient interactive framework based on a graph convolutional network (GCN) for medical imagesegmentation. The initial segmentation results showed that a set of boundary control points can be generated for further interactive segmentation. We presented an adaptive interactive manner that allows the user to click on the boundary for fast interaction or drag the erroneous predicted control points for accurate correction. Furthermore, we proposed an interactive segmentation network (referred to as IVIF-GCN) to learn user experience in the interactive process by transforming interactive cues into annotations. In IVIF-GCN, a module of information fusion of image features and vertex position features (IVIF) is proposed to learn the location relationship between the current vertex and the neighboring vertices. Finally, the locations of control points around the interaction point is predicted and updatedautomatically. The proposed method achieves mean Dice of 96.6% and 91.3% on PROMISE12 and our in-house nasopharyngeal carcinoma (NPC) test sets, respectively. The experimental results showed that the proposed method outperforms the state-of-the-art segmentationmethods. The proposed interactive medical image segmentation method can efficiently improve segmentation results for clinical applications in the absence of training data. The GUI tool based on our method is available at https://github.com/Tian-lab/IGMedSeg.

Interpretable Deep Learning in Computational Histopathology for refined identification of Alzheimer’s Disease biomarkers

AbstractBackgroundIn this research, an explainable deep‐learning based framework is proposed for the segmentation of tau protein biomarkers like plaques, and tangles in histopathological whole slide images (WSI). The concept is to integrate an explainable deep learning model with collaborative human feedback to improve the model precision of identifying tau biomarkers as well as making the results interpretable to human experts. The final goal of this project is to refine AD patients’ stratification.MethodA pilot study has been done on 6 WSIs stained with ALZ50 monoclonal antibody. The preliminary models have been extended to a larger database of 15 WSIs stained with an AT8 monoclonal antibody. All WSIs are post‐mortem brain samples corresponding to the frontal cortex. In the proposed system, firstly, for automatic segmentation of Tau aggregates, deep learning (DL) models are trained and tested using the initial database (6 WSIs). Secondly, we train deep learning models with interpretability features, such as Attention U‐Net, CA‐net and graph neural networks. With these models, two operational loops (Figure 1) are implemented. In the first loop, a technical expert looks at the predictions of the trained model and provides corrections. The corrected predictions are fed back to the training dataset to improve the model performance. In the second loop, a clinical expert looks at the explainability results from the model and provides further corrective feedback.ResultA preliminary version of our DL models achieved promising F1 scores for tangles and neuritic plaques as reported in [1]. Currently, a second larger dataset of 15 WSIs (AT8) is studied and used for training an evolved version of the DL model, by including explainable visual features. These new overall results will be presented and discussed during the conference.ConclusionThe proposed interactive and interpretable tau protein segmentation framework is envisioned to be impactful towards Alzheimer's patient stratification problems which requires high accuracy identification of AD biomarkers.[1] K Maňoušková, V Abadie, M Ounissi, G Jimenez, L Stimmer, B. Delatour, S. Durrleman, D. Racoceanu. Tau Protein Discrete Aggregates in Alzheimer's Disease: Neuritic Plaques and Tangles Detection and Segmentation using Computational Histopathology. SPIE Medical Imaging 2022, Feb 2022, San Diego, United States.

Object Detection Based on the GrabCut Method for Automatic Mask Generation.

The Mask R-CNN-based object detection method is typically very time-consuming and laborious since it involves obtaining the required target object masks during training. Therefore, in order to automatically generate the image mask, we propose a GrabCut-based automated mask generation method for object detection. The proposed method consists of two stages. The first stage is based on GrabCut's interactive image segmentation method to generate the mask. The second stage is based on the object detection network of Mask R-CNN, which uses the mask from the previous stage together with the original input image and the associated label information for training. The Mask R-CNN model then automatically detects the relevant objects during testing. During experimentation with three objects from the Berkeley Instance Recognition Dataset, this method achieved a mean of average precision (mAP) value of over 95% for segmentation. The proposed method is simple and highly efficient in obtaining the mask of a segmented target object.

The Dahu graph-cut for interactive segmentation on 2D/3D images

Interactive image segmentation is an important application in computer vision for selecting objects of interest in images. Several interactive segmentation methods are based on distance transform algorithms. However, the most known distance transform, geodesic distance, is sensitive to noise in the image and to seed placement. Recently, the Dahu pseudo-distance, a continuous version of the minimum barrier distance (MBD), is proved to be more powerful than the geodesic distance in noisy and blurred images. This paper presents a method for combining the Dahu pseudo-distance with edge information in a graph-cut optimization framework and leveraging each’s complementary strengths. Our method works efficiently on both 2D/3D images and videos. Results show that our method achieves better performance than other distance-based and graph-cut methods, thereby reducing the user’s efforts.

Deep Interactive Learning-based ovarian cancer segmentation of H&E-stained whole slide images to study morphological patterns of BRCA mutation

Deep learning has been widely used to analyze digitized hematoxylin and eosin (H&E)-stained histopathology whole slide images. Automated cancer segmentation using deep learning can be used to diagnose malignancy and to find novel morphological patterns to predict molecular subtypes. To train pixel-wise cancer segmentation models, manual annotation from pathologists is generally a bottleneck due to its time-consuming nature. In this paper, we propose Deep Interactive Learning with a pretrained segmentation model from a different cancer type to reduce manual annotation time. Instead of annotating all pixels from cancer and non-cancer regions on giga-pixel whole slide images, an iterative process of annotating mislabeled regions from a segmentation model and training/finetuning the model with the additional annotation can reduce the time. Especially, employing a pretrained segmentation model can further reduce the time than starting annotation from scratch. We trained an accurate ovarian cancer segmentation model with a pretrained breast segmentation model by 3.5 hours of manual annotation which achieved intersection-over-union of 0.74, recall of 0.86, and precision of 0.84. With automatically extracted high-grade serous ovarian cancer patches, we attempted to train an additional classification deep learning model to predict BRCA mutation. The segmentation model and code have been released at https://github.com/MSKCC-Computational-Pathology/DMMN-ovary.

Interactive image segmentation based on multi-layer random forest classifiers

Interactive image segmentation based on multi-layer random forest classifiers

Tongue Contour Tracking and Segmentation in Lingual Ultrasound for Speech Recognition: A Review.

Lingual ultrasound imaging is essential in linguistic research and speech recognition. It has been used widely in different applications as visual feedback to enhance language learning for non-native speakers, study speech-related disorders and remediation, articulation research and analysis, swallowing study, tongue 3D modelling, and silent speech interface. This article provides a comparative analysis and review based on quantitative and qualitative criteria of the two main streams of tongue contour segmentation from ultrasound images. The first stream utilizes traditional computer vision and image processing algorithms for tongue segmentation. The second stream uses machine and deep learning algorithms for tongue segmentation. The results show that tongue tracking using machine learning-based techniques is superior to traditional techniques, considering the performance and algorithm generalization ability. Meanwhile, traditional techniques are helpful for implementing interactive image segmentation to extract valuable features during training and postprocessing. We recommend using a hybrid approach to combine machine learning and traditional techniques to implement a real-time tongue segmentation tool.

Interactive Image Segmentation Based on Feature-Aware Attention

Interactive segmentation is a technique for picking objects of interest in images according to users’ input interactions. Some recent works take the users’ interactive input to guide the deep neural network training, where the users’ click information is utilized as weak-supervised information. However, limited by the learning capability of the model, this structure does not accurately represent the user’s interaction intention. In this work, we propose a multi-click interactive segmentation solution for employing human intention to refine the segmentation results. We propose a coarse segmentation network to extract semantic information and generate rough results. Then, we designed a feature-aware attention module according to the symmetry of user intention and image semantic information. Finally, we establish a refinement module to combine the feature-aware results with coarse masks to generate precise intentional segmentation. Furthermore, the feature-aware module is trained as a plug-and-play tool, which can be embedded into most deep image segmentation models for exploiting users’ click information in the training process. We conduct experiments on five common datasets (SBD, GrabCut, DAVIS, Berkeley, MS COCO) and the results prove our attention module can improve the performance of image segmentation networks.

A Multi-Layer Auto-Segmentation Quality Assurance and Correction Pipeline for MR-Guided Adaptive Radiotherapy

<h3>Purpose/Objective(s)</h3> Fast and accurate auto-segmentation on daily image is essential for MR-guided adaptive radiotherapy (MRgART). However, the state-of-the-art deep learning auto-segmentation (DLAS) still has limited success particularly for complex structures such as those in the abdomen. Inaccurate contours from DLAS need to be inspected and edited and this manual process can be labor intensive, time consuming and subject to human errors. This work aims to develop a multi-layer pipeline that includes an automatic contour quality assurance (ACQA) test, an automatic contour refinement (ACR) process, and an interactive semi-automatic segmentation (ISS) tool to quickly identify and correct for inaccurate auto-segmented contours, thus replacing the manual process in MRgART. <h3>Materials/Methods</h3> The proposed pipeline includes the ACQA, ACR and ISS modules that are developed to detect and correct for inaccurate auto-segmented contours in the following steps: (1) identifying contours/image slices of acceptable or with minor or major errors from the input auto-segmented contours using ACQA, (2) correcting the identified inaccurate contours/slices using ACR, (3) repeating steps (1) and (2) on the contours obtained from the previous ACR until the contours/slices become acceptable or the performance of ACR becomes saturated, and (4) correcting the un-correctable contours from ACR using ISS tool. The steps 1-3 are designed to be fully automated. Details on ACQA, ACR and ISS are being investigated in separate studies. Briefly, the ACQA module uses either machine learning or DL classification algorithms based on organ and image properties to validate contour quality. The ACR module utilizes multiple correction methods, including an active contour model (ACM), U-Net and Dense UNet DL models to sequentially refine the inaccurate contours with minor and/or major errors. The ISS tool uses the accurate contours on an adjacent slice as prior knowledge informant to guide a 2D UNet DL model to predict contours on a subsequent slice. We have demonstrated the performance of the multi-layer pipeline on complex abdominal organs on various MRIs. <h3>Results</h3> The ACQA models achieved AUC/accuracy of 0.95/0.89 for pancreas and 0.92/0.88 for stomach contours. The ACM model improved 54%/49% of bowels/pancreas contours from major into minor errors and the UNet/Dense UNet ACR model improved 43%/33% of the minor bowel/stomach contours to be acceptable. The ISS tool achieved DSC of 0.92, 0.91, 0.91, 0.87, and 0.87 for colon, small bowel, stomach, pancreas, and duodenum. The expected execution time of the pipeline on an auto-segmented MRI was within 5 minutes. <h3>Conclusion</h3> This proposed multi-layer pipeline can complement deep learning auto-segmentation and can improve the segmentation efficiency and accuracy. As part of our on-going efforts, the pipeline is being implemented as a post DLAS process, substantially speeding up daily recontouring in MRgART.

ISegMSI: An Interactive Strategy to Improve Spatial Segmentation of Mass Spectrometry Imaging Data.

Spatial segmentation is a critical procedure in mass spectrometry imaging (MSI)-based biochemical analysis. However, the commonly used unsupervised MSI segmentation methods may lead to inappropriate segmentation results as the MSI data is characterized by high dimensionality and low signal-to-noise ratio. This process can be improved by the incorporation of precise prior knowledge, which is hard to obtain in most cases. In this study, we show that the incorporation of partial or coarse prior knowledge from different sources such as reference images or biological knowledge may also help to improve MSI segmentation results. Here, we propose a novel interactive segmentation strategy for MSI data called iSegMSI, which incorporates prior information in the form of scribble-regularization of the unsupervised model to fine-tune the segmentation results. By using two typical MSI data sets (including a whole-body mouse fetus and human thyroid cancer), the present results demonstrate the effectiveness of the iSegMSI strategy in improving the MSI segmentations. Specifically, the method can be used to subdivide a region into several subregions specified by the user-defined scribbles or to merge several subregions into a single region. Additionally, these fine-tuned results are highly tolerant to the imprecision of the scribbles. Our results suggest that the proposed iSegMSI method may be an effective preprocessing strategy to facilitate the analysis of MSI data.