Graph-based Segmentation Method Research Articles

Joint Reconstruction-Segmentation on Graphs

Practical image segmentation tasks concern images which must be reconstructed from noisy, distorted, and/or incomplete observations. A recent approach for solving such tasks is to perform this reconstruction jointly with the segmentation, using each to guide the other. However, this work has so far employed relatively simple segmentation methods, such as the Chan–Vese algorithm. In this paper, we present a method for joint reconstruction-segmentation using graph-based segmentation methods, which have been seeing increasing recent interest. Complications arise due to the large size of the matrices involved, and we show how these complications can be managed. We then analyze the convergence properties of our scheme. Finally, we apply this scheme to distorted versions of “two cows” images familiar from previous graph-based segmentation literature, first to a highly noised version and second to a blurred version, achieving highly accurate segmentations in both cases. We compare these results to those obtained by sequential reconstruction-segmentation approaches, finding that our method competes with, or even outperforms, those approaches in terms of reconstruction and segmentation accuracy.

Adaptive fusion affinity graph with noise-free online low-rank representation for natural image segmentation

Affinity graph-based segmentation methods have become a major trend in computer vision. The performance of these methods rely on the constructed affinity graph, with particular emphasis on the neighborhood topology and pairwise affinities among superpixels. However, these graph-based methods ignore the noisy data from images, that influence the accuracy of pairwise similarities. Multiscale combinatorial grouping and graph fusion also generate a higher computational complexity. In this paper, we propose an adaptive fusion affinity graph with noise-free low-rank representation in an online manner for natural image segmentation. An input image is first over-segmented into superpixels at different scales and then filtered by an improved kernel density estimation method. Moreover, we select global nodes of these superpixels on the basis of their subspace-preserving presentation, which reveals the feature distribution of superpixels exactly. To reduce time complexity while improving performance, a sparse representation of global nodes based on noise-free online low-rank representation is used to obtain a global graph at each scale. Experimental results on BSD300, BSD500, MSRC, SBD, and PASCAL VOC show the effectiveness of our method in comparison with the state-of-the-art approaches. The code is available at https://github.com/Yangzhangcst/AFA-graph.

Component segmentation of engineering drawings using Graph Convolutional Networks

We present a data-driven framework to automate the vectorization and machine interpretation of 2D engineering part drawings. In industrial settings, most manufacturing engineers still rely on manual reads to identify the topological and manufacturing requirements from drawings submitted by designers. The interpretation process is laborious and time-consuming, which severely inhibits the efficiency of part quotation and manufacturing tasks. While recent advances in image-based computer vision methods have demonstrated great potential in interpreting natural images through semantic segmentation approaches, the application of such methods in parsing engineering technical drawings into semantically accurate components remains a significant challenge. The severe pixel sparsity in engineering drawings also restricts the effective featurization of image-based data-driven methods. To overcome these challenges, we propose a deep learning based framework that predicts the semantic type of each vectorized component. Taking a raster image as input, we vectorize all components through thinning, stroke tracing, and cubic bezier fitting. Then a graph of such components is generated based on the connectivity between the components. Finally, a graph convolutional neural network is trained on this graph data to identify the semantic type of each component. We test our framework in the context of semantic segmentation of text, dimension and, contour components in engineering drawings. Results show that our method yields the best performance compared to recent image-based, and graph-based segmentation methods.

The Study of Inspection on Thin Film Resistance Strain Gauge Contact Failure by Electrical Excitation Thermal-Wave Imaging

Contact failure as one of primary failures, which occurs between thin film resistance strain gauge (FRSG) and metal measured objective substrate, seriously influences the measurement validity and precision of the strain gauge. In this article, an electrical excitation thermal-wave imaging (EE-TWI) method is proposed for the inspection of contact failure. Equivalent circuits of the contact resistance under an external electrical source are used to identify the location of the contact failure. Long pulse thermography and lock-in thermography (LIT) are experimentally studied on the detection of contact failure. Pulsed phase thermography and principal component thermography in long pulse thermography as well as the amplitude, quadrature component in LIT are desirable to form the EE-TWI feature images, respectively. A graph-based segmentation method is employed to distinguish and localize the contact failure from EE-TWI images. EE-TWI provides a powerful tool for the contact failure evaluation of integrated thin film resistance strain gauge (FRSG) and has a significant potential to ensure the thin FRSG sticking performance and quality with application of mechanical properties measurements.

A new color distance measure formulated from the cooperation of the Euclidean and the vector angular differences for lidar point cloud segmentation

Two important features of the points in the LiDAR point clouds are the spatial and the color features. The spatial feature is mostly used in the point cloud processing field due to its 3D informative and distinctive characteristic. The local geometric difference derived from the spatial features of the points is usually benefited by graph-based point cloud segmentation methods, because the geometric features of the local point groups are highly distinctive. In this paper, we use both the geometric and color differences of the adjacent local point groups at the impact rates 0.3, 0.5, and 0.7 and cooperate the Euclidean and the vector color differences within several averaging techniques for the color difference. The difference forms have been tested within a graph-based segmentation method on four point cloud segmentation datasets, two indoor and two outdoor, using their spatial and color information. The geometric mean as an averaging techniques increases the segmentation success for the all datasets except one outdoor when the color differences are used in the segmentation at the impact rate 0.3, while the harmonic mean increases the success for the all datasets the successes except the other outdoor at the same impact rate. According to the test results, the cooperating of the Euclidean and vector angular color difference measurements can considerable increase the segmentation success on the point clouds with color information in a high quality.

SEEK: A Framework of Superpixel Learning with CNN Features for Unsupervised Segmentation

Supervised semantic segmentation algorithms have been a hot area of exploration recently, but now the attention is being drawn towards completely unsupervised semantic segmentation. In an unsupervised framework, neither the targets nor the ground truth labels are provided to the network. That being said, the network is unaware about any class instance or object present in the given data sample. So, we propose a convolutional neural network (CNN) based architecture for unsupervised segmentation. We used the squeeze and excitation network, due to its peculiar ability to capture the features’ interdependencies, which increases the network’s sensitivity to more salient features. We iteratively enable our CNN architecture to learn the target generated by a graph-based segmentation method, while simultaneously preventing our network from falling into the pit of over-segmentation. Along with this CNN architecture, image enhancement and refinement techniques are exploited to improve the segmentation results. Our proposed algorithm produces improved segmented regions that meet the human level segmentation results. In addition, we evaluate our approach using different metrics to show the quantitative outperformance.

A novel hybrid atlas-free hierarchical graph-based segmentation of newborn brain MRI using wavelet filter banks

Objective: The newborn brain MRI (magnetic resonance imaging) tissue segmentation plays a vital part in assessment of primary brain growth. In the newborn stage (nearly less than 28 days old), in T1- as well as T2-weighted MR images similar levels of intensity are exhibited by WM and GM, makes segmentation of the tissue extremely challenging. In this newborn stage for tissue segmentation, very few methods are developed. Hence the development of accurate brain tissue segmentation of neonate is prime objective of this paper.Methods: In this research work, we propose a novel hybrid atlas-free hierarchical graph-based tissue segmentation method for newborn infants. Wavelet filter banks are a class of deep models wherein filters and local neighborhood processes are used alternately for efficient segmentation on the raw input images, and fuzzy-based SVM (support vector machine) is used for precise tissue classification.Results: Specifically, from T1, T2 images multimodality information are used as inputs and then as outputs the segmentation maps are generated. The proposed approach considerably outperforms preceding methods of tissue segmentation as reflected in results. With this approach, the newborn MRI images that are even suffered from noise, poor resolution or the low contrasted images are also segmented more effectively with precision of 90% and sensitivity 98%.Conclusion: In addition, our findings indicate that the incorporation of multi-modality image led to significant improvements in performance. Thus, the proposed work effectively tackles the unreliability as well as the other issues faced with the prior methodologies with an interactive accurate segmentation outline.

Novel Descriptors of Coronary Wall Properties Measured by OCT at Baseline Are Associated with CAV-Related Intimal Thickening One Year after Heart Transplant

Purpose Cardiac Allograft Vasculopathy (CAV) is a frequent complication of heart transplantation (HTx). To help identify patients at risk of cardiac graft failure, early determination of coronary wall thickening is of major importance. Using highly automated 3D segmentation of coronary wall layers, novel indices of intimal and medial roughness are reported. Ability of these indices obtained at 1 month (1M) to predict intimal thickening 12 months (12M) after HTx was studied. Methods Lumen surface, intimal and media layers (as depicted by OCT) were identified using our multi-surface graph-based LOGISMOS segmentation method, reported earlier. Automated results were adjusted by experts utilizing our Just-Enough-Interaction approach. Intimal thickness (IT) was determined for each image frame of the OCT pullback obtained 1M and 12M after HTx. CAV risk was quantified as average OCT-frame-specific intimal thickening (IT12M - IT1M) after 1M—12M image registration. Intimal (medial) roughness IR (MR), reflecting the circumferential variability of the intimal (medial) outer surface, was computed as the average distance between the thin-plate spline fitted to the intimal (medial) surface and the regional peaks of individual surface points. As such, smooth (rough) intimal or medial surface would be represented by small (large) IR or MR values. Results Associations between frame-based IR or MR at 1M and frame-based IT progression were studied in 116 patients for which 1M and 12M OCT pullbacks (538 frames each) were acquired. Mixed effect analysis with individual patient as grouping factor was employed. IR and MR determined at 1M were strongly associated with IT progression 12M after HTx (IR: β=-9.49, p=0.004; MR: β=-9.76, p=0.001). In a patient-level analysis using Pearson's correlation test, the negative association was further confirmed between 1M roughness and IT progression in the per-patient-most-progressing 3 mm-long vessel segment (IR: R=-0.28, p=0.002; MR: R=-0.28, p=0.002). Conclusion Our new descriptors of intimal and medial roughness derived from fully 3D segmentation of coronary wall layers in OCT image pullbacks 1 month after HTx were associated with CAV-related progression of intimal thickness at 12 months. As such, IR and MR indices showed CAV-onset prediction ability as early as 1 month after HTx.

Interactive image segmentation using label propagation through complex networks

Interactive image segmentation is a topic of many studies in image processing. In a conventional approach, a user marks some pixels of the object(s) of interest and background, and an algorithm propagates these labels to the rest of the image. This paper presents a new graph-based method for interactive segmentation with two stages. In the first stage, nodes representing pixels are connected to their k-nearest neighbors to build a complex network with the small-world property to propagate the labels quickly. In the second stage, a regular network in a grid format is used to refine the segmentation on the object borders. Despite its simplicity, the proposed method can perform the task with high accuracy. Computer simulations are performed using some real-world images to show its effectiveness in both two-classes and multi-classes problems. It is also applied to all the images from the Microsoft GrabCut dataset for comparison, and the segmentation accuracy is comparable to those achieved by some state-of-the-art methods, while it is faster than them. In particular, it outperforms some recent approaches when the user input is composed only by a few “scribbles” draw over the objects. Its computational complexity is only linear on the image size at the best-case scenario and linearithmic in the worst case.

Simultaneous cosegmentation of tumors in PET-CT images using deep fully convolutional networks.

To investigate the use and efficiency of 3-D deep learning, fully convolutional networks (DFCN) for simultaneous tumor cosegmentation on dual-modality nonsmall cell lung cancer (NSCLC) and positron emission tomography (PET)-computed tomography (CT) images. We used DFCN cosegmentation for NSCLC tumors in PET-CT images, considering both the CT and PET information. The proposed DFCN-based cosegmentation method consists of two coupled three-dimensional (3D)-UNets with an encoder-decoder architecture, which can communicate with the other in order to share complementary information between PET and CT. The weighted average sensitivity and positive predictive values denoted as Scores, dice similarity coefficients (DSCs), and the average symmetric surface distances were used to assess the performance of the proposed approach on 60 pairs of PET/CTs. A Simultaneous Truth and Performance Level Estimation Algorithm (STAPLE) of 3 expert physicians' delineations were used as a reference. The proposed DFCN framework was compared to 3 graph-based cosegmentation methods. Strong agreement was observed when using the STAPLE references for the proposed DFCN cosegmentation on the PET-CT images. The average DSCs on CT and PET are 0.861±0.037 and 0.828±0.087, respectively, using DFCN, compared to 0.638±0.165 and 0.643±0.141, respectively, when using the graph-based cosegmentation method. The proposed DFCN cosegmentation using both PET and CT also outperforms the deep learning method using either PET or CT alone. The proposed DFCN cosegmentation is able to outperform existing graph-based segmentation methods. The proposed DFCN cosegmentation shows promise for further integration with quantitative multimodality imaging tools in clinical trials.

Stereo matching using census cost over cross window and segmentation-based disparity refinement

Stereo matching is a vital requirement for many applications, such as three-dimensional (3-D) reconstruction, robot navigation, object detection, and industrial measurement. To improve the practicability of stereo matching, a method using census cost over cross window and segmentation-based disparity refinement is proposed. First, a cross window is obtained using distance difference and intensity similarity in binocular images. Census cost over the cross window and color cost are combined as the matching cost, which is aggregated by the guided filter. Then, winner-takes-all strategy is used to calculate the initial disparities. Second, a graph-based segmentation method is combined with color and edge information to achieve moderate under-segmentation. The segmented regions are classified into reliable regions and unreliable regions by consistency checking. Finally, the two regions are optimized by plane fitting and propagation, respectively, to match the ambiguous pixels. The experimental results are on Middlebury Stereo Datasets, which show that the proposed method has good performance in occluded and discontinuous regions, and it obtains smoother disparity maps with a lower average matching error rate compared with other algorithms.

WEAKLY SUPERVISED FOOD IMAGE SEGMENTATION USING CLASS ACTIVATION MAPS.

Food image segmentation plays a crucial role in image-based dietary assessment and management. Successful methods for object segmentation generally rely on a large amount of labeled data on the pixel level. However, such training data are not yet available for food images and expensive to obtain. In this paper, we describe a weakly supervised convolutional neural network (CNN) which only requires image level annotation. We propose a graph based segmentation method which uses the class activation maps trained on food datasets as a top-down saliency model. We evaluate the proposed method for both classification and segmentation tasks. We achieve competitive classification accuracy compared to the previously reported results.

Dynamic Post-Earthquake Image Segmentation with an Adaptive Spectral-Spatial Descriptor

The region merging algorithm is a widely used segmentation technique for very high resolution (VHR) remote sensing images. However, the segmentation of post-earthquake VHR images is more difficult due to the complexity of these images, especially high intra-class and low inter-class variability among damage objects. Herein two key issues must be resolved: the first is to find an appropriate descriptor to measure the similarity of two adjacent regions since they exhibit high complexity among the diverse damage objects, such as landslides, debris flow, and collapsed buildings. The other is how to solve over-segmentation and under-segmentation problems, which are commonly encountered with conventional merging strategies due to their strong dependence on local information. To tackle these two issues, an adaptive dynamic region merging approach (ADRM) is introduced, which combines an adaptive spectral-spatial descriptor and a dynamic merging strategy to adapt to the changes of merging regions for successfully detecting objects scattered globally in a post-earthquake image. In the new descriptor, the spectral similarity and spatial similarity of any two adjacent regions are automatically combined to measure their similarity. Accordingly, the new descriptor offers adaptive semantic descriptions for geo-objects and thus is capable of characterizing different damage objects. Besides, in the dynamic region merging strategy, the adaptive spectral-spatial descriptor is embedded in the defined testing order and combined with graph models to construct a dynamic merging strategy. The new strategy can find the global optimal merging order and ensures that the most similar regions are merged at first. With combination of the two strategies, ADRM can identify spatially scattered objects and alleviates the phenomenon of over-segmentation and under-segmentation. The performance of ADRM has been evaluated by comparing with four state-of-the-art segmentation methods, including the fractal net evolution approach (FNEA, as implemented in the eCognition software, Trimble Inc., Westminster, CO, USA), the J-value segmentation (JSEG) method, the graph-based segmentation (GSEG) method, and the statistical region merging (SRM) approach. The experiments were conducted on six VHR subarea images captured by RGB sensors mounted on aerial platforms, which were acquired after the 2008 Wenchuan Ms 8.0 earthquake. Quantitative and qualitative assessments demonstrated that the proposed method offers high feasibility and improved accuracy in the segmentation of post-earthquake VHR aerial images.

Automatic evaluation of vessel diameter variation from 2D X-ray angiography.

Early detection of blood vessel pathologies can be made through the evaluation of functional and structural abnormalities in the arteries, including the arterial distensibility measure. We propose a feasibility study on computing arterial distensibility automatically from monoplane 2D X-ray sequences for both small arteries (such as coronary arteries) and larger arteries (such as the aorta). To compute the distensibility measure, three steps were developed: First, the segment of an artery is extracted using our graph-based segmentation method. Then, the same segment is tracked in the moving sequence using our spatio-temporal segmentation method: the Temporal Vessel Walker. Finally, the diameter of the artery is measured automatically at each frame of the sequence based on the segmentation results. The method was evaluated using one simulated sequence and 4 patients' angiograms depicting the coronary arteries and three depicting the ascending aorta. Results of the simulated sequence achieved a Dice index of 98%, with a mean squared error in diameter measurement of [Formula: see text]mm. Results obtained from patients' X-ray sequences are consistent with manual assessment of the diameter by experts. The proposed method measures changes in diameter of a specific segment of a blood vessel during the cardiac sequence, automatically based on monoplane 2D X-ray sequence. Such information might become a key to help physicians in the detection of variations of arterial stiffness associated with early stages of various vasculopathies.

Automatic MPST-cut for segmentation of carpal bones from MR volumes

In the context of rheumatic diseases, several studies suggest that Magnetic Resonance Imaging (MRI) allows the detection of the three main signs of Rheumatoid Arthritis (RA) at higher sensitivities than available through conventional radiology. The rapid, accurate segmentation of bones is an essential preliminary step for quantitative diagnosis, erosion evaluation, and multi-temporal data fusion. In the present paper, a new, semi-automatic, 3D graph-based segmentation method to extract carpal bone data is proposed. The method is unsupervised, does not employ any a priori model or knowledge, and is adaptive to the individual variability of the acquired data. After selecting one source point inside the Region of Interest (ROI), a segmentation process is initiated, which consists of two automatic stages: a cost-labeling phase and a graph-cutting phase. The algorithm finds optimal paths based on a new cost function by creating a Minimum Path Spanning Tree (MPST). To extract the region, a cut of the obtained tree is necessary. A new criterion of the MPST-cut based on compactness shape factor was conceived and developed.The proposed approach is applied to a large database of 96 T1-weighted MR bone volumes. Performance quality is evaluated by comparing the results with gold-standard bone volumes manually defined by rheumatologists through the computation of metrics extracted from the confusion matrix. Furthermore, comparisons with the existing literature are carried out. The results show that this method is efficient and provides satisfactory performance for bone segmentation on low-field MR volumes.

A Novel Segmentation Approach Combining Region- and Edge-Based Information for Ultrasound Images.

Ultrasound imaging has become one of the most popular medical imaging modalities with numerous diagnostic applications. However, ultrasound (US) image segmentation, which is the essential process for further analysis, is a challenging task due to the poor image quality. In this paper, we propose a new segmentation scheme to combine both region- and edge-based information into the robust graph-based (RGB) segmentation method. The only interaction required is to select two diagonal points to determine a region of interest (ROI) on the original image. The ROI image is smoothed by a bilateral filter and then contrast-enhanced by histogram equalization. Then, the enhanced image is filtered by pyramid mean shift to improve homogeneity. With the optimization of particle swarm optimization (PSO) algorithm, the RGB segmentation method is performed to segment the filtered image. The segmentation results of our method have been compared with the corresponding results obtained by three existing approaches, and four metrics have been used to measure the segmentation performance. The experimental results show that the method achieves the best overall performance and gets the lowest ARE (10.77%), the second highest TPVF (85.34%), and the second lowest FPVF (4.48%).

4D Graph-Based Segmentation for Reproducible and Sensitive Choroid Quantification From Longitudinal OCT Scans.

PurposeLongitudinal imaging is becoming more commonplace for studies of disease progression, response to treatment, and healthy maturation. Accurate and reproducible quantification methods are desirable to fully mine the wealth of data in such datasets. However, most current retinal OCT segmentation methods are cross-sectional and fail to leverage the inherent context present in longitudinal sequences of images.MethodsWe propose a novel graph-based method for segmentation of multiple three-dimensional (3D) scans over time (termed 3D + time or 4D). The usefulness of this approach in retinal imaging is illustrated in the segmentation of the choroidal surfaces from longitudinal optical coherence tomography (OCT) scans. A total of 3219 synthetic (3070) and patient (149) OCT images were segmented for validation of our approach.ResultsThe results show that the proposed 4D segmentation method is significantly more reproducible (P < 0.001) than the 3D approach and is significantly more sensitive to temporal changes (P < 0.0001) achieved by the substantial increase of measurement robustness.ConclusionsThis is the first automated 4D method for jointly quantifying choroidal thickness in longitudinal OCT studies. Our method is robust to image noise and produces more reproducible choroidal thickness measurements than a sequence of independent 3D segmentations, without sacrificing sensitivity to temporal changes.

An efficient segmentation method based on dynamic graph merging

A novel energy functional based on the Mumford–Shah model is established for performing automatic image segmentation. And in order to optimize the global model using graph-based methods, we develop a localized formula. Then, we propose a merging predicate for determining whether an edge connecting two neighboring pixels or regions merge. The dynamic graph merging (DGM) method is applied based on this merging predicate. That is, those edges with large energy merge and the edges with low energy are remained, such that the energy functional is minimized. Compared with other graph-based segmentation methods, our algorithm based on DGM has an important characteristic which is its ability to produce good segmentation on some complex texture images. Another characteristic is that this segmentation algorithm can avoid the “shrinking bias” problem. We also apply DGM to interactive image segmentation and find the results to be encouraging too.

Nondestructive analysis of automotive paints with spectral domain optical coherence tomography.

We have demonstrated for the first time, to our knowledge, the use of optical coherence tomography (OCT) as an analytical tool for nondestructively characterizing the individual paint layer thickness of multiple layered automotive paints. A graph-based segmentation method was used for automatic analysis of the thickness distribution for the top layers of solid color paints. The thicknesses measured with OCT were in good agreement with the optical microscope and ultrasonic techniques that are the current standard in the automobile industry. Because of its high axial resolution (5.5 μm), the OCT technique was shown to be able to resolve the thickness of individual paint layers down to 11 μm. With its high lateral resolution (12.4 μm), the OCT system was also able to measure the cross-sectional area of the aluminum flakes in a metallic automotive paint. The range of values measured was 300-1850 μm2. In summary, the proposed OCT is a noncontact, high-resolution technique that has the potential for inclusion as part of the quality assurance process in automobile coating.

Automated Segmentability Index for Layer Segmentation of Macular SD-OCT Images.

PurposeTo automatically identify which spectral-domain optical coherence tomography (SD-OCT) scans will provide reliable automated layer segmentations for more accurate layer thickness analyses in population studies.MethodsSix hundred ninety macular SD-OCT image volumes (6.0 × 6.0 × 2.3 mm3) were obtained from one eyes of 690 subjects (74.6 ± 9.7 [mean ± SD] years, 37.8% of males) randomly selected from the population-based Rotterdam Study. The dataset consisted of 420 OCT volumes with successful automated retinal nerve fiber layer (RNFL) segmentations obtained from our previously reported graph-based segmentation method and 270 volumes with failed segmentations. To evaluate the reliability of the layer segmentations, we have developed a new metric, segmentability index SI, which is obtained from a random forest regressor based on 12 features using OCT voxel intensities, edge-based costs, and on-surface costs. The SI was compared with well-known quality indices, quality index (QI), and maximum tissue contrast index (mTCI), using receiver operating characteristic (ROC) analysis.ResultsThe 95% confidence interval (CI) and the area under the curve (AUC) for the QI are 0.621 to 0.805 with AUC 0.713, for the mTCI 0.673 to 0.838 with AUC 0.756, and for the SI 0.784 to 0.920 with AUC 0.852. The SI AUC is significantly larger than either the QI or mTCI AUC (P < 0.01).ConclusionsThe segmentability index SI is well suited to identify SD-OCT scans for which successful automated intraretinal layer segmentations can be expected.Translational RelevanceInterpreting the quantification of SD-OCT images requires the underlying segmentation to be reliable, but standard SD-OCT quality metrics do not predict which segmentations are reliable and which are not. The segmentability index SI presented in this study does allow reliable segmentations to be identified, which is important for more accurate layer thickness analyses in research and population studies.