Hierarchical Image Segmentation Research Articles

A Novel Semantic Segmentation Algorithm for RGB-D Images Based on Non-Symmetry and Anti-Packing Pattern Representation Model

With the rapid development of deep learning technology, the accuracy of image semantic segmentation tasks has been greatly improved. However, indoor RGB-D semantic segmentation remains a challenging problem because of the complexity of indoor environments. The emergence of depth sensors makes depth information gradually used to improve the effect of semantic segmentation. The splicing of weights such as between the RGB features and the depth features which are used as the input features of the neural network can effectively improve the accuracy of the indoor semantic segmentation tasks. Most previous researches focused on improving the performance of semantic segmentation by adjusting the structure of convolutional neural network. These researches have either added attention mechanism or performed data augmentation on input features, but they didn’t make full use of the boundary information and the texture information of the original RGB image. In this paper, we propose a semantic segmentation algorithm for RGB-D images based on Non-symmetry and Anti-packing pattern representation Model (NAM). The core idea of the proposed algorithm is to take the channel-wise concatenation of pre-segmentation labels provided by the traditional hierarchical image segmentation and RGB-D features as the input of the neural network so as to guide the semantic segmentation tasks. The extensive experiments are conducted on the popular indoor RGB-D semantic segmentation datasets. When compared with the state-of-art algorithms, the experimental results presented in this paper show that our proposed method has improved the performance of image semantic segmentation networks on several popular neural network architectures.

Semi-Supervised Skin Lesion Segmentation via Iterative Mask Optimization

Deep learning-based skin lesion segmentation methods have achieved promising results in the community. However, they are usually based on fully supervised learning and require many high-quality ground truths. Labeling the ground truths takes a lot of labor, material, and financial resources. We propose a novel semi-supervised skin lesion segmentation method to solve this problem. First, a hierarchical image segmentation algorithm is used to generate optimal segmentation maps. Then, fully supervised training is performed on a small part of the images with ground truths. The resulting pseudo masks are generated to train the rest of the images. The optimal segmentation maps are utilized in this process to refine the pseudo masks. Experiments show that the proposed method can improve the performance of semi-supervised learning for skin lesion segmentation by reducing the gap with fully supervised learning methods. Moreover, it can reduce the workload of labeling the ground truths. Extensive experiments are conducted on the open dataset to validate the efficiency of the proposed method. The results show that our method is competitive in improving the quality of semi-supervised segmentation.

Weakly supervised skin lesion segmentation based on spot‐seeds guided optimal regions

Automatic skin lesion segmentation is the most critical and relevant task in computer-aided skin cancer diagnosis. Methods based on convolutional neural networks (CNNs) are mainly used in current skin lesion segmentation. The requirement of huge pixel-level labels is a significant obstacle to achieve semantic segmentation of skin lesion by CNNs. In this paper, a novel weakly supervised framework for skin lesion segmentation is presented, which generates high-quality pixel-level annotations and optimizes the segmentation network. A hierarchical image segmentation algorithm can predict a boundary map for training images. Then, the optimal regions of candidate hierarchical levels are selected. Afterward, Superpixels-CRF built on the optimal regions is guided by spot seeds to propagate information from spot seeds to unlabeled regions, resulting in high-quality pixel-level annotations. Using these high-quality pixel-level annotations, a segmentation network can be trained and segmentation masks can be predicted. To iteratively optimize the segmentation network, the predicted segmentation masks are refined and the segmentation network are retrained. Comparative experiments demonstrate that the proposed segmentation framework reduces the gap between weakly and fully supervised skin lesion segmentation methods, and achieves state-of-the-art performance while reducing human labeling efforts.

Uncertainty‐aware Visualization of Regional Time Series Correlation in Spatio‐temporal Ensembles

AbstractGiven a time‐varying scalar field, the analysis of correlations between different spatial regions, i.e., the linear dependence of time series within these regions, provides insights into the structural properties of the data. In this context, regions are connected components of the spatial domain with high time series correlations. The detection and analysis of such regions is often performed globally, which requires pairwise correlation computations that are quadratic in the number of spatial data samples. Thus, operations based on all pairwise correlations are computationally demanding, especially when dealing with ensembles that model the uncertainty in the spatio‐temporal phenomena using multiple simulation runs. We propose a two‐step procedure: In a first step, we map the spatial samples to a 3D embedding based on a pairwise correlation matrix computed from the ensemble of time series. The 3D embedding allows for a one‐to‐one mapping to a 3D color space such that the outcome can be visually investigated by rendering the colors for all samples in the spatial domain. In a second step, we generate a hierarchical image segmentation based on the color images. From then on, we can visually analyze correlations of regions at all levels in the hierarchy within an interactive setting, which includes the uncertainty‐aware analysis of the region's time series correlation and respective time lags.

Optimal Scale of Hierarchical Image Segmentation with Scribbles Guidance for Weakly Supervised Semantic Segmentation

Deep convolutional neural networks (DCNNs) trained on the pixel-level annotated images have achieved improvements in semantic segmentation. Due to the high cost of labeling training data, their applications may have great limitation. However, weakly supervised segmentation approaches can significantly reduce human labeling efforts. In this paper, we introduce a new framework to generate high-quality initial pixel-level annotations. By using a hierarchical image segmentation algorithm to predict the boundary map, we select the optimal scale of high-quality hierarchies. In the initialization step, scribble annotations and the saliency map are combined to construct a graphic model over the optimal scale segmentation. By solving the minimal cut problem, it can spread information from scribbles to unmarked regions. In the training process, the segmentation network is trained by using the initial pixel-level annotations. To iteratively optimize the segmentation, we use a graphical model to refine segmentation masks and retrain the segmentation network to get more precise pixel-level annotations. The experimental results on Pascal VOC 2012 dataset demonstrate that the proposed framework outperforms most of weakly supervised semantic segmentation methods and achieves the state-of-the-art performance, which is [Formula: see text] mIoU.

Hierarchical Image Segmentation Based on Nonsymmetry and Anti-Packing Pattern Representation Model.

Image segmentation is the foundation of high-level image analysis and image understanding. How to effectively segment an image into regions that are "meaningful" to the human visual perception and ensure that the segmented regions are consistent at different resolutions is still a very challenging issue. Inspired by the idea of the Nonsymmetry and Anti-packing pattern representation Model in the Lab color space (NAMLab) and the "global-first" invariant perceptual theory, in this paper, we propose a novel framework for hierarchical image segmentation. Firstly, by defining the dissimilarity between two pixels in the Lab color space, we propose an NAMLab-based color image representation approach that is more in line with the human visual perception characteristics and can make the image pixels fast and effectively merge into the NAMLab blocks. Then, by defining the dissimilarity between two NAMLab-based regions and iteratively executing NAMLab-based merging algorithm of adjacent regions into larger ones to progressively generate a segmentation dendrogram, we propose a fast NAMLab-based algorithm for hierarchical image segmentation. Finally, the complexities of our proposed NAMLab-based algorithm for hierarchical image segmentation are analyzed in details. The experimental results presented in this paper show that our proposed algorithm when compared with the state-of-the-art algorithms not only can preserve more details of the object boundaries, but also it can better identify the foreground objects with similar color distributions. Also, our proposed algorithm can be executed much faster and takes up less memory and therefore it is a better algorithm for hierarchical image segmentation.

Multi-Faceted Hierarchical Image Segmentation Taxonomy (MFHIST)

An abundance of various segmentation techniques are available in the literature, that cater to wide range of image understanding applications. The paper proposes a unified way of systematic categorization of the research work on image segmentation called Multi-Faceted Hierarchical Image Segmentation Taxonomy (MFHIST), which consist of six facets presented in a hierarchical manner - scope, requirement, control, feature, image representation and approach specifications. Every scope is exemplified with research works from the literature for better understanding. The paper gives an illustration of populating MFHIST, to provide the reader a quick grasp of few important state-of-art image segmentation research works and their adaptations. As a case study, the illustrations display a limited version to uncover the journey of basic to modern adaptations in the areas region based segmentation approach, such as Markov Random Fields, Spectral Clustering, Active Contour Model, Mean Shift Clustering. The other segmentation approaches have not been considered here, owing to the enormous volume of works in the past four to five decades and limitation in articulating all of them using MFHIST. The performance analysis of the algorithms using quantitative metrics is not in the present scope and will be considered in future version of MFHIST.

Object scale selection of hierarchical image segmentation with deep seeds

Object scale selection of hierarchical image segmentation with deep seeds

Multi-scale region composition of hierarchical image segmentation

Hierarchical image segmentation is a prominent trend in the literature as a way to improve the segmentation quality. Generally, meaningful objects in an image are described by segments from different scales. Thus, one may spend extra effort on searching for the best representation of objects in the hierarchical segmentation result. In this paper, a novel algorithm is proposed to optimally select the segmentation scale, which leads to a composite segmentation as the output. To this end, the quality of regions from different scales of the hierarchical segmentation is evaluated. Then, a graphical model is constructed as a set of nodes. The weights among nodes are computed according to the segmentation quality of regions in multiple levels. In order to optimize the labeling of each node in the graph, the composition process is performed twice with two sampling intervals. Comprehensive experiments are conducted on different datasets for popular hierarchical image segmentation algorithms. The results show that the output of the proposed algorithm can improve the quality of hierarchical segmentation in a single scale at a low cost of computation.

Mapping Heterogeneous Urban Landscapes from the Fusion of Digital Surface Model and Unmanned Aerial Vehicle-Based Images Using Adaptive Multiscale Image Segmentation and Classification

Considering the high-level details in an ultrahigh-spatial-resolution (UHSR) unmanned aerial vehicle (UAV) dataset, detailed mapping of heterogeneous urban landscapes is extremely challenging because of the spectral similarity between classes. In this study, adaptive hierarchical image segmentation optimization, multilevel feature selection, and multiscale (MS) supervised machine learning (ML) models were integrated to accurately generate detailed maps for heterogeneous urban areas from the fusion of the UHSR orthomosaic and digital surface model (DSM). The integrated approach commenced through a preliminary MS image segmentation parameter selection, followed by the application of three supervised ML models, namely, random forest (RF), support vector machine (SVM), and decision tree (DT). These models were implemented at the optimal MS levels to identify preliminary information, such as the optimal segmentation level(s) and relevant features, for extracting 12 land use/land cover (LULC) urban classes from the fused datasets. Using the information obtained from the first phase of the analysis, detailed MS classification was iteratively conducted to improve the classification accuracy and derive the final urban LULC maps. Two UAV-based datasets were used to develop and assess the effectiveness of the proposed framework. The hierarchical classification of the pilot study area showed that the RF was superior with an overall accuracy (OA) of 94.40% and a kappa coefficient (K) of 0.938, followed by SVM (OA = 92.50% and K = 0.917) and DT (OA = 91.60% and K = 0.908). The classification results of the second dataset revealed that SVM was superior with an OA of 94.45% and K of 0.938, followed by RF (OA = 92.46% and K = 0.916) and DT (OA = 90.46% and K = 0.893). The proposed framework exhibited an excellent potential for the detailed mapping of heterogeneous urban landscapes from the fusion of UHSR orthophoto and DSM images using various ML models.

Learning to realign hierarchy for image segmentation

A hierarchical image segmentation is a set of image segmentations at different detail levels. However, objects (or even parts of the same object) may appear at different scales due to their size differences or to their distinct distances from the camera. One possible solution to cope with that is to realign the hierarchy such that every region containing an object (or its parts) is at the same level. In this work, we have explored the use of regression models to predict score values for regions belonging to a hierarchy of partitions, which are used to realign it. We have also proposed a new score calculation and a new assessment strategy considering all user-defined segmentations that exist in the ground-truth. Experimental results have pointed out that the use of new proposed score was able to improve final segmentation results.

Simplifying TugGraph using zipping algorithms

Graphs are an invaluable modelling tool in many domains, but visualising large graphs in their entirety can be difficult. Hierarchical graph visualisation – recursively clustering a graph’s nodes to view it at a higher level of abstraction – has thus become popular. However, this can hide important information that a user needs to understand a graph’s topology, e.g. nodes’ neighbourhoods. TugGraph addressed this by ‘separating out’ a given node’s neighbours from their hierarchy ancestors to visualise them independently. Its original implementation was straightforward, but copied parts of the hierarchy, making it slow and memory-hungry. An optimised later version, which we refer to as FastTug, avoided this, but at a cost in clarity. Optimising TugGraph without sacrificing clarity is difficult because of the need to keep every hierarchy node connected, a common challenge for graph hierarchy editing algorithms. Recently, this problem has been addressed by ‘zipping’ algorithms, multi-level split/merge algorithms that preserve hierarchy node connectedness and can be built upon for higher-level editing. In this paper, we generalise the original unzipping algorithms to implement SimpleTug, a simple, modular version of TugGraph that is easy to understand and implement, and even faster and more memory-efficient than FastTug. We formally prove its equivalence to FastTug, and show how both can be parallelised. Using our millipede hierarchical image segmentation system, we show experimentally that both the serial and parallel versions of SimpleTug are around 25% faster than their FastTug counterparts, whilst using considerably less memory. Finally, we discuss the interesting theoretical connections between TugGraph and zipping, and suggest ideas for further work.

Hierarchical segmentation from a non-increasing edge observation attribute

Hierarchical image segmentation provides region-oriented scale-spaces: sets of image segmentations at different detail levels in which the segmentations at finer levels are nested with respect to those at coarser levels. Guimarães et al. proposed a hierarchical graph-based image segmentation (HGB) method based on the Felzenszwalb-Huttenlocher dissimilarity. It computes, for each edge of a graph, the minimum scale in a hierarchy at which two regions linked by this edge should be merged according to the dissimilarity. We provide an explicit definition of the (edge-) observation attribute and Boolean criterion which are at the basis of this method and show that they are not increasing. Then, we propose an algorithm to compute all the scales for which the criterion holds true. Finally, we propose new methods to regularize the observation attribute and criterion and to set up the observation scale value of each edge of a graph, following the current trend in mathematical morphology to study criteria which are not increasing on a hierarchy. Assessments on Pascal VOC 2010 and 2012 show that these strategies lead to better segmentation results than the ones obtained with the original HGB method.

Variational shape prior segmentation with an initial curve based on image registration technique

In general images, it is practically hard to distinguish only the desired object using the conventional image segmentation methods. In many cases, we can segment the desired object by using the shape information of the object in addition to the standard image segmentation. Chan and Zhu's model is not robust to the intensity changes of objects. In this paper, we propose a novel model for the shape prior segmentation that produces robust results using the hierarchical image segmentation and an attraction term. Moreover, we adopt an image registration technique and a multi-region image segmentation to get an initial for a given shape prior. Finally, we consider the free-form deformation in obtaining the shape function from the reference shape prior for real-world images. Numerical experiments demonstrate the results independent of intensities of objects and the location of the reference shape prior. All numerical calculations are automatic and progress without any user input.

Efficient Algorithms for Hierarchical Graph-Based Segmentation Relying on the Felzenszwalb–Huttenlocher Dissimilarity

Hierarchical image segmentation provides a region-oriented scale-space, i.e. a set of image segmentations at different detail levels in which the segmentations at finer levels are nested with respect to those at coarser levels. However, most image segmentation algorithms, among which a graph-based image segmentation method relying on a region merging criterion was proposed by Felzenszwalb–Huttenlocher in 2004, do not lead to a hierarchy. In order to cope with a demand for hierarchical segmentation, Guimarães et al. proposed in 2012 a method for hierarchizing the popular Felzenszwalb–Huttenlocher method, without providing an algorithm to compute the proposed hierarchy. This paper is devoted to providing a series of algorithms to compute the result of this hierarchical graph-based image segmentation method efficiently, based mainly on two ideas: optimal dissimilarity measuring and incremental update of the hierarchical structure. Experiments show that, for an image of size 321 × 481 pixels, the most efficient algorithm produces the result in half a second whereas the most naive one requires more than 4 h.

Hierarchical Improvement of Foreground Segmentation Masks in Background Subtraction

A plethora of algorithms have been defined for foreground segmentation, a fundamental stage for many computer vision applications. In this paper, we propose a post-processing framework to improve the foreground segmentation performance of background subtraction algorithms. We define a hierarchical framework for extending segmented foreground pixels to undetected foreground object areas and for removing erroneously segmented foreground. First, we create a motion-aware hierarchical image segmentation of each frame that prevents merging foreground and background image regions. Then, we estimate the quality of the foreground mask through the fitness of the binary regions in the mask and the hierarchy of segmented regions. Finally, the improved foreground mask is obtained as an optimal labeling by jointly exploiting foreground quality and spatial color relations in a pixel-wise fully connected conditional random field. Experiments are conducted over four large and heterogeneous data sets with varied challenges (CDNET2014, LASIESTA, SABS, and BMC) demonstrating the capability of the proposed framework to improve background subtraction results.

Hierarchical Image Segmentation Ensemble for Objectness in RGB-D Images

Objectness has recently become a standard step in many computer vision tasks. Among various techniques, those based on hierarchical image segmentation play a fundamental role for developments in new data modalities. In this paper, we address the problem of objectness in RGB-D images and propose a novel and effective approach, namely, hierarchical image segmentation ensemble (HISE). Different from existing image segmentation based methods that generate object segments or proposals largely by heuristics or empirical rules, HISE learns superpixel mergings with a hierarchical tree-structured ensemble, where individual merging models of the ensemble are formed by traversing different paths of the tree, and where both the merging accuracy and proposal diversity are emphasized. Furthermore, we use efficient feature measurements that support easy integration of additional clues. Extensive experiments conducted on the benchmark NYU-v2 RGB-D and SUN RGB-D data sets show the competency of our proposed method.

The Automatic Detection of Fire Scar in Alaska using Multi-Temporal PALSAR Polarimetric SAR Data

The existing fire scar detection methods based on multi-temporal analysis only used the difference in either backscattering intensity or polarimetric characteristics between the pre- and post-fire PolSAR data to calculate the difference image (DI), and then applied an object-based image analysis approach (OBIA) to generate the fire scar binary map. These methods all ignored the polarimetric correlation between both the pre- and post-fire PolSAR data. And many parameters of the OBIA method were determined empirically. Therefore, this study proposes a new detection method, which integrates the Hotelling–Lawley trace (HLT) statistic with a hierarchical automatic image segmentation method (GGKI-MRF). The HLT can simultaneously capture changes in both polarimetry and intensity, which helps to distinguish between burned and unburned areas. And the GGKI-MRF method is developed by combining the generalized Gaussian-based Kittler–Illingworth (GG-KI) thresholding with Markov Random Field (MRF) model, which has a good performance in reducing the isolate points (false alarms) and the omission errors. The experimental results from multi-temporal and polarimetric PALSAR PolSAR data show that the proposed method can achieve high detection accuracy (i.e., the kappa coefficient of 0.81 and the overall accuracy up to 0.92).

Multiscale and Multifeature Segmentation of High-Spatial Resolution Remote Sensing Images Using Superpixels with Mutual Optimal Strategy

High spatial resolution (HSR) image segmentation is considered to be a major challenge for object-oriented remote sensing applications that have been extensively studied in the past. In this paper, we propose a fast and efficient framework for multiscale and multifeatured hierarchical image segmentation (MMHS). First, the HSR image pixels were clustered into a small number of superpixels using a simple linear iterative clustering algorithm (SLIC) on modern graphic processing units (GPUs), and then a region adjacency graph (RAG) and nearest neighbors graph (NNG) were constructed based on adjacent superpixels. At the same time, the RAG and NNG successfully integrated spectral information, texture information, and structural information from a small number of superpixels to enhance its expressiveness. Finally, a multiscale hierarchical grouping algorithm was implemented to merge these superpixels using local-mutual best region merging (LMM). We compared the experiments with three state-of-the-art segmentation algorithms, i.e., the watershed transform segmentation (WTS) method, the mean shift (MS) method, the multiresolution segmentation (MRS) method integrated in commercial software, eCognition9, on New York HSR image datasets, and the ISPRS Potsdam dataset. Computationally, our algorithm was dozens of times faster than the others, and it also had the best segmentation effect through visual assessment. The supervised and unsupervised evaluation results further proved the superiority of the MMHS algorithm.

Hierarchical image segmentation via recursive superpixel with adaptive regularity

A fast and accurate segmentation algorithm in a hierarchical way based on a recursive superpixel technique is presented. We propose a superpixel energy formulation in which the trade-off between data fidelity and regularization is dynamically determined based on the local residual in the energy optimization procedure. We also present an energy optimization algorithm that allows a pixel to be shared by multiple regions to improve the accuracy and appropriate the number of segments. The qualitative and quantitative evaluations demonstrate that our algorithm, combining the proposed energy and optimization, outperforms the conventional k -means algorithm by up to 29.10% in F -measure. We also perform comparative analysis with state-of-the-art algorithms in the hierarchical segmentation. Our algorithm yields smooth regions throughout the hierarchy as opposed to the others that include insignificant details. Our algorithm overtakes the other algorithms in terms of balance between accuracy and computational time. Specifically, our method runs 36.48% faster than the region-merging approach, which is the fastest of the comparing algorithms, while achieving a comparable accuracy.