Multi-scale Segmentations Research Articles

A multiscale graph convolutional network for change detection in homogeneous and heterogeneous remote sensing images

To date, although numerous methods of Change detection (CD) in remote sensing images have been proposed, accurately identifying changes is still a great challenge, due to the difficulties in effectively modeling the features from ground objects with different patterns. In this paper, a novel CD method based on the graph convolutional network (GCN) and multiscale object-based technique is proposed for both homogeneous and heterogeneous images. First, the object-wise high level features are obtained through a pre-trained U-net and the multiscale segmentations. Second, by treating each parcel as a node, the graph representations can be formed and then fed into the proposed multiscale graph convolutional network with each channel corresponding to one scale. The multiscale GCN propagates the label information from a small amount of labeled nodes to the other unlabeled ones. Finally, to comprehensively incorporate the information from the output channels of multiscale GCN, a fusion strategy is designed using the parent–child relationships between scales. Extensive experiments on optical, SAR and heterogeneous optical/SAR data sets demonstrate that the proposed method outperforms some state-of-the-art methods in both qualitative and quantitative evaluations.

Comparison of two object-oriented technologies for detecting land use change

This study compared two object-oriented land use change detection methods, namely detection after classification (DAC) and classification after detection (CAD) using digital elevation model, slope data, multi-temporal Thematic Mapper (TM) images in 2000 and Enhanced TM Plus images (ETM+) in 2010. Results revealed that the accuracy of the former was higher than that of the latter. A minimal discrepancy was observed in the accuracies of the two methods in detecting deciduous broad-leaved, evergreen coniferous and mixed forests; uplands; paddies; bare lands and human habitation, which can be extracted using spectral indices due to large spectrum differences. The accuracy of DAC was higher than that of CAD for areas with similar spectra, such as industrial and traffic lands, green shrubs, rivers, channels, reservoirs/pools and lake, because the former can fully utilise various information, including spectral, spatial, structural, textural and contextual information, during the two-stage classification. Moreover, the change area boundary was not limited and adjustable during the classification process. DAC can overcome the smoothing effect to a great extent using multi-scale segmentations and multiple indicators for detection. Although DAC can yield better results than CAD can, the former was more time-consuming than the latter because of its two-stage classification. Therefore, the hybrid of the two methods could be the appropriate approach.

Adaptive MultiScale Segmentations for Hyperspectral Image Classification

The number of superpixels (i.e., segmentation scale) is crucial for spectral–spatial hyperspectral image (HSI) classification. Existing methods always set the segmentation scale through a manually experimental strategy, which is time-consuming and unsuitable for various complicated practical applications. The information fusion of complementary multiple scales is proven to be more effective than the single scale for HSI classification, but the scale level is still set manually. In this article, we propose a novel adaptive multiscale segmentations (AMSs) method that can automatically provide a set of suitable scales that are adapted to different hyperspectral data. Specifically, based on the assumption that the segmentation scale of HSI is related to the image complexity itself, the texture ratio and the number of land cover classes are used to examine a candidate scale pool. A good scale means that it contains a small spectral difference between pixels within the same superpixel (intrasuperpixel discrimination index) and a large discrepancy between neighboring superpixels (intersuperpixel discrimination index). Thus, an intra–interscale discrimination index is defined and applied to depict the characteristics of the scale. Then, the scale with the best intra–inter discrimination index, which usually has satisfactory performance, is treated as the initially selected scale. The remaining suitable scales are iteratively compared with the selected ones and then added to the target scale pool, until the newly added scale can no longer provide significantly complementary information. Extensive experimental results on three HSI data sets have demonstrated the effectiveness of the proposed AMS when compared with state-of-the-art methods.

Object-specific optimization of hierarchical multiscale segmentations for high-spatial resolution remote sensing images

Accurate segmentation of high-spatial resolution remote sensing images remains a challenging problem for geographic object-based image analysis. An object-specific optimization method for hierarchical multiscale segmentations is proposed in this study by fusing multiple segmentations into an optimized segmentation with specific consideration of each object. Based on a segment tree model representing hierarchical multiscale segmentations, the framework of object-specific optimization is achieved by identifying and fusing the meaningful nodes in each path originating from a leaf node. Within the optimization framework, an optimization measure for identifying meaningful node is designed according to the maximum change of homogeneity in a path. The proposed optimization method is experimentally validated to hold the advantage of improving segmentation accuracy by the manner of object-specific optimization as well as the potential of automatically producing optimized segmentation for successive object-based analysis.

Combined multiscale segmentation convolutional neural network for rapid damage mapping from postearthquake very high-resolution images

Classifying land use from postearthquake very high-resolution (VHR) images is challenging due to the complexity of objects in Earth surface after an earthquake. Convolutional neural network (CNN) exhibits satisfied performance in differentiating complex postearthquake objects, thanks to its automatic extraction of high-level features and accurate identification of target geo-objects. Nevertheless, in view of the scale variance of natural objects, the fact that CNN suffers from the fixed receptive field, the reduced feature resolution, and the insufficient training sample has severely contributed to its limitation in the rapid damage mapping. Multiscale segmentation technique is considered as a promising solution as it can generate the homogenous regions and provide the boundary information. Therefore, we propose a combined multiscale segmentation convolutional neural network (CMSCNN) method for postearthquake VHR image classification. First, multiscale training samples are selected based on segments derived from the multiscale segmentation. Then, CNN is directly trained to classify the original image to further produce the preliminary classification maps. To enhance the localization accuracy, the output of CNN is further refined using multiscale segmentations from fine to coarse iteratively to obtain the multiscale classification maps. As a result, the combination strategy is able to capture objects and image context simultaneously. Experimental results show that the proposed CMSCNN method can reflect the multiscale information of complex scenes and obtain satisfied classification results for mapping postearthquake damage using VHR remote sensing images.

Self-adaptive segmentation of satellite images based on a weighted aggregation approach

Image segmentation is a decisive process in object-based image analysis, while the uncertainty of segmentation scales can significantly influence the results. To resolve this issue, this study proposes a Self-Adaptive Segmentation (SAS) method which bridges the gap between the inherent scale and segmentation scale of each object. Firstly, SAS is defined as a variable-scale segmentation approach, aiming at generating local optimum results. It is then implemented based on Segmentation by Weighted Aggregation (SWA) method and optimized by selfhood scale estimation technique. Secondly, two images derived from WorldView-2 (an urban area) and Landsat-8 (a farmland area) are employed to verify the effectiveness and adaptability of this method. Thirdly, it is further compared with SWA and mean shift method by reference to two evaluations, including an unsupervised evaluation, i.e., Global Score (GS), and a supervised one named Geometry-based comparison. The experimental results indicate the SAS method presented in this study is effective in solving the scale issues in multiscale segmentations. It shows higher accuracy than SWA through assigning an appropriate segmentation scale for each object. Moreover, the segmentation results achieved by SAS method in both study areas are more accurate than those by mean shift method, which demonstrates the stable performance of SAS method across diverse scenes and data sets. This advantage is more obvious in farmland area than in urban scene, according to the accuracy assessment results.

Regional Multi-Scale Approach for Visually Pleasing Explanations of Deep Neural Networks

Recently, many methods to interpret and visualize deep neural network predictions have been proposed and significant progress has been made. However, a more class-discriminative and visually pleasing explanation is required. Thus, this paper proposes a region-based approach that estimates feature importance in terms of appropriately segmented regions. By fusing the saliency maps generated from multi-scale segmentations, a more class-discriminative and visually pleasing map is obtained. We incorporate this regional multi-scale concept into a prediction difference method that is model-agnostic. An input image is segmented in several scales using the super-pixel method, and exclusion of a region is simulated by sampling a normal distribution constructed using the boundary prior. The experimental results demonstrate that the regional multi-scale method produces much more class-discriminative and visually pleasing saliency maps.

Toward combining thematic information with hierarchical multiscale segmentations using tree Markov random field model

It has been a common idea to produce multiscale segmentations to represent the various geographic objects in high-spatial resolution remote sensing (HR) images. However, it remains a great challenge to automatically select the proper segmentation scale(s) just according to the image information. In this study, we propose a novel way of information fusion at object level by combining hierarchical multiscale segmentations with existed thematic information produced by classification or recognition. The tree Markov random field (T-MRF) model is designed for the multiscale combination framework, through which the object type is determined as close as the existed thematic information. At the same time, the object boundary is jointly determined by the thematic labels and the multiscale segments through the minimization of the energy function. The benefits of the proposed T-MRF combination model include: (1) reducing the dependence of segmentation scale selection when utilizing multiscale segmentations; (2) exploring the hierarchical context naturally imbedded in the multiscale segmentations. The HR images in both urban and rural areas are used in the experiments to show the effectiveness of the proposed combination framework on these two aspects.

The Euclidean embedding learning based on convolutional neural network for stereo matching

Stereo matching is one of the most important and fundamental topics in computer vision. The calculation of matching cost plays a very important role for stereo matching algorithms. The stereo matching algorithm proposed by Zbontar and LeCun focusing on the training of the matching cost has showed the good performance of the convolutional neural network. Unfortunately, computing a convolutional neural network for matching cost is computationally very expensive. This paper proposes a method based on learning a Euclidean embedding using a convolutional neural network with a triplet-based loss function, where the matching cost is directly computed by the squared L2 distances between two vectors in the embedding space. The cost is refined by Semiglobal Matching with an adaptive smoothness constraint based on multi-scale segmentations. The proposed method has a comparable performance with the state-of-the-art algorithms, and it overcomes a problem of heavy computation. The proposed method takes only about 5 s for predicting a single image pair, where the computing of convolutional neural networks needs less than 2 s with CPU, that is much faster than the algorithm by Zbontar and LeCun where the computing of convolutional neural network takes 67 s with GPU.

Toward Evaluating Multiscale Segmentations of High Spatial Resolution Remote Sensing Images

Object-based analysis of high spatial resolution remote sensing images addresses the matter of multiscale segmentation. However, existing segmentation evaluation methods mainly focus on single-scale segmentation. In this paper, we examine the issue of supervised multiscale segmentation evaluation and propose two discrepancy measures to determine the manner in which geographic objects are delineated by multiscale segmentations. A QuickBird scene in Hangzhou, China, is used to conduct the evaluation. The results reveal the effectiveness of the proposed measures, in terms of method comparison and parameter optimization, for multiscale segmentation of high spatial resolution images. Moreover, meaningful indications for selecting suitable multiple segmentation scales are presented. The proposed measures are applicable to performance evaluation and parameter optimization for multiscale segmentation algorithms.

Optimal segmentation of a high-resolution remote-sensing image guided by area and boundary

Image segmentation is the premise of object-based image analysis (OBIA), and obtaining an optimal segmentation result has been a desire for many researchers. This article proposes an optimal segmentation method for a high-resolution remote-sensing image that is guided by spatial features of area and boundary. This method achieves an optimal result through stepwise refinement on multi-scale segmentations. First, boundary strength is integrated into the choice for the optimal scale based on an improved unsupervised evaluation. Then, under-segmented objects (USOs) and over-segmented objects (OSOs) at the selected optimal scale are identified using a heterogeneity histogram and a slider-like threshold with the guidance of area and boundary. Finally, the corresponding objects, in a specific finer segmentation, are taken to replace the USOs at the optimal scale, and then the USOs and OSOs are refined by an effective merging mechanism. A heterogeneity-change-based merging criterion considering boundary, shape, spectral, and texture features is constructed for the merging of neighbouring objects. The proposed method is more effective than the unsupervised image segmentation evaluation and refinement (UISER) method as it uses spatial features to guide optimal choice of scale, and USO and OSO identification and refinement. Comparative experiments show that the spatial features used in the proposed method are effective for achieving an enhanced segmentation result.

Saliency Detection with Multi-Scale Superpixels

We propose a salient object detection algorithm via multi-scale analysis on superpixels. First, multi-scale segmentations of an input image are computed and represented by superpixels. In contrast to prior work, we utilize various Gaussian smoothing parameters to generate coarse or fine results, thereby facilitating the analysis of salient regions. At each scale, three essential cues from local contrast, integrity and center bias are considered within the Bayesian framework. Next, we compute saliency maps by weighted summation and normalization. The final saliency map is optimized by a guided filter which further improves the detection results. Extensive experiments on two large benchmark datasets demonstrate the proposed algorithm performs favorably against state-of-the-art methods. The proposed method achieves the highest precision value of 97.39% when evaluated on one of the most popular datasets, the ASD dataset.

Object-based approach to national land cover mapping using HJ satellite imagery

To meet the carbon storage estimate in ecosystems for a national carbon strategy, we introduce a consistent database of China land cover. The Chinese Huan Jing (HJ) satellite is proven efficient in the cloud-free acquisition of seasonal image series in a monsoon region and in vegetation identification for mesoscale land cover mapping. Thirty-eight classes of level II land cover are generated based on the Land Cover Classification System of the United Nations Food and Agriculture Organization that follows a standard and quantitative definition. Twenty-four layers of derivative spectral, environmental, and spatial features compose the classification database. Object-based approach characterizing additional nonspectral features is conducted through mapping, and multiscale segmentations are applied on object boundary match to target real-world conditions. This method sufficiently employs spatial information, in addition to spectral characteristics, to improve classification accuracy. The algorithm of hierarchical classification is employed to follow step-by-step procedures that effectively control classification quality. This algorithm divides the dual structures of universal and local trees. Consistent universal trees suitable to most regions are performed first, followed by local trees that depend on specific features of nine climate stratifications. The independent validation indicates the overall accuracy reaches 86%.

Multi-scale segmentation approach for object-based land-cover classification using high-resolution imagery

Image segmentation is a basic and important procedure in object-based classification of remote-sensing data. This study presents an approach to multi-scale optimal segmentation (OS), given that single-scale segmentation may not be the most suitable approach to map a variety of land-cover types characterized by various spatial structures; it objectively measures the appropriate segmentation scale for each object at various scales and projects them onto a single layer. A 1.8 m spatial resolution Worldview-2 image was used to perform successive multi-scale segmentations. The pixel standard deviation of an object was used to measure the optimal scale that occurred on the longest, feature unchanged scale range during multi-scale segmentation. Results indicate that the classification of multi-scale object OS can improve the overall accuracy by five percentage points compared to traditional single segmentation.

Boundary-constrained multi-scale segmentation method for remote sensing images

Image segmentation is the key step of Object-Based Image Analysis (OBIA) in remote sensing. This paper proposes a Boundary-Constrained Multi-Scale Segmentation (BCMS) method. Firstly, adjacent pixels are aggregated to generate initial segmentation according to the local best region growing strategy. Then, the Region Adjacency Graph (RAG) is built based on initial segmentation. Finally, the local mutual best region merging strategy is applied on RAG to produce multi-scale segmentation results. During the region merging process, a Step-Wise Scale Parameter (SWSP) strategy is proposed to produce boundary-constrained multi-scale segmentation results. Moreover, in order to improve the accuracy of object boundaries, the property of edge strength is introduced as a merging criterion. A set of high spatial resolution remote sensing images is used in the experiment, e.g., QuickBird, WorldView, and aerial image, to evaluate the effectiveness of the proposed method. The segmentation results of BCMS are compared with those of the commercial image analysis software eCognition. The experiment shows that BCMS can produce nested multi-scale segmentations with accurate and smooth boundaries, which proves the robustness of the proposed method.

Multi-Scale Semantic Model for Unsupervised Object Segmentation

It is difficult to segment instances of object classes accurately unsupervised in images, because of the complexity of structures, inter-class differences, background interference and so on. A multi-scale semantic model method is proposed to overcome the disadvantages existing in most of the relative methods. This method uses generative model to deal with the objects obtained by multi-scale segmentations instead of whole image, and calculates kinds of visual features to mine the topic information of every object. In the segmentation process, a semantic correlative function of every segment block based on KL divergence is built up and minimized to select the object correct regions. Experimental results demonstrate the effectiveness of the proposed method.

Unsupervised image segmentation evaluation and refinement using a multi-scale approach

In this study, a multi-scale approach is used to improve the segmentation of a high spatial resolution (30 cm) color infrared image of a residential area. First, a series of 25 image segmentations are performed in Definiens Professional 5 using different scale parameters. The optimal image segmentation is identified using an unsupervised evaluation method of segmentation quality that takes into account global intra-segment and inter-segment heterogeneity measures (weighted variance and Moran’s I, respectively). Once the optimal segmentation is determined, under-segmented and over-segmented regions in this segmentation are identified using local heterogeneity measures (variance and Local Moran’s I). The under- and over-segmented regions are refined by (1) further segmenting under-segmented regions at finer scales, and (2) merging over-segmented regions with spectrally similar neighbors. This process leads to the creation of several segmentations consisting of segments generated at three different segmentation scales. Comparison of single- and multi-scale segmentations shows that identifying and refining under- and over-segmented regions using local statistics can improve global segmentation results.

Patch‐type Segmentation of Voxel Shapes using Simplified Surface Skeletons

AbstractWe present a new method for decomposing a 3D voxel shape into disjoint segments using the shape's simplified surface‐skeleton. The surface skeleton of a shape consists of 2D manifolds inside its volume. Each skeleton point has a maximally inscribed ball that touches the boundary in at least two contact points. A key observation is that the boundaries of the simplified fore‐ and background skeletons map one‐to‐one to increasingly fuzzy, soft convex, respectively concave, edges of the shape. Using this property, we build a method for segmentation of 3D shapes which has several desirable properties. Our method segments both noisy shapes and shapes with soft edges which vanish over low‐curvature regions. Multiscale segmentations can be obtained by varying the simplification level of the skeleton. We present a voxel‐based implementation of our approach and illustrate it on several realistic examples.

Multiscale Active Contours

We propose a new multiscale image segmentation model, based on the active contour/snake model and the Polyakov action. The concept of scale, general issue in physics and signal processing, is introduced in the active contour model, which is a well-known image segmentation model that consists of evolving a contour in images toward the boundaries of objects. The Polyakov action, introduced in image processing by Sochen-Kimmel-Malladi in Sochen et al. (1998), provides an efficient mathematical framework to define a multiscale segmentation model because it generalizes the concept of harmonic maps embedded in higher-dimensional Riemannian manifolds such as multiscale images. Our multiscale segmentation model, unlike classical multiscale segmentations which work scale by scale to speed up the segmentation process, uses all scales simultaneously, i.e. the whole scale space, to introduce the geometry of multiscale images in the segmentation process. The extracted multiscale structures will be useful to efficiently improve the robustness and the performance of standard shape analysis techniques such as shape recognition and shape registration. Another advantage of our method is to use not only the Gaussian scale space but also many other multiscale spaces such as the Perona-Malik scale space, the curvature scale space or the Beltrami scale space. Finally, this multiscale segmentation technique is coupled with a multiscale edge detecting function based on the gradient vector flow model, which is able to extract convex and concave object boundaries independent of the initial condition. We apply our multiscale segmentation model on a synthetic image and a medical image.