Multi-scale Superpixel Research Articles

Hierarchical homogeneity-based superpixel segmentation: application to hyperspectral image analysis

ABSTRACT Hyperspectral image (HI) analysis approaches have recently become increasingly complex and sophisticated. Recently, the combination of spectral-spatial information and superpixel techniques have addressed some hyperspectral data issues, such as the higher spatial variability of spectral signatures and dimensionality of the data. However, most existing superpixel approaches do not account for specific HI characteristics resulting from its high spectral dimension. In this work, we propose a multiscale superpixel method that is computationally efficient for processing hyperspectral data. The Simple Linear Iterative Clustering (SLIC) oversegmentation algorithm, on which the technique is based, has been extended hierarchically. Using a novel robust homogeneity testing, the proposed hierarchical approach leads to superpixels of variable sizes but with higher spectral homogeneity when compared to the classical SLIC segmentation. For validation, the proposed homogeneity-based hierarchical method was applied as a preprocessing step in the spectral unmixing and classification tasks carried out using, respectively, the Multiscale sparse Unmixing Algorithm (MUA) and the CNN-Enhanced Graph Convolutional Network (CEGCN) methods. Simulation results with both synthetic and real data show that the technique is competitive with state-of-the-art solutions.

A Dual-Branch Fusion of a Graph Convolutional Network and a Convolutional Neural Network for Hyperspectral Image Classification.

Semi-supervised graph convolutional networks (SSGCNs) have been proven to be effective in hyperspectral image classification (HSIC). However, limited training data and spectral uncertainty restrict the classification performance, and the computational demands of a graph convolution network (GCN) present challenges for real-time applications. To overcome these issues, a dual-branch fusion of a GCN and convolutional neural network (DFGCN) is proposed for HSIC tasks. The GCN branch uses an adaptive multi-scale superpixel segmentation method to build fusion adjacency matrices at various scales, which improves the graph convolution efficiency and node representations. Additionally, a spectral feature enhancement module (SFEM) enhances the transmission of crucial channel information between the two graph convolutions. Meanwhile, the CNN branch uses a convolutional network with an attention mechanism to focus on detailed features of local areas. By combining the multi-scale superpixel features from the GCN branch and the local pixel features from the CNN branch, this method leverages complementary features to fully learn rich spatial-spectral information. Our experimental results demonstrate that the proposed method outperforms existing advanced approaches in terms of classification efficiency and accuracy across three benchmark data sets.

SPMUNet: Semantic segmentation of citrus surface defects driven by superpixel feature

The segmentation of surface defects in citrus fruit makes a significant contribution to quality assessment and post-harvest processing. However, semantic segmentation of citrus surface defects remains challenging owing to the complexity of defect features, unclear defect boundaries and semantic gaps in models. Therefore, a superpixel driven and multiscale channel cross-fusion transformer (MCCT)-based UNet (named SPMUNet) was proposed to segment citrus surface defects. The model first created a multiscale superpixel region feature (MSRF) branch using the simple linear iterative cluster (SLIC) algorithm to form the key features embedded in the downsampling layer. Then, the proposed Superpixel Loss optimized the loss function based on superpixel grid to provide error information for boundary segmentation. In addition, the MCCT module was used to improve the skip connection layers, and feature fusion was achieved by collaborative learning. Finally, the performance of the model was validated through ablation studies, feature map visualization, and comparison experiments. The results showed that the mPA, mIoU, and dice coefficients of SPMUNet for citrus surface segmentation were 89.1%, 83.8%, and 88.6%, respectively. Compared with the baseline, the proposed MSRF branch, MCCT, and Superpixel Loss improved mIoU by 2.6%, 1.3%, and 4.2%, respectively. And the effectiveness of the proposed methods was also validated by the feature map visualization. Furthermore, the proposed SPMUNet achieved optimal segmentation performance compared to other semantic segmentation models (UNet, ResUNet++, DeepLabv3+, SwinUNet, TransUNet). SPMUNet also excelled in the segmentation of small targets and boundaries, improving by 10.2% and 7.0%, respectively, over the baseline. The improvement in segmentation performance demonstrated the effectiveness of the selected superpixel features. The proposed citrus surface defect segmentation model achieved high-precision segmentation and serves as a technical reference for citrus sorting and quality assessment.

Superpixel-based multi-scale multi-instance learning for hyperspectral image classification

Superpixels can define meaningful local regions within a hyperspectral image (HSI) and have become the building blocks of various HSI classification methods. The superpixels in HSIs consist of multiple spectral pixels, sharing a similar structure with the data in multi-instance learning (MIL). However, the potential of MIL methods in the field of HSI classification has been rarely explored. In this paper, we propose the superpixel-based multi-scale multi-instance learning (MSMIL) framework, enhancing the superpixel representation with MIL for the first time. Segmenting the HSIs with superpixels, MSMIL converts the HSI classification into MIL problems and extracts superpixel representations via the MIL method, namely multi-instance factor analysis (MIFA). Compared with the existing methods focusing exclusively on the local information, MIFA utilizes the deviations from an overall generative model to describe the superpixels, retaining both the local and the global information. Moreover, MSMIL introduces multi-scale superpixels and a spectral–spatial decision fusion strategy for further refinement, where the results of multi-scale superpixel maps are weighted according to prediction certainty and spatial consistency. The proposed method is evaluated on four benchmark datasets and achieves competitive results. For example, MSMIL outperforms the comparison methods with a margin of 5% overall accuracy on the Indian Pines dataset, when 2% pixels are selected as the training set.

Enhanced Atrous Extractor and Self-Dynamic Gate Network for Superpixel Segmentation

A superpixel is a group of pixels with similar low-level and mid-level properties, which can be seen as a basic unit in the pre-processing of remote sensing images. Therefore, superpixel segmentation can reduce the computation cost largely. However, all the deep-learning-based methods still suffer from the under-segmentation and low compactness problem of remote sensing images. To fix the problem, we propose EAGNet, an enhanced atrous extractor and self-dynamic gate network. The enhanced atrous extractor is used to extract the multi-scale superpixel feature with contextual information. The multi-scale superpixel feature with contextual information can solve the low compactness effectively. The self-dynamic gate network introduces the gating and dynamic mechanisms to inject detailed information, which solves the under-segmentation effectively. Massive experiments have shown that our EAGNet can achieve the state-of-the-art performance between k-means and deep-learning-based methods. Our methods achieved 97.61 in ASA and 18.85 in CO on the BSDS500. Furthermore, we also conduct the experiment on the remote sensing dataset to show the generalization of our EAGNet in remote sensing fields.

Pellet image segmentation model of superpixel feature-based support vector machine in digital twin

A digital twin model based on superpixel features is established to solve the problem of noise and similar gray values between foreground and background of pellet images. With superpixel as the basic unit of segmentation, the influence of single pixel on segmentation results is reduced, and allows for higher segmentation accuracy. The gray-level co-occurrence matrix is used to represent the superpixel characteristic information, and the color moment and gray level distribution are combined to comprehensively characterize the superpixel. Through principal component analysis and correlation analysis, The feature compression of superpixel is realized, and the computational efficiency is improved. The superpixel binary classification data set is built, and the multi-dimensional feature information of superpixel is extracted as input vector to train the binary classification model of SVM, and the image segmentation problem is transformed into foreground and background classification problem. A multi-scale superpixel segmentation boundary optimization method is proposed to further refine the boundary region of foreground and background. A four-neighborhood search algorithm is proposed to reduce the missegmentation rate of edge superpixels. Experimental results show that the accuracy of the proposed method can reach 95.87%, the precision of image edge segmentation is high, and the foreground and background of granular image are accurately segmented. The digital twin model is established which can provide basis for the subsequent visual analysis and decision.

A spatially correlated fractional integral-based method for denoising geiger-mode avalanche photodiode light detection and ranging depth images

Geiger-mode avalanche photodiode (GM-APD) light detection and ranging (LiDAR) target echo signals are susceptible to interference from atmospheric backscatter, daylight, and other noise, and extracted target depth images contain a large amount of anomalous noise. Accordingly, this paper devises a method based on a spatially correlated fractional integral for denoising GM-APD LiDAR depth images. First, a multi-scale superpixel fusion algorithm is employed to perform null-pixel complementation on a target depth image extracted using histogram statistics. Next, a pixel neighborhood spatial correlation kernel function is used to optimize the Grünwald–Letnikov fractional integral operator to correct the large amount of anomalous noise and thus achieve GM-APD depth image denoising. Simulation and experimental results demonstrate that the denoising performance of the algorithm on GM-APD LiDAR depth images is significantly better than that of median filtering and bilateral filtering, with at least 22.8 % and 4.8 % improvements in the target reduction degree (K) and peak signal-to-noise ratio (PSNR), respectively. In addition, the K and PSNR reach 91.36 % and 18.0241 dB, respectively, with 25 statistical frames in an outdoor imaging experiment. This demonstrates that the algorithm can effectively denoise GM-APD LiDAR depth images with few statistical frames and thus improve the frame rate of GM-APD LiDAR imaging.

Background subtraction via regional multi-feature-frequency model in complex scenes

Although a wide variety of background subtraction methods has been proposed in recent years, none has been able to fully address multi-scale moving objects and dynamic background in real surveillance tasks. In this paper, a novel and effective background subtraction method, named regional multi-feature-frequency (RMFF), is proposed to detect multi-scale moving objects under dynamic background. Unlike many existing methods construct background model using simple multi-feature combinations, RMFF exploits the spatiotemporal cues of multi-feature as well as superpixels at each scale, thus allowing for more robust information to be exploited for background modeling. Specifically, the spatial relationship between pixels in a neighborhood and the frequencies of features over time are first exploited, enabling accurate detection of moving objects while ignoring most dynamic background changes. Then, the use of multi-scale superpixels for exploiting the structural information existing in real-world scenes further enhances robustness to multi-scale objects and environmental variations. Finally, an adaptive strategy is employed to dynamically adjust the foreground/background segmentation threshold for each region without user intervention. This adaptive threshold is defined for each region separately, and can adjust dynamically based on continuous monitoring of the background changes, thereby effectively reducing potential segmentation noise. Experiments on the 2014 version of the ChangeDetection.net dataset demonstrate that the proposed method outperforms the 12 state-of-the-art algorithms in terms of overall F-Measure and performs effectively in many complex scenes. Consequently, it is verified that the developed approach is feasible and useful for robust application in practical video surveillance.

Multi-scale Superpixel based Hierarchical Attention model for brain CT classification

Brain CT image classification is critical for assisting brain disease diagnosis. The brain CT images contain much noisy information, and the lesions are unstable in shape and location, making the classification task more difficult when using conventional CNN models. In this paper, we propose a novel Multi-scale Superpixel based Hierarchical Attention (MSHA) model for brain CT classification by introducing the multi-scale superpixels to a hierarchical fusion structure to remove noise and help the model focus on the lesion areas. MSHA contains three modules: (1) a Semantic-level Information Extractor that extracts appearance and geometry information based on the superpixel of the image, (2) a Mixed Multi-head Attention module that obtains the mixed attention features from the semantic-level information, and (3) a Hierarchical Fusion Structure that fuses the multi-scale attention features from coarse to fine. Experiments on the brain CT dataset demonstrate the effectiveness of the proposed model.

A Dual Neighborhood Hypergraph Neural Network for Change Detection in VHR Remote Sensing Images

The very high spatial resolution (VHR) remote sensing images have been an extremely valuable source for monitoring changes occurring on the Earth’s surface. However, precisely detecting relevant changes in VHR images still remains a challenge, due to the complexity of the relationships among ground objects. To address this limitation, a dual neighborhood hypergraph neural network is proposed in this article, which combines multiscale superpixel segmentation and hypergraph convolution to model and exploit the complex relationships. First, the bi-temporal image pairs are segmented under two scales and fed to a pre-trained U-net to obtain node features by treating each object under the fine scale as a node. The dual neighborhood is then defined using the father-child and adjacent relationships of the segmented objects to construct the hypergraph, which permits models to represent higher-order structured information far more complex than the conventional pairwise relationships. The hypergraph convolutions are conducted on the constructed hypergraph to propagate the label information from a small amount of labeled nodes to the other unlabeled ones by the node-edge-node transformation. Moreover, to alleviate the problem of imbalanced sampling, the focal loss function is adopted to train the hypergraph neural network. The experimental results on optical, SAR and heterogeneous optical/SAR data sets demonstrate that the proposed method offersbetter effectiveness and robustness compared to many state-of-the-art methods.

Image classification using graph neural network and multiscale wavelet superpixels

Prior studies using graph neural networks (GNNs) for image classification have focused on graphs generated from a regular grid of pixels or similar-sized superpixels. In the latter, a single target number of superpixels is defined for an entire dataset irrespective of differences across images and their intrinsic multiscale structure. On the contrary, this study investigates image classification using graphs generated from an image-specific number of multiscale superpixels. We propose WaveMesh, a new wavelet-based superpixeling algorithm, where the number and sizes of superpixels in an image are systematically computed based on its content. WaveMesh superpixel graphs are structurally different from similar-sized superpixel graphs. We use SplineCNN, a state-of-the-art network for image graph classification, to compare WaveMesh and similar-sized superpixels. Using SplineCNN, we perform extensive experiments on three benchmark datasets under three local-pooling settings: 1) no pooling, 2) GraclusPool, and 3) WavePool, a novel spatially heterogeneous pooling scheme tailored to WaveMesh superpixels. Our experiments demonstrate that SplineCNN learns from multiscale WaveMesh superpixels on-par with similar-sized superpixels. In all WaveMesh experiments, GraclusPool performs poorer than no pooling / WavePool, showing that poor cluster assignment negatively affects the performance of the network while learning from multiscale superpixels.

Backpack detection model using multi-scale superpixel and body-part segmentation

Abstract A backpack is a type of carried object (CO) widely used for various purposes because of its practicality. Various valuable items such as wallets, laptops, cameras, and cellphones may be kept in backpacks. Detecting backpacks in video surveillance is challenging due to their varying shapes, sizes, and colors. The process of localizing the area of the backpack in the image is a critical stage and dramatically influences the success of detection. This paper focuses on the process of localizing the backpack area through a multi-scale segmentation approach, where different scales are intended to detect the various size of the backpacks. Based on the assumption that the backpack is generally located above the bend line, the body-part method is then used to select superpixels. The selected superpixel feature is then extracted and used to train the model. Model testing is carried out in two scenarios. In the first scenario, the model is tested using the HOG (histogram of oriented gradients) feature, while in the second scenario, the model is tested using a combination of the HOG and histogram features. The experiment results show that on the DIKE20 dataset, the proposed model obtained an average F1 score of 69%. On PETS2006 and i-LIDS datasets, the proposed model shows an average F1 score of 68%, better than the average F1 score obtained by the state-of-the-art method.

Multiscale Superpixel Guided Discriminative Forest for Hyperspectral Anomaly Detection

Recently, the isolation forest (IF) methods have received increasing attention for their promising performance in hyperspectral anomaly detection (HAD). However, limited by the ability of exploiting spatial-spectral information, existing IF-based methods suffer from a lot of false alarms and disappointing performance of detecting local anomalies. To overcome the two problems, a multiscale superpixel guided discriminative forest method is proposed for HAD. First, the multiscale superpixel segmentation is employed to generate some homogeneous regions, and it can effectively extract spatial information to guide anomaly detection for the discriminative forest in local areas. Then, a novel discriminative forest (DF) model with the gain split criterion is designed, which enhances the sensitivity of the DF to local anomalies by the utilization of multi-dimension spectral bands for node division; meanwhile, the acceptable range of hyperplane attribute values is introduced to capture any unseen anomaly pixels that are out-of-range in the evaluation stage. Finally, for the high false alarm rate situation in the existing IF-based algorithms, the multiscale fusion with guided filtering is put forward to refine the initial detection results from the DF. In addition, the extensive experimental results on four real hyperspectral datasets demonstrate the effectiveness of the proposed method.

Hierarchical Superpixel Segmentation for PolSAR Images Based on the Boruvka Algorithm

Superpixel segmentation for polarimetric synthetic aperture radar (PolSAR) images plays a key role in remote-sensing tasks, such as ship detection and land-cover classification. However, the existing methods cannot directly generate multi-scale superpixels in a hierarchical style and they will take a long time when multi-scale segmentation is executed separately. In this article, we propose an effective and accurate hierarchical superpixel segmentation method, by introducing a minimum spanning tree (MST) algorithm called the Boruvka algorithm. To accurately measure the difference between neighboring pixels, we obtain the scattering mechanism information derived from the model-based refined 5-component decomposition (RFCD) and construct a comprehensive dissimilarity measure. In addition, the edge strength map and homogeneity measurement are considered to make use of the structural and spatial distribution information in the PolSAR image. On this basis, we can generate superpixels using the distance metric along with the MST framework. The proposed method can maintain good segmentation accuracy at multiple scales, and it generates superpixels in real time. According to the experimental results on the ESAR and AIRSAR datasets, our method is faster than the current state-of-the-art algorithms and preserves somewhat more image details in different segmentation scales.

Foreground Detection Based on Superpixel and Semantic Segmentation.

Foreground detection is an essential step in computer vision and video processing. Accurate foreground object extraction is crucial for subsequent high-level tasks such as target recognition and tracking. Although many foreground detection algorithms have been proposed, foreground detection in complex scenes is still a challenging problem. This paper presents a foreground detection algorithm based on superpixel and semantic segmentation. It first uses multiscale superpixel segmentation to obtain the initial foreground mask. At the same time, a semantic segmentation network is applied to separate potential foreground objects, and then use the defined rules to combine the results of superpixel and semantic segmentation to get the final foreground object. Finally, the background model is updated with the refined foreground result. Experiments on the CDNet2014 dataset demonstrate the effectiveness of the proposed algorithm, which can accurately segment foreground objects in complex scenes.

Multiscale Superpixel-Based Fine Classification of Crops in the UAV-Based Hyperspectral Imagery

As an effective approach to obtaining agricultural information, the remote sensing technique has been applied in the classification of crop types. The unmanned aerial vehicle (UAV)-manned hyperspectral sensors provide imagery with high spatial and high spectral resolutions. Moreover, the detailed spatial information, as well as abundant spectral properties of UAV-manned hyperspectral imagery, opens a new avenue to the fine classification of crops. In this manuscript, multiscale superpixel-based approaches are proposed for the fine identification of crops in the UAV-manned hyperspectral imagery. Specifically, the multiscale superpixel segmentation is performed and a series of superpixel maps can be obtained. Then, the multiscale information is integrated into image classification by two strategies, namely pre-processing and post-processing. For the pre-processing strategy, the superpixel is regarded as the minimum unit for image classification, whose feature is obtained by using the average of spectral values of pixels within it. At each scale, the classification is performed on the basis of the superpixel. Then, the multiscale classification results are combined to generate the final map. For the post-processing strategy, the pixel-wise classification is implemented to obtain the label and posterior probabilities of each pixel. Subsequently, the superpixel-based voting is conducted at each scale, and these obtained voting results are fused to generate the multiscale voting result. To evaluate the effectiveness of the proposed approaches, three open-sourced hyperspectral UAV-manned datasets are employed in the experiments. Meanwhile, seven training sets with different numbers of labeled samples and two classifiers are taken into account for further analysis. The results demonstrate that the multiscale superpixel-based approaches outperform the single-scale approaches. Meanwhile, the post-processing strategy is superior to the pre-processing strategy in terms of higher classification accuracies in all the datasets.

Ship detection in SAR images by saliency analysis of multiscale superpixels

ABSTRACT This paper proposes a new ship detection algorithm via saliency analysis on multiscale superpixels. First, Synthetic Aperture Radar (SAR) imagery is over-segmented into many homogenous regions, called superpixels. In contrast to prior work, these superpixels are generated with various scales to handle size variation of ships in the SAR imagery. At each scale, local contrast and ship geometrical characteristics are considered to enhance the saliency of ship pixels. Then, the final saliency map is computed by integrating saliency values of all scales. Experimental results on two real SAR images demonstrate the efficiency of the proposed algorithm.

Intensity-guided depth image estimation in long-range lidar

Long-range lidar systems usually record large but extremely sparse data cubes. It is a great challenge to estimate accurate depth images of the photon-starved regime with fewer memory requirements and lower computational complexity. An intensity-guided method is introduced to estimate the depth image by using temporal-spatial correlation of the reflected signals. Multi-scale superpixels and fast time-domain windows are established in the preprocessing step, leading to smaller data cubes with reduced empty and noisy pixels. To strike a balance between smoothness and preserving sharp edges, the fast-converging alternating direction method of multipliers (ADMM) is used in the modified cost function to estimate depth images. Experimental results show that the proposed method yields better depth estimates than other state-of-the-art methods, especially for long-rang targets of field experiments with a low signal return level of ∼1 photon per pixel.

Hyperspectral Classification Based on Coupling Multiscale Super-Pixels and Spatial Spectral Features

Hyperspectral remotely sensed images contain not only the spatial information of ground objects, but also their rich spectral information. Effectively improve the classification accuracy of hyperspectral images (HSIs), the purpose is to accurately grasp the current land resource utilization information, which is of great significance to the formulation and implementation of future land and space planning. Existing research on the classification of hyperspectral remotely sensed images uses a single-scale superpixel method for image segmentation. The optimal number of superpixels cannot be determined, image details may get missed, and a single kernel matrix cannot represent the information from multiple features, which results in reduced classification accuracy. Therefore, this study intends to use the superpixel segmentation method to perform multiscale superpixel segmentation on the principal components of HSIs at multiple scales, and to couple the multiscale superpixel spatial spectral kernel (SSK) with the original spectral kernel to form a synthetic kernel using weights for HSI classification. The three HSIs of Pavia University, Pavia Center, and Washington DC Mall are used as the experimental data to test and analyze this method. The experimental results show that the effective classification accuracies for the three datasets obtained by this method are 5.28%, 5.90%, and 7.71% higher than the five comparison methods, at best. The results prove that this method can effectively solve the problem of imprecise image feature extraction and an unknown number of initial superpixels and can significantly improve the classification accuracy of HSIs.

Superpixel Guided Network for Three-Dimensional Stereo Matching

Superpixel provides local pixel coherence and respects object boundary, which is beneficial to stereo matching. Recently, superpixel cues are introduced into deep stereo networks. These methods develop a superpixel-based sampling scheme to downsample input color images and upsample output disparity maps. However, in this way, the image details are inevitably lost in the downsampling and the upsampling process introduces errors in the final disparity as well. Besides, this mechanism further limits the possibility of utilizing larger and multi-scale superpixels, which are important to alleviate the matching ambiguity. To address these problems, a superpixel-guided stereo matching method (LSG-Stereo) is proposed, which explicitly exploits the feature and disparity consistency within multi-scale superpixels to improve disparity estimation. To effectively incorporate superpixel cues into a stereo matching network, two novel modules are designed, including Superpixel Attention Spatial Pyramid Pooling (SA-SPP) and Superpixel-Guided Refinement (SGR). The SA-SPP module takes advantage of the content-aware superpixel pooling to construct an adaptive spatial pooling pyramid for better feature extraction. The SGR module explicitly utilizes the disparity consistency over multi-scale superpixels to further refine the disparity estimation in details and ill-posed regions. The proposed method is evaluated on the Scene Flow dataset, KITTI 2012, and KITTI 2015 stereo benchmarks with comprehensive experiments. Experimental results demonstrate that our method can significantly improve the accuracy of stereo matching, especially in details, occlusions, and texture-less regions.