Natural Image Segmentation Research Articles

BiTSRS: A Bi-Decoder Transformer Segmentor for High-Spatial-Resolution Remote Sensing Images

Semantic segmentation of high-spatial-resolution (HSR) remote sensing (RS) images has been extensively studied, and most of the existing methods are based on convolutional neural network (CNN) models. However, the CNN is regarded to have less power in global representation modeling. In the past few years, methods using transformer have attracted increasing attention and generate improved results in semantic segmentation of natural images, owing to their powerful ability in global information acquisition. Nevertheless, these transformer-based methods exhibit limited performance in semantic segmentation of RS images, probably because of the lack of comprehensive understanding in the feature decoding process. In this paper, a novel transformer-based model named the bi-decoder transformer segmentor for remote sensing (BiTSRS) is proposed, aiming at alleviating the problem of flexible feature decoding, through a bi-decoder design for semantic segmentation of RS images. In the proposed BiTSRS, the Swin transformer is adopted as encoder to take both global and local representations into consideration, and a unique design module (ITM) is designed to deal with the limitation of input size for Swin transformer. Furthermore, BiTSRS adopts a bi-decoder structure consisting of a Dilated-Uper decoder and a fully deformable convolutional network (FDCN) module embedded with focal loss, with which it is capable of decoding a wide range of features and local detail deformations. Both ablation experiments and comparison experiments were conducted on three representative RS images datasets. The ablation analysis demonstrates the contributions of specifically designed modules in the proposed BiTSRS to performance improvement. The comparison experimental results illustrate that the proposed BiTSRS clearly outperforms some state-of-the-art semantic segmentation methods.

Bridging 2D and 3D segmentation networks for computation-efficient volumetric medical image segmentation: An empirical study of 2.5D solutions.

Recently, deep convolutional neural networks have achieved great success for medical image segmentation. However, unlike segmentation of natural images, most medical images such as MRI and CT are volumetric data. In order to make full use of volumetric information, 3D CNNs are widely used. However, 3D CNNs suffer from higher inference time and computation cost, which hinders their further clinical applications. Additionally, with the increased number of parameters, the risk of overfitting is higher, especially for medical images where data and annotations are expensive to acquire. To issue this problem, many 2.5D segmentation methods have been proposed to make use of volumetric spatial information with less computation cost. Despite these works lead to improvements on a variety of segmentation tasks, to the best of our knowledge, there has not previously been a large-scale empirical comparison of these methods. In this paper, we aim to present a review of the latest developments of 2.5D methods for volumetric medical image segmentation. Additionally, to compare the performance and effectiveness of these methods, we provide an empirical study of these methods on three representative segmentation tasks involving different modalities and targets. Our experimental results highlight that 3D CNNs may not always be the best choice. Despite all these 2.5D methods can bring performance gains to 2D baseline, not all the methods hold the benefits on different datasets. We hope the results and conclusions of our study will prove useful for the community on exploring and developing efficient volumetric medical image segmentation methods.

Deep Neural Networks Learn Meta-Structures from Noisy Labels in Semantic Segmentation

How deep neural networks (DNNs) learn from noisy labels has been studied extensively in image classification but much less in image segmentation. So far, our understanding of the learning behavior of DNNs trained by noisy segmentation labels remains limited. In this study, we address this deficiency in both binary segmentation of biological microscopy images and multi-class segmentation of natural images. We generate extremely noisy labels by randomly sampling a small fraction (e.g., 10%) or flipping a large fraction (e.g., 90%) of the ground truth labels. When trained with these noisy labels, DNNs provide largely the same segmentation performance as trained by the original ground truth. This indicates that DNNs learn structures hidden in labels rather than pixel-level labels per se in their supervised training for semantic segmentation. We refer to these hidden structures in labels as meta-structures. When DNNs are trained by labels with different perturbations to the meta-structure, we find consistent degradation in their segmentation performance. In contrast, incorporation of meta-structure information substantially improves performance of an unsupervised segmentation model developed for binary semantic segmentation. We define meta-structures mathematically as spatial density distributions and show both theoretically and experimentally how this formulation explains key observed learning behavior of DNNs.

Semi-supervised deep learning and low-cost cameras for the semantic segmentation of natural images in viticulture

Automatic yield monitoring and in-field robotic harvesting by low-cost cameras require object detection and segmentation solutions to tackle the poor quality of natural images and the lack of exactly-labeled datasets of consistent sizes. This work proposed the application of deep learning for semantic segmentation of natural images acquired by a low-cost RGB-D camera in a commercial vineyard. Several deep architectures were trained and compared on 85 labeled images. Three semi-supervised learning methods (PseudoLabeling, Distillation and Model Distillation) were proposed to take advantage of 320 non-annotated images. In these experiments, the DeepLabV3+ architecture with a ResNext50 backbone, trained with the set of labeled images, achieved the best overall accuracy of 84.78%. In contrast, the Manet architecture combined with the EfficientnetB3 backbone reached the highest accuracy for the bunch class (85.69%). The application of semi-supervised learning methods boosted the segmentation accuracy between 5.62 and 6.01%, on average. Further discussions are presented to show the effects of a fine-grained manual image annotation on the accuracy of the proposed methods and to compare time requirements.

Brain Tumor Image Segmentation Based on Grouped Convolution

The automatic segmentation of MRI multi-modal images of brain tumors is one of the important research contents of disease detection and analysis. Due to the heterogeneity of tumors, it is difficult to achieve efficient and accurate automatic segmentation of brain tumors. Traditional segmentation methods based on machine learning cannot handle complex scenes such as complex edges and overlapping categories. In clinical assisted diagnosis, it is of great significance to apply deep learning to two-dimensional natural image segmentation and three-dimensional medical image segmentation. In this paper, we propose a three-dimensional network model to achieve precise segmentation of brain tumors. The model adopts an encoder-decoder structure and replaces ordinary convolution with grouped convolution to reduce network parameters and improve network performance. The model improves the problem of information exchange between different groups through channel mixing. Experiments conducted on the BraTS (Brain Tumor Segmentation) 2018 challenge dataset prove that our network greatly reduces the computational cost while ensuring segmentation accuracy.

Transfer learning for data-efficient abdominal muscle segmentation with convolutional neural networks.

BackgroundSkeletal muscle segmentation is an important procedure for assessing sarcopenia, an emerging imaging biomarker of patient frailty. Data annotation remains the bottleneck for training deep learning auto‐segmentation models.PurposeThere is a need to define methodologies for applying models to different domains (e.g., anatomical regions or imaging modalities) without dramatically increasing data annotation.MethodsTo address this problem, we empirically evaluate the generalizability of various source tasks for transfer learning: natural image classification, natural image segmentation, unsupervised image reconstruction, and self‐supervised jigsaw solving. Axial CT slices at L3 were extracted from PET‐CT scans for 204 oesophago‐gastric cancer patients and the skeletal muscle manually delineated by an expert. Features were transferred and segmentation models trained on subsets (n=5,10,25,50,75,100,125) of the manually annotated training set. Four‐fold cross‐validation was performed to evaluate model generalizability. Human‐level performance was established by performing an inter‐observer study consisting of ten trained radiographers.ResultsWe find that accurate segmentation models can be trained on a fraction of the data required by current approaches. The Dice similarity coefficient and root mean square distance‐to‐agreement were calculated for each prediction and used to assess model performance. Models pre‐trained on a segmentation task and fine‐tuned on 10 images produce delineations that are comparable to those from trained observers and extract reliable measures of muscle health.ConclusionsAppropriate transfer learning can generate convolutional neural networks for abdominal muscle segmentation that achieve human‐level performance while decreasing the required data by an order of magnitude, compared to previous methods (n=160→10). This work enables the development of future models for assessing skeletal muscle at other anatomical sites where large annotated data sets are scarce and clinical needs are yet to be addressed.

Flexibly regularized mixture models and application to image segmentation

Probabilistic finite mixture models are widely used for unsupervised clustering. These models can often be improved by adapting them to the topology of the data. For instance, in order to classify spatially adjacent data points similarly, it is common to introduce a Laplacian constraint on the posterior probability that each data point belongs to a class. Alternatively, the mixing probabilities can be treated as free parameters, while assuming Gauss–Markov or more complex priors to regularize those mixing probabilities. However, these approaches are constrained by the shape of the prior and often lead to complicated or intractable inference. Here, we propose a new parametrization of the Dirichlet distribution to flexibly regularize the mixing probabilities of over-parametrized mixture distributions. Using the Expectation–Maximization algorithm, we show that our approach allows us to define any linear update rule for the mixing probabilities, including spatial smoothing regularization as a special case. We then show that this flexible design can be extended to share class information between multiple mixture models. We apply our algorithm to artificial and natural image segmentation tasks, and we provide quantitative and qualitative comparison of the performance of Gaussian and Student-t mixtures on the Berkeley Segmentation Dataset. We also demonstrate how to propagate class information across the layers of deep convolutional neural networks in a probabilistically optimal way, suggesting a new interpretation for feedback signals in biological visual systems. Our flexible approach can be easily generalized to adapt probabilistic mixture models to arbitrary data topologies.

Spine Image Segmentation Based on U-Net and Atrous spatial pyramid pooling

In the past ten years, deep learning has achieved remarkable results in the area of natural image segmentation, and has gradually turned to the field of medical image segmentation. The precise segmentation of spine images can be used for early screening of spondylopathy, which is convenient for early detection and treatment of patients. Aiming at the segmentation of the spine image by the U-Net network, which structure will lead to large model calculation, network overfitting, image size, noise information and other issues. This paper introduces a new method based on spatial pyramid pooling module ASpp on U-Net network and applies the proposed network structure to the segmentation of the spine image. The model is first enhanced by the Spp module and Densenet data; secondly, the U-Net encoding and decoding structure is adopted, and the deep separable convolution is used. This method greatly reduces the complexity and computation of the model, and uses a rectangular convolution kernel to increase the computation of the model in a small amount. Based on the volume, the receptive field of the convolution operation is expanded; finally, in order to effectively enhance the segmentation area of the image, the SE-Net attention mechanism module is added to the lateral connection. The method proposed in this paper conducts ablation experiments on a public dataset of spine images and compares it with the U-Net network. The method proposed IOU in this paper can reach 72%, and the F1-score can reach 0.92. The comparison of the experimental results of spine medical image datasets shows that the method proposed in this paper can effectively improve the accuracy and accuracy of spine image segmentation.

The Hybrid Novel Self-Organising Map Combined with Fuzzy C-Means and K-Means Convolution for a Soft and Hard Natural Digital Image Segmentation

The Hybrid Novel Self-Organising Map Combined with Fuzzy C-Means and K-Means Convolution for a Soft and Hard Natural Digital Image Segmentation

SDFCNv2: An Improved FCN Framework for Remote Sensing Images Semantic Segmentation

Semantic segmentation is a fundamental task in remote sensing image analysis (RSIA). Fully convolutional networks (FCNs) have achieved state-of-the-art performance in the task of semantic segmentation of natural scene images. However, due to distinctive differences between natural scene images and remotely-sensed (RS) images, FCN-based semantic segmentation methods from the field of computer vision cannot achieve promising performances on RS images without modifications. In previous work, we proposed an RS image semantic segmentation framework SDFCNv1, combined with a majority voting postprocessing method. Nevertheless, it still has some drawbacks, such as small receptive field and large number of parameters. In this paper, we propose an improved semantic segmentation framework SDFCNv2 based on SDFCNv1, to conduct optimal semantic segmentation on RS images. We first construct a novel FCN model with hybrid basic convolutional (HBC) blocks and spatial-channel-fusion squeeze-and-excitation (SCFSE) modules, which occupies a larger receptive field and fewer network model parameters. We also put forward a data augmentation method based on spectral-specific stochastic-gamma-transform-based (SSSGT-based) during the model training process to improve generalizability of our model. Besides, we design a mask-weighted voting decision fusion postprocessing algorithm for image segmentation on overlarge RS images. We conducted several comparative experiments on two public datasets and a real surveying and mapping dataset. Extensive experimental results demonstrate that compared with the SDFCNv1 framework, our SDFCNv2 framework can increase the mIoU metric by up to 5.22% while only using about half of parameters.

3D Quantum Cuts for automatic segmentation of porous media in tomography images

Binary segmentation of volumetric images of porous media is a crucial step towards gaining a deeper understanding of the factors governing biogeochemical processes at minute scales. Contemporary work primarily revolves around primitive techniques based on global or local adaptive thresholding that have known common drawbacks in image segmentation. Moreover, the absence of a unified benchmark prohibits quantitative evaluation, which further undermines the impact of existing methodologies. In this study, we tackle the issue on both fronts. First, by drawing parallels with natural image segmentation, we propose a novel, and automatic segmentation technique, 3D Quantum Cuts (QCuts-3D) grounded on a state-of-the-art spectral clustering technique. Secondly, we curate and present a publicly available dataset of 68 multiphase volumetric images of porous media with diverse solid geometries, along with voxel-wise ground truth annotations for each constituting phase. We provide comparative evaluations between QCuts-3D and the current state-of-the-art over this dataset across a variety of evaluation metrics. The proposed systematic approach achieves a 26% increase in AUROC (Area Under Receiver Operating Characteristics) while achieving a substantial reduction of the computational complexity over state-of-the-art competitors. Moreover, statistical analysis reveals that the proposed method exhibits significant robustness against the compositional variations of porous media.

Segmentation of Natural Images with K-Means and Hierarchical Algorithm based on Mixture of Pearson Distributions

In this paper, an attempt has been made to analyze the performance of the image segmented algorithms with the addition of the Pearsonian Type III mixture model. By using the Type III Pearsonian system of distributions the image segmentation process was carried out in the current article which is a novel technique. With the help of K-component combination of Pearsonian Type III distribution, it is considered that the whole input images are characterized. The performance parameters PRI (Probabilistic Rand Index), GCE (Global Consistency Error) and VOI (Volume of Interest) for the currently considered model are estimated with the help of EM (Expectation Maximization) algorithm. For analyzing the proposed model’s performance, four random images are selected as input for the current model from Berkeley image database. The performance metric parameters PRI, GCE and VOI values given the results as the currently proposed method is providing more précise results for the input images where the regions of the input images selected are with tiles having long upper model and the left skewed images. By the help of image quality measures, the proposed method is performing well for the purpose of retrieving the images with respect to the picture segmenting process which is based on GMM (Gaussian Mixture Model). The current model performance was compared with the other existing models like the k-means hierarchical clustering model and the 3-paprameter regression models.

Enhanced Mask R-CNN for herd segmentation

Livestock image segmentation is an important task in the field of vision and image processing. Since utilizing the concentration of forage in the grazing area with shielding the surrounding farm plants and crops is necessary for making effective cattle ranch arrangements, there is a need for a segmentation method that can handle multiple objects segmentation. Moreover, the indistinct boundaries and irregular shapes of cattle bodies discourage the application of the existing Mask Region-based Convolutional Neural Network (Mask R-CNN) which was primarily modeled for the segmentation of natural images. To address this, an enhanced Mask R-CNN model was proposed for multiple objects instance segmentation to support indistinct boundaries and irregular shapes of cattle bodies for precision livestock farming. The contributions of this method are in multiple folds: 1) optimal filter size smaller than a residual network for extracting smaller and composite features; 2) region proposals for utilizing multiscale semantic features; 3) Mask R-CNN’s fully connected layer integrated with sub-network for an enhanced segmentation. The experiment conducted on pre-processed datasets produced a mean average precision (mAP) of 0.93, which was higher than the results from the existing state-of-the-art models. Keywords: livestock farming, image segmentation, mask R-CNN, herd, enhancement DOI: 10.25165/j.ijabe.20211404.6398 Citation: Bello R W, Mohamed A S A, Talib A Z. Enhanced Mask R-CNN for herd segmentation. Int J Agric & Biol Eng, 2021; 14(4): 238–244.

Division of images into areas of local extrema with a monotonic change in pixel brightness

In this paper, the problem of segmentation of halftone images is considered, in which areas of local maxima and minima (extrema) are distinguished with a monotonic change in the brightness of pixels from local extrema to the boundaries of areas. To solve this problem, a mathematical model is proposed and a segmentation algorithm is developed on the basis of counter-wave growing of local extremum regions. The developed algorithm differs from the known segmentation algorithms by using a set of brightness thresholds (by the number of regions), varying by one in each cycle, starting from the values of local extrema, taking into account the increase or decrease in brightness to select adjacent pixels that are attached to the regions formed from these local extrema. The algorithm provides a greater deviation of pixel brightness from the average value within the region compared to known segmentation algorithms. This does not allow evaluating its efficiency using known indicators based on the variance of the brightness within the region. In this regard, estimates of the monotonicity of changes in the brightness of regions are proposed based on a) the shortest distances from each pixel of the region to the corresponding local extremum along the routes determined by the maximum increase (for the region of the local maximum) or decrease (for the region of the local minimum) the brightness of pixels and b) taking into account the number pixels that break the monotony of the segment brightness change. Using these estimates, it is shown that the proposed algorithm provides segmentation of artificial and natural grayscale images with a monotonic change in the brightness of pixels in the areas of local extrema. These properties allow us to consider the developed algorithm as a basis for the selection of texels, spots, low-contrast objects in images.

Redundancy between spectral and higher-order texture statistics for natural image segmentation

Visual texture, defined by local image statistics, provides important information to the human visual system for perceptual segmentation. Second-order or spectral statistics (equivalent to the Fourier power spectrum) are a well-studied segmentation cue. However, the role of higher-order statistics (HOS) in segmentation remains unclear, particularly for natural images. Recent experiments indicate that, in peripheral vision, the HOS of the widely adopted Portilla-Simoncelli texture model are a weak segmentation cue compared to spectral statistics, despite the fact that both are necessary to explain other perceptual phenomena and to support high-quality texture synthesis. Here we test whether this discrepancy reflects a property of natural image statistics. First, we observe that differences in spectral statistics across segments of natural images are redundant with differences in HOS. Second, using linear and nonlinear classifiers, we show that each set of statistics individually affords high performance in natural scenes and texture segmentation tasks, but combining spectral statistics and HOS produces relatively small improvements. Third, we find that HOS improve segmentation for a subset of images, although these images are difficult to identify. We also find that different subsets of HOS improve segmentation to a different extent, in agreement with previous physiological and perceptual work. These results show that the HOS add modestly to spectral statistics for natural image segmentation. We speculate that tuning to natural image statistics under resource constraints could explain the weak contribution of HOS to perceptual segmentation in human peripheral vision.

A Hybrid Morphological Active Contour for Natural Images

Morphological active contours for image segmentation have become popular due to their low computational complexity coupled with their accurate approximation of the partial differential equations involved in the energy minimization of the segmentation process. In this paper, a morphological active contour which mimics the energy minimization of the popular Chan-Vese Active Contour without Edges is coupled with a morphological edge-driven segmentation term to accurately segment natural images. By using morphological approximations of the energy minimization steps, the algorithm has a low computational complexity. Additionally, the coupling of the edge-based and region-based segmentation techniques allows the proposed method to be robust and accurate. We will demonstrate the accuracy and robustness of the algorithm using images from the Weizmann Segmentation Evaluation Database and report on the segmentation results using the Sorensen-Dice similarity coefficient.

Affinity Fusion Graph-Based Framework for Natural Image Segmentation

This paper proposes an affinity fusion graph framework to effectively connect different graphs with highly discriminating power and nonlinearity for natural image segmentation. The proposed framework combines adjacency-graphs and kernel spectral clustering based graphs (KSC-graphs) according to a new definition named affinity nodes of multi-scale superpixels. These affinity nodes are selected based on a better affiliation of superpixels, namely subspace-preserving representation which is generated by sparse subspace clustering based on subspace pursuit. Then a KSC-graph is built via a novel kernel spectral clustering to explore the nonlinear relationships among these affinity nodes. Moreover, an adjacency-graph at each scale is constructed, which is further used to update the proposed KSC-graph at affinity nodes. The fusion graph is built across different scales, and it is partitioned to obtain final segmentation result. Experimental results on the Berkeley segmentation dataset and Microsoft Research Cambridge dataset show the superiority of our framework in comparison with the state-of-the-art methods. The code is available at <uri>https://github.com/Yangzhangcst/AF-graph</uri>.

Efficient Image Segmentation of Natural Images with Noise using Energy Curved Based on Electromagnetism Optimization Algorithm

The method of digital image processing is an important step for extraction features from images, especially details from the medical image by dividing the image into various regions. In the last two decades, many significant image segmentation schemes modeled for different applications like medical to weather forecasting by images from the satellite. From available methods, multilevel thresholding of pixel values using the histogram of an image is an exceedingly important method developed for vide range of applications. Otsu’s method is the primer multilevel thresholding method based on the maximizing the inter-class variance based on the histogram. In Otsu’s method, the pixels of an image are grouped into different regions based on threshold levels calculated on the histogram of the image. All the pixels are segmented into two groups based on threshold level in the case of bi-level thresholding. The effectiveness of segmentation is depends on threshold levels, efficiency can be enhanced by computing optimized threshold level, Electromagnetism Optimization (EMO) technique used for this purpose in this paper. The techniques based on histograms of image are not holding the spatial particulars of contextual information for computing the optimized threshold levels, to nullify this shortcoming Energy Curve is used as an alternative of histogram, which is restore the special relation of pixels with its neighbor pixels. To examine the results several benchmark images considered, the final results shows that the method with Otsu’s technique with EMO on energy curve is giving better results as compared with methods based on the image histogram.

Segmentation of natural images based on super pixel and graph merging

The task of natural image segmentation is one of the most researched topics of computer vision. There are mainly two principal approaches for the task, the statistical approach and the supervised approach. The proposed methodology segments natural images combining a set of statistical algorithms. First, the image is preprocessed to enhance the edges. Weighted average of the denoised image and its derivatives is the preprocessed output. Thereafter, an energy based super pixelation is applied to over segment the image. Finally, a connectivity graph is built where nodes correspond to super pixels and edges connect the adjacent super pixels. The adjacent super pixels are merged based on the confidence value defined in terms of their textural and colour similarity. Proposed methodology has been applied on the images of BSDS500 dataset. Performance of the proposed work has been compared with that of other works based on detected edge maps. Few works generate ultrametric contour maps (UCM). To compare the performance with those works, UCM is also generated by the proposed methodology. To do so images at multiple scales are considered. It is observed that the output of segmentation is better in case of the proposed methodology. Proposed methodology is much faster than others. Thus, makes it suitable for real time application in robot vision.

Point-Sampling Method Based on 3D U-Net Architecture to Reduce the Influence of False Positive and Solve Boundary Blur Problem in 3D CT Image Segmentation

Malignant lesions are a huge threat to human health and have a high mortality rate. Locating the contour of organs is a preparation step, and it helps doctors diagnose correctly. Therefore, there is an urgent clinical need for a segmentation model specifically designed for medical imaging. However, most current medical image segmentation models directly migrate from natural image segmentation models, thus ignoring some characteristic features for medical images, such as false positive phenomena and the blurred boundary problem in 3D volume data. The research on organ segmentation models for medical images is still challenging and demanding. As a consequence, we redesign a 3D convolutional neural network (CNN) based on 3D U-Net and adopted the render method from computer graphics for 3D medical images segmentation, named Render 3D U-Net. This network adapts a subdivision-based point-sampling method to replace the original upsampling method for rendering high-quality boundaries. Besides, Render 3D U-Net integrates the point-sampling method into 3D ANU-Net architecture under deep supervision. Meanwhile, to reduce false positive phenomena in clinical diagnosis and to achieve more accurate segmentation, Render 3D U-Net specially designs a module for screening false positive. Finally, three public challenge datasets (MICCAI 2017 LiTS, MICCAI 2019 KiTS, and ISBI 2019 segTHOR) were selected as experiment datasets and to evaluate the performance on target organs. Compared with other models, Render 3D U-Net improved the performance on both overall organ and boundary in the CT image segmentation tasks, including in the liver, kidney, and heart.