Semantic Segmentation Framework Research Articles

Webly-supervised semantic segmentation via curriculum learning

In this paper, we propose a weakly supervised semantic segmentation method by directly learning from web images, which are crawled from the Internet by using text queries, without any explicit user annotation or even data filtering. With the goal of handling the massive amount of noisy labels in web images, we design a three-stage approach for weakly-supervised semantic segmentation based on curriculum learning. We first generate pixel-level masks for the training images via a popular weakly-supervised semantic segmentation framework. Then, we consider the noise of the web data in two ways. At the image-level, the complexity of data is measured using its distribution density in a classification feature space. At the pixel-level, the complexity of the mask is evaluated by exploiting the relationship between the saliency map and those segmented images in an unsupervised manner. The key insight to this design is that, common and simple object patterns in images should be salient with both the saliency detector and weakly supervised DCNNs, where they should be sparse with high regional consistency between them. This allows for an efficient implementation of curriculum learning from noisy web images. Experiments on the popular PASCAL VOC 2012 benchmark show that we achieve very competitive performance with scores of 64.0% mIoU using our pure web dataset, which contains noisy, single-label images. We further improve the performance to 69.7% mIoU by using the CurriculumWebSegNet fine-tuned on the PASCAL VOC dataset, which has more precise multi-label supervision.

Semi-FCMNet: Semi-Supervised Learning for Forest Cover Mapping from Satellite Imagery via Ensemble Self-Training and Perturbation

Forest cover mapping is of paramount importance for environmental monitoring, biodiversity assessment, and forest resource management. In the realm of forest cover mapping, significant advancements have been made by leveraging fully supervised semantic segmentation models. However, the process of acquiring a substantial quantity of pixel-level labelled data is prone to time-consuming and labour-intensive procedures. To address this issue, this paper proposes a novel semi-supervised-learning-based semantic segmentation framework that leverages limited labelled and numerous unlabelled data, integrating multi-level perturbations and model ensembles. Our framework incorporates a multi-level perturbation module that integrates input-level, feature-level, and model-level perturbations. This module aids in effectively emphasising salient features from remote sensing (RS) images during different training stages and facilitates the stability of model learning, thereby effectively preventing overfitting. We also propose an ensemble-voting-based label generation strategy that enhances the reliability of model-generated labels, achieving smooth label predictions for challenging boundary regions. Additionally, we designed an adaptive loss function that dynamically adjusts the focus on poorly learned categories and dynamically adapts the attention towards labels generated during both the student and teacher stages. The proposed framework was comprehensively evaluated using two satellite RS datasets, showcasing its competitive performance in semi-supervised forest-cover-mapping scenarios. Notably, the method outperforms the fully supervised approach by 1–3% across diverse partitions, as quantified by metrics including mIoU, accuracy, and mPrecision. Furthermore, it exhibits superiority over other state-of-the-art semi-supervised methods. These results indicate the practical significance of our solution in various domains, including environmental monitoring, forest management, and conservation decision-making processes.

A new real-time image semantic segmentation framework based on a lightweight deep convolutional encoder-decoder architecture for robotic environment sensing

Background: Image semantic segmentation can be understood as the allocation of a predefined category label to each pixel in the image to achieve the region segmentation of the image. Different categories in the image are identified with different colors. While achieving pixel classification, the position information of pixel points of different categories in the image is retained. Purpose: Due to the influence of background and complex environment, the traditional semantic segmentation methods have low accuracy. To alleviate the above problems, this paper proposes a new real-time image semantic segmentation framework based on a lightweight deep convolutional encoder-decoder architecture for robotic environment sensing. Methodology: This new framework is divided into three stages: encoding stage, decoding stage and dimension reduction stage. In the coding stage, a cross-layer feature map fusion (CLFMF) method is proposed to improve the effect of feature extraction. In the decoding stage, a new lightweight decoder (LD) structure is designed to reduce the number of convolutional layers to speed up model training and prediction. In the dimension reduction stage, the convolution dimension reduction method (CDR) is presented to connect the encoder and decoder layer by layer to enhance the decoder effect. Results: Compared with other state-of-the-art image semantic segmentation methods, we conduct comparison experiments on datasets Cityscapes, SUN RGB-D, CamVid, KITTI. The Category iIoU combined with the proposed method is more than 70%, and the Category IoU is as high as 89.7%. Conclusion: The results reflect that the new method can achieve the better semantic segmentation effect.

Deep learning based automatic Debakey classification of aortic dissection using CT plain scan

ObjectiveThis paper presents an automatic segmentation and classification of aortic dissection by utilizing neural network based on CT scanning. MethodologyFirst we present a way of establishing models of indivisible aorta parametrization, designed multiple seed points as the initial parameter and based on the three-dimensional area growth algorithm, finally add morphological processing to realize the segmentation of inner membrane, and then introduced the semantic segmentation framework, and analyzed the characteristics and function of each part, improve the Deeplabv3 + network. This paper analyzes the results of the different experiments with two-dimensional lattices and presents the advantages of being sensitive to differences in shape and position and of not being able to use three-dimensional spatial continuity. Our research establishs the aorta reconstruction, which is simpler for learning the transformation of both the shape and position of, according to a last for three-dimensional CT images interfloor of the entire helix network aorta 3D MPRNet.A new module of multi-scale receptive field information fusion is designed, which realizes the function of acquiring different receptive field features, taking into account details and global features, so that the model obtains the unique ability more suitable for segmentation problems. In the decoder stage, the cross-layer connection mode similar to U-Net is still adopted to realize the fusion of semantic information at high and low levels, which ensures that the network can learn enough features. ResultsThe full convolutional network and traditional algorithms based on domain knowledge of aortic dissection images achieve a robust, automated and high precision CT image aortic dissection segmentation scheme. ConclusionThe large number of decoder parameters is increased, which increases the difficulty of network training and reduces the reasoning speed of the model. At the same time, not all features are useful information, while the high frequency and low frequency, the feature graph only adopts cross-layer connection to connect shallow features to high-level information may bring the redundancy of information and interference with useful information. Therefore, designing an efficient decoder is another research idea to accelerate model inference and improve segmentation accuracy.

Comparison of U-Net’s Variants for Segmentation of Polyp Images

Medical image analysis involves examining pictures acquired by medical imaging technologies in order to address clinical issues. The aim is to increase the quality of clinical diagnosis and extract useful information. Automatic segmentation based on deep learning (DL) techniques has gained popularity recently. In contrast to the conventional manual learning method, a neural network can now automatically learn image features. One of the most crucial convolutional neural network (CNN) semantic segmentation frameworks is U-net. It is frequently used for classification, anatomical segmentation, and lesion segmentation in the field of medical image analysis. This network framework's benefit is that it not only effectively processes and objectively evaluates medical images, properly segments the desired feature target, and helps to increase the accuracy of medical image-based diagnosis.

Development of AI- and Robotics-Assisted Automated Pavement-Crack-Evaluation System

Crack inspection is important to monitor the structural health of pavement structures and make the repair process easier. Currently, pavement crack inspection is conducted manually, which is inefficient and costly at the same time. To solve the problem, this work has developed a robotic system for automated data collection and analysis in real-time. The robotic system navigates the pavement and collects visual images from the surface. A deep-learning-based semantic segmentation framework named RCDNet was proposed. The RCDNet was implemented on the onboard computer of the robot to identify cracks from the visual images. The encoder-decoder architecture was utilized as the base framework of the proposed RCDNet. The RCDNet comprises a dual-channel encoder and a decoder module. The encoder and decoder parts contain a context-embedded channel attention (CECA) module and a global attention module (GAM), respectively. Simulation results show that the deep learning model obtained 96.29% accuracy for predicting the images. The proposed robotic system was tested in both indoor and outdoor environments. The robot was observed to complete the inspection of a 3 m × 2 m grid within 10 min and a 2.5 m × 1 m grid within 6 min. This outcome shows that the proposed robotic method can drastically reduce the time of manual inspection. Furthermore, a severity map was generated using the visual image results. This map highlights areas that require greater attention for repair in the test grid.

Weed Identification in Maize Fields Based on Improved Swin-Unet

The maize field environment is complex. Weeds and maize have similar colors and may overlap, and lighting and weather conditions vary. Thus, many methods for the automated differentiation of maize and weeds achieve poor segmentation or cannot be used in real time. In this paper, a weed recognition model based on improved Swin-Unet is proposed. The model first performs semantic segmentation of maize seedlings and uses the resulting mask to identify weeds. U-Net acts as the semantic segmentation framework, and a Swin transformer module is introduced to improve performance. DropBlock regularization, which randomly hides some blocks in crop feature maps, is applied to enhance the generalization ability of the model. Finally, weed areas are identified and segmented with the aid of an improved morphological processing algorithm. The DeepLabv3+, PSANet, Mask R-CNN, original Swin-Unet, and proposed models are trained on a dataset of maize seedling images. The proposed Swin-Unet model outperforms the others, achieving a mean intersection over union of 92.75%, mean pixel accuracy of 95.57%, and inference speed of 15.1 FPS. Our model could be used for accurate, real-time segmentation of crops and weeds and as a reference for the development of intelligent agricultural equipment.

A novel transformer-based semantic segmentation framework for structural condition assessment

Conventional structural health monitoring (SHM) evaluates the condition of civil structures by analyzing the data acquired by advanced sensors. The requirement of overinvestment in specialized equipment and labor for implementation prevents the traditional SHM from large-scale usage. On the other hand, computer vision techniques offer cost-effective solutions for SHM thanks to its inherent advantage in data acquirement and processing. More importantly, it has been demonstrated that these emerging solutions can produce reliable condition diagnoses for civil structures using pure image data. In this article, a novel transformer-based neural network is proposed for vision-based structural condition assessment which is formulated to a semantic segmentation problem. The network employs Swin Transformer as the backbone and MaskFormer as the overall architecture to recognize components (sleepers, slabs, columns, etc.) and damage (concrete damage, exposed rebar) of structures. Unlike the commonly used fully convolutional networks, the proposed model tackles semantic segmentation as a mask classification rather than a pixel classification problem. To deal with the lack of training data, an image data augmentation method called Copy-Paste is extended and applied for training data generation, resulting in an increase of around 40% data for component segmentation and 71% data for damage segmentation. Experimental validations on the Tokaido railway viaduct dataset show that the proposed approach is very accurate, achieving 97% and 90% mean Intersection Over Union for component and damage segmentation, outperforming the existing methods by a significant margin. The accurate segmentation results can provide meaningful information for downstream SHM tasks.

Valuing vicinity: Memory attention framework for context-based semantic segmentation in histopathology.

The segmentation of histopathological whole slide images into tumourous and non-tumourous types of tissue is a challenging task that requires the consideration of both local and global spatial contexts to classify tumourous regions precisely. The identification of subtypes of tumour tissue complicates the issue as the sharpness of separation decreases and the pathologist's reasoning is even more guided by spatial context. However, the identification of detailed tissue types is crucial for providing personalized cancer therapies. Due to the high resolution of whole slide images, existing semantic segmentation methods, restricted to isolated image sections, are incapable of processing context information beyond. To take a step towards better context comprehension, we propose a patch neighbour attention mechanism to query the neighbouring tissue context from a patch embedding memory bank and infuse context embeddings into bottleneck hidden feature maps. Our memory attention framework (MAF) mimics a pathologist's annotation procedure - zooming out and considering surrounding tissue context. The framework can be integrated into any encoder-decoder segmentation method. We evaluate the MAF on two public breast cancer and liver cancer data sets and an internal kidney cancer data set using famous segmentation models (U-Net, DeeplabV3) and demonstrate the superiority over other context-integrating algorithms - achieving a substantial improvement of up to 17% on Dice score. The code is publicly available at https://github.com/tio-ikim/valuing-vicinity.

Uncertainty-Aware Point-Cloud Semantic Segmentation for Unstructured Roads

Semantic segmentation is one of the fundamental elements for achieving effective and safe autonomous driving. However, due to the irregular boundaries and variable illumination of unstructured roads, applying it in these scenarios is confronted with great challenges. To address these problems, a novel point-cloud semantic segmentation framework for unstructured roads is proposed. It contains three sections: spherical projection, an uncertainty-aware semantic segmentation network, and post-processing. Firstly, point cloud will be projected to the range image which can be processed by the 2D convolution network. Then, the uncertainty-aware semantic segmentation network is constructed. It consists of context-aware attention module and direction attention up-sampling module, which can improve the performance for segmentation of unstructured roads. In addition, a gaussian mixture model is introduced at the end of the network to predict the result with uncertainty, indicating the confidence level of the output. Finally, the segmentation result is refined during the post-processing to help filter the noise points. Experiment data from mine sites was collected to validate the performance for unstructured roads. In addition, the proposed method was also evaluated on the public unstructured dataset RELLIS-3D. The experiments show that the proposed architecture achieved 74.9% mIoU and 40.4% mIoU, which performs better than comparison methods. Additionally, the network is more robust to noisy data by achieving improvements of 4.6%-7.6% under different levels of noise data.

Weakly Supervised 3D Segmentation via Receptive-Driven Pseudo Label Consistency and Structural Consistency

As manual point-wise label is time and labor-intensive for fully supervised large-scale point cloud semantic segmentation, weakly supervised method is increasingly active. However, existing methods fail to generate high-quality pseudo labels effectively, leading to unsatisfactory results. In this paper, we propose a weakly supervised point cloud semantic segmentation framework via receptive-driven pseudo label consistency and structural consistency to mine potential knowledge. Specifically, we propose three consistency contrains: pseudo label consistency among different scales, semantic structure consistency between intra-class features and class-level relation structure consistency between pair-wise categories. Three consistency constraints are jointly used to effectively prepares and utilizes pseudo labels simultaneously for stable training. Finally, extensive experimental results on three challenging datasets demonstrate that our method significantly outperforms state-of-the-art weakly supervised methods and even achieves comparable performance to the fully supervised methods.

Towards accurate mapping of loess waterworn gully by integrating google earth imagery and DEM using deep learning

Accurate mapping of loess waterworn gully (LWG) is essential to further study gully erosion and geomorphological evolution for the Chinese Loess Plateau (CLP). Due to the vertical joint and collapsibility of loess, LWGs have the characteristics of zigzag and unique slope abruptness under synthetic action of hydraulic force and gravity. This forces existing LWG mapping methods to either focus on the improvement of mapping accuracy or center on the increase of mapping efficiency. However, simultaneously achieving accurate and efficient mapping of LWG is still in its infancy under complex topographic conditions. Here, we proposed a method that innovatively integrates the loess slope abruptness feature into an improved deep learning semantic segmentation framework for LWG mapping using 0.6 m Google imagery and 5 m DEM data. We selected four study areas representing typical loess landforms to test the performance of our method. The proposed method can achieve satisfactory mapping results, with the F1 score, mean Intersection-over-Union (mIoU), and overall accuracy of 90.5%, 85.3%, and 92.3%, respectively. In addition, the proposed model also showed significant accuracy improvement by inputting additional topographic information (especially the slope of slope). Compared with existing algorithms (Random forests, original DeepLabV3+, and Unet), the proposed approach in this study achieved a better accuracy-efficiency trade-off. Overall, the method can ensure high accuracy and efficiency of the LWG mapping for different loess landform types and can be extended to study various loess gully mapping and water and soil conservation.

A Tongue Segmentation Algorithm Based on Deeplabv3+ Network Model

When collecting tongue images in an open en- vironment with a mobile portable collection device, there will be problems of different shooting angles and unstable lighting. Due to the strong mobility of the portable acquisition device, the captured images will inevitably be blurred by jitter, which further increases the difficulty of segmentation. This paper applies neural network to tongue images segmentation, and proposes a tongue images segmentation method based on deep convolutional neural network. This method is a tongue images segmentation method based on the semantic segmentation framework of DeeplabV3+. First, we modify the output category of the network. Because only the tongue region is segmented, segmentation targets can be divided into two categories when performing tongue images segmentation. One is the tongue region and the other is the background region. Then we replace the backbone network of DeeplabV3+ with a lightweight network and add an attention mechanism. Finally, we use the collected tongue images in the open environment to train the network. After the network obtains the initial segmentation result, tongue images are restored according to the same type of label, so as to obtain the required tongue images only containing tongues. The experimental results show that the method has higher segmentation accuracy for tongue images in open environment, and can better meet the needs of people for tongue images segmentation.

Improving Semi-Supervised and Domain-Adaptive Semantic Segmentation with Self-Supervised Depth Estimation

Training deep networks for semantic segmentation requires large amounts of labeled training data, which presents a major challenge in practice, as labeling segmentation masks is a highly labor-intensive process. To address this issue, we present a framework for semi-supervised and domain-adaptive semantic segmentation, which is enhanced by self-supervised monocular depth estimation (SDE) trained only on unlabeled image sequences. In particular, we utilize SDE as an auxiliary task comprehensively across the entire learning framework: First, we automatically select the most useful samples to be annotated for semantic segmentation based on the correlation of sample diversity and difficulty between SDE and semantic segmentation. Second, we implement a strong data augmentation by mixing images and labels using the geometry of the scene. Third, we transfer knowledge from features learned during SDE to semantic segmentation by means of transfer and multi-task learning. And fourth, we exploit additional labeled synthetic data with Cross-Domain DepthMix and Matching Geometry Sampling to align synthetic and real data. We validate the proposed model on the Cityscapes dataset, where all four contributions demonstrate significant performance gains, and achieve state-of-the-art results for semi-supervised semantic segmentation as well as for semi-supervised domain adaptation. In particular, with only 1/30 of the Cityscapes labels, our method achieves 92% of the fully-supervised baseline performance and even 97% when exploiting additional data from GTA. The source code is available at https://github.com/lhoyer/improving_segmentation_with_selfsupervised_depth.

A labor-free index-guided semantic segmentation approach for urban vegetation mapping from high-resolution true color imagery

ABSTRACT Accurate and timely information on urban vegetation (UV) can be used as an important indicator to estimate the health of cities. Due to the low cost of RGB cameras, true color imagery (TCI) has been widely used for high spatial resolution UV mapping. However, the current index-based and classifier-based UV mapping approaches face problems of the poor ability to accurately distinguish UV and the high reliance on massive annotated samples, respectively. To address this issue, an index-guided semantic segmentation (IGSS) framework is proposed in this paper. Firstly, a novel cross-scale vegetation index (CSVI) is calculated by the combination of TCI and Sentinel-2 images, and the index value can be used to provide an initial UV map. Secondly, reliable UV and non-UV samples are automatically generated for training the semantic segmentation model, and then the refined UV map can be produced. The experimental results show that the proposed CSVI outperformed the existing five RGB vegetation indices in highlighting UV cover and suppressing complex backgrounds, and the proposed IGSS workflow achieved satisfactory results with an OA of 87.72% ∼ 88.16% and an F1 score of 87.73% ∼ 88.37%, which is comparable with the fully-supervised method.

A surface graph based deep learning framework for large-scale urban mesh semantic segmentation

A surface graph based deep learning framework for large-scale urban mesh semantic segmentation

A dynamic few-shot learning framework for medical image stream mining based on self-training

Few-shot semantic segmentation (FSS) has been widely used in the field of information medicine and intelligent diagnosis. Due to the high cost of medical data collection and the privacy protection of patients, labeled medical images are difficult to obtain. Compared with other semantic segmentation dataset which can be automatically generated in a large scale, the medical image data tend to be continually generated. Most of the existing FSS techniques require abundant annotated semantic classes for pre-training and cannot deal with its dynamic nature of medical data stream. To deal with this issue, we propose a dynamic few-shot learning framework for medical semantic segmentation, which can fully utilize the features of newly-collected/generated data stream. We introduce a new pseudo-label generation strategy for continuously generating pseudo-labels and avoiding model collapse during self-training. Furthermore, an efficient consistency regularization strategy is proposed to fully utilize the limited data. The proposed framework is iteratively trained on three tasks: abdominal organ segmentation for CT and MRI, and cardiac segmentation for MRI. Experiments results demonstrate significant performance gain on medical data stream mining compared with the baseline method.

MCIBI++: Soft Mining Contextual Information Beyond Image for Semantic Segmentation.

Co-occurrent visual pattern makes context aggregation become an essential paradigm for semantic segmentation. The existing studies focus on modeling the contexts within image while neglecting the valuable semantics of the corresponding category beyond image. To this end, we propose a novel soft mining contextual information beyond image paradigm named MCIBI++ to further boost the pixel-level representations. Specifically, we first set up a dynamically updated memory module to store the dataset-level distribution information of various categories and then leverage the information to yield the dataset-level category representations during network forward. After that, we generate a class probability distribution for each pixel representation and conduct the dataset-level context aggregation with the class probability distribution as weights. Finally, the original pixel representations are augmented with the aggregated dataset-level and the conventional image-level contextual information. Moreover, in the inference phase, we additionally design a coarse-to-fine iterative inference strategy to further boost the segmentation results. MCIBI++ can be effortlessly incorporated into the existing segmentation frameworks and bring consistent performance improvements. Also, MCIBI++ can be extended into the video semantic segmentation framework with considerable improvements over the baseline. Equipped with MCIBI++, we achieved the state-of-the-art performance on seven challenging image or video semantic segmentation benchmarks.

Relict landslide detection using deep-learning architectures for image segmentation in rainforest areas: a new framework

ABSTRACT Landslides are destructive and recurrent natural disasters on steep slopes and represent a risk to lives and properties. Knowledge of relict landslides’ location is vital to understand their mechanisms, update inventory maps and improve risk assessment. However, relict landslide mapping is complex in tropical regions covered with rainforest vegetation. A new CNN framework is proposed for semi-automatic detection of relict landslides, which uses a dataset generated by a k-means clustering algorithm and has a pre-training step. The weights computed in the pre-training are used to fine-tune the CNN training process. A comparison between the proposed and the standard framework is performed using CBERS-04A WPM images. Three CNNs for semantic segmentation are used (Unet, FPN and Linknet) with two augmented datasets. A total of 42 combinations of CNNs are tested. Values of precision and recall were very similar between the combinations tested. Recall was higher than 75% for every combination, but precision values were usually smaller than 20%. False positives (FP) samples were addressed as the cause for these low precision values. Predictions of the proposed framework were more accurate and correctly detected more landslides. This work demonstrates that there are limitations for detecting relict landslides in areas covered with rainforest, mainly related to similarities between the spectral response of pastures and deforested areas with Gleichenella sp. ferns, commonly used as an indicator of landslide scars. Relict landslide detection has limitations in rainforest environments The proposed framework for CNN semantic segmentation proved to be useful for relict landslide detection Land cover features spectral characteristics, and input data technical specificities are crucial to the accuracy of CNN models No optimal combination of parameters has been identified No CNN proved to be superior

Path Segmentation from Point Cloud Data for Autonomous Navigation

Autonomous vehicles require in-depth knowledge of their surroundings, making path segmentation and object detection crucial for determining the feasible region for path planning. Uniform characteristics of a road portion can be denoted by segmentations. Currently, road segmentation techniques mostly depend on the quality of camera images under different lighting conditions. However, Light Detection and Ranging (LiDAR) sensors can provide extremely precise 3D geometry information about the surroundings, leading to increased accuracy with increased memory consumption and computational overhead. This paper introduces a novel methodology which combines LiDAR and camera data for road detection, bridging the gap between 3D LiDAR Point Clouds (PCs). The assignment of semantic labels to 3D points is essential in various fields, including remote sensing, autonomous vehicles, and computer vision. This research discusses how to select the most relevant geometric features for path planning and improve autonomous navigation. An automatic framework for Semantic Segmentation (SS) is introduced, consisting of four processes: selecting neighborhoods, extracting classification features, and selecting features. The aim is to make the various components usable for end users without specialized knowledge by considering simplicity, effectiveness, and reproducibility. Through an extensive evaluation of different neighborhoods, geometric features, feature selection methods, classifiers, and benchmark datasets, the outcomes show that selecting the appropriate neighborhoods significantly develops 3D path segmentation. Additionally, selecting the right feature subsets can reduce computation time, memory usage, and enhance the quality of the results.