Semantic Segmentation Research Articles

A road semantic segmentation system for remote sensing images based on deep learning

A road semantic segmentation system for remote sensing images based on deep learning

Multi-Bottleneck progressive propulsion network for medical image semantic segmentation with integrated macro-micro dual-stage feature enhancement and refinement

Semantic segmentation can assist doctors to locate key lesion areas intelligently, rapidly and with high precision in medical imaging, which is of great clinical significance for the diagnosis and rehabilitation of patients in various medical fields such as oncology and neurology. Existing segmentation networks face accuracy bottlenecks because enriched information at the decoding end can’t disperse over multiple layers of semantic depth. This is compounded by challenges like complex anatomical structures and varying imaging modalities. In this study, we propose a Multi-Bottleneck Progressive Semantic Segmentation Network (MBP-SSNet), which is essentially a new Baseline based on our proposal: Multi-Bottleneck based Progressive Propulsion Feature Guidance Network (MB-PPFG Net). Its Multi-Bottleneck solves the problem of coding and decoding multiplexed transmission of enriched information, and the propulsive transmission alleviates the problem of insufficient semantic fusion. In addition, under the macro–micro processing idea, MacroFocus Pre-Bottleneck Feature Enhancer (MPBFE) is proposed at the codec intersection in order to better enhance the image contextual semantic dependency association; while MicroDetail Post- Decoding Feature Refiner (MPDFR) is proposed at the tail of the network for fine-grained polishing. In the experiment, the MBP-SSNet was compared with 17 other models across 7 datasets, including diverse medical imaging scenarios, involving metric exploration, image comparison, process validation, and statistical analysis. Concurrently, 22 types of ablation analyses were conducted on it, along with detailed discussions on two critical issues. The experimental results demonstrate that MBP-SSNet achieves an average performance improvement of 0.6809 % in medical image segmentation compared to the optimal benchmark model. The official web address of MBP-SSNet is at: https://github.com/YF-W/MBP-SSNet.

Status of the Deep Learning-Based Shattered Pellet Injection Shard Tracking at ASDEX Upgrade

Plasma disruptions pose an intolerable risk to large tokamaks, such as ITER. If a disruption can no longer be avoided, ITER’s last line of defense will be the Shattered Pellet Injection. An experimental test bench was created at ASDEX Upgrade to inform the design decisions for controlling the shattering of the pellets and develop the techniques for the generation of the fragment distributions necessary for optimal disruption mitigation. In an effort to analyze the videos resulting from the more than 1000 tests and determine the impact of different settings on the resulting shard cloud, an analysis pipeline, based on traditional computer vision (CV), was created. This pipeline enabled the analysis of 173 of the videos, but at the same time showed the limits of traditional CV when applied in applications with a highly heterogeneous dataset such as this. We created a machine learning-based (ML) alternative as a drop-in replacement to the original image processing code using a semantic segmentation model to exploit the innate adaptability and robustness of deep learning models. This model is capable of labeling the entire dataset quickly, accurately and reliably. This contribution details the implementation of the ML model and the current state and future plans of the project.

PRCL: Probabilistic Representation Contrastive Learning for Semi-Supervised Semantic Segmentation

PRCL: Probabilistic Representation Contrastive Learning for Semi-Supervised Semantic Segmentation

HMNNet: research on exposure-based nighttime semantic segmentation

HMNNet: research on exposure-based nighttime semantic segmentation

Surface Defect Detection and Semantic Segmentation with a Novel Lightweight Deep Neural Network

Abstract Current approaches to defect detection and segmentation make essential use of machine learning methods. To develop lightweight models is one of key tasks for many defect detection and segmentation applications. In this work, we present a lightweight trilateral parallel feature extraction with multi-feature aggregation network (TriMFANet) for surface defect detection and segmentation. In TriMFANet, the top lateral is the feature-rich extraction (FrE) used to capture detailed information. The other two laterals, efficient semantic feature extraction (ESFE) and reverse efficient semantic feature extraction (reESFE), leverage Hadamard product (HP) attention to jointly extract deep-level global feature information. Additionally, the multi-feature aggregation (MFA) module employs origin-symmetric sigmoid (OS) attention to enhance deep feature information and integrates the triple features. We conducted binary defect segmentation tasks on the SD-saliency-900 and RSDDs datasets, achieving outstanding performance in both Sα and Eξ. For multi-class defect detection tasks on the NEU-Seg and MSD datasets, we rank first with mIoU scores of 79.0% and 81.2% respectively. Experimental results demonstrate that our lightweight model with only 90K parameters exhibits excellent performance.

ABNet: An Aggregated Backbone Network Architecture for Fine Landcover Classification

High-precision landcover classification is a fundamental prerequisite for resource and environmental monitoring and land-use status surveys. Imbued with intricate spatial information and texture features, very high spatial resolution remote sensing images accentuate the divergence between features within the same category, thereby amplifying the complexity of landcover classification. Consequently, semantic segmentation models leveraging deep backbone networks have emerged as stalwarts in landcover classification tasks owing to their adeptness in feature representation. However, the classification efficacy of a solitary backbone network model fluctuates across diverse scenarios and datasets, posing a persistent challenge in the construction or selection of an appropriate backbone network for distinct classification tasks. To elevate the classification performance and bolster the generalization of semantic segmentation models, we propose a novel semantic segmentation network architecture, named the aggregated backbone network (ABNet), for the meticulous landcover classification. ABNet aggregates three prevailing backbone networks (ResNet, HRNet, and VoVNet), distinguished by significant structural disparities, using a same-stage fusion approach. Subsequently, it amalgamates these networks with the Deeplabv3+ head after integrating the convolutional block attention mechanism (CBAM). Notably, this amalgamation harmonizes distinct scale features extracted by the three backbone networks, thus enriching the model’s spatial contextual comprehension and expanding its receptive field, thereby facilitating more effective semantic feature extraction across different stages. The convolutional block attention mechanism primarily orchestrates channel adjustments and curtails redundant information within the aggregated feature layers. Ablation experiments demonstrate an enhancement of no less than 3% in the mean intersection over union (mIoU) of ABNet on both the LoveDA and GID15 datasets when compared with a single backbone network model. Furthermore, in contrast to seven classical or state-of-the-art models (UNet, FPN, PSPNet, DANet, CBNet, CCNet, and UPerNet), ABNet evinces excellent segmentation performance across the aforementioned datasets, underscoring the efficiency and robust generalization capabilities of the proposed approach.

Semantic Segmentation for Change Detection in Satellite Imaging

Change detection is a common and actual problem in the field of remote sensing. The classical approaches using raw pixel information are very sensitive to noise. In this study we propose the usage of additional semantic information for change detection. We use the semantic segmentation methods like geospatial Segment Anything Model and encoder based U-Net to evaluate the predictions and tracing the semantic information as well as raw information in change detection. Later the multidimensional time series data is used via the Vector Autoregression model to predict the future changes in the landscape. The observations which fall out of the prediction interval are considered as the changes in the landscape. The proposed method is evaluated on the dataset of the random locations across the Baltic region. The research is accompanied by the data and reproducible code at Github repository.

SLM-SLAM: A Visual SLAM System Based on Segmented Large-Scale Model in Dynamic Scenes and Zero-shot Conditions

Abstract In the real world, the presence of various dynamic objects can impact the localization accuracy of the majority of typical Visual Simultaneous Localization and Mapping (VSLAM) systems. Simultaneously, many dynamic VSLAM systems based on neural networks require pre-training for specific application scenarios. We present SLM-SLAM, the first VSLAM system that implements zero-shot processing of dynamic scenes. It achieves the capability to handle various dynamic objects without the necessity for pre-training, enabling straightforward adaptation to different application scenarios. Firstly, we designed an open-world semantic segmentation module based on a segmented large-scale model to acquire semantic information in the scene. Subsequently, we devised a label-based strategy for selecting feature points, jointly optimizing poses with the weighted labels provided by both semantic and geometric information. Finally, we refined the keyframe selection strategy of ORB-SLAM3 to prevent matching errors caused by an insufficient number of remaining static feature points in the scene. We conducted experiments on the TUM dataset, the KITTI dataset, and real world scenarios. The results indicate that in dynamic scenes, our SLM-SLAM significantly improves localization accuracy compared to ORB-SLAM3, and its performance is comparable to state-of-the-art dynamic VSLAM systems.

UAV Imagery-based Railroad Station Building Inspection Using Hybrid Learning Architecture

Abstract The apparent defects of railroad station building such as the broken glass, missing screw, and rust corrosion need to be inspected regularly and timely to ensure the safety of passenger. Unmanned Aerial Vehicle (UAV) imagery-based inspections have the potential to revolutionize current manual visual inspections by providing a better overhead view and mitigating safety concerns. This paper proposes a hybrid learning architecture called YOLOS (You Only Look Once Station Scene) to simultaneously detect and segment station building surface defects of UAV images. First, a novel Squeeze-and-Excitation (SE) attention block is integrated into the detection branch to adaptively learn the weights of the feature channels, thereby promoting the network to pay attention to the critical deep features of the objects. And then, a new semantic segmentation branch parallel to the detection branch is designed and assembled in YOLOS for pixel-level defect recognition. Finally, extensive experiments on railroad station dataset established with drone imagery prove the effectiveness and robustness of the SE-based network on station surface defect detection. This method can quickly convert UAV imagery into useful information with a high detection rate.

Weakly Supervised Learning Method for Semantic Segmentation of Large-Scale 3D Point Cloud Based on Transformers

Abstract. Nowadays, semantic segmentation results of 3D point cloud have been widely applied in the fields of robotics, autonomous driving, and augmented reality etc. Thanks to the development of relevant deep learning models (such as PointNet), supervised training methods have become hotspot, in which two common limitations exists: inferior feature representation of 3D points and massive annotations. To improve 3D point feature, inspired by the idea of transformer, we employ a so-call LCP network that extracts better feature by investigating attentions between target 3D points and its corresponding local neighbors via local context propagation. Training transformer-based network needs amount of training samples, which itself is a labor-intensive, costly and error-prone work, therefore, this work proposes a weakly supervised framework, in particular, pseudo-labels are estimated based on the feature distances between unlabeled points and prototypes, which are calculated based on labeled data. The extensive experimental results show that, the proposed PL-LCP can yield considerable results (67.6% mIOU for indoor and 67.3% for outdoor) even if only using 1% real labels, and comparing to several state-of-the-art method using all labels, we achieve superior results in mIOU, OA for indoor (65.9%, 89.2%).

Fine-tuning vision foundation model for crack segmentation in civil infrastructures

Large-scale foundation models have become the mainstream deep learning method, while in civil engineering, the scale of AI models is strictly limited. In this work, a vision foundation model is introduced for crack segmentation. Two parameter-efficient fine-tuning methods, adapter and low-rank adaptation, are adopted to fine-tune the foundation model in semantic segmentation: the Segment Anything Model (SAM). The fine-tuned CrackSAM shows excellent performance on different scenes and materials. To test the zero-shot performance of the proposed method, two unique datasets related to road and exterior wall cracks are collected, annotated and open-sourced, for a total of 810 images. Comparative experiments are conducted with twelve mature semantic segmentation models. On datasets with artificial noise and previously unseen datasets, the performance of CrackSAM far exceeds that of all state-of-the-art models. CrackSAM exhibits remarkable superiority, particularly under challenging conditions such as dim lighting, shadows, road markings, construction joints, and other interference factors. These cross-scenario results demonstrate the outstanding zero-shot capability of foundation models and provide new ideas for developing vision models in civil engineering.

A Multiscale Attention Segment Network-Based Semantic Segmentation Model for Landslide Remote Sensing Images

Landslide disasters have garnered significant attention due to their extensive devastating impact, leading to a growing emphasis on the prompt and precise identification and detection of landslides as a prominent area of research. Previous research has primarily relied on human–computer interactions and visual interpretation from remote sensing to identify landslides. However, these methods are time-consuming, labor-intensive, subjective, and have a low level of accuracy in extracting data. An essential task in deep learning, semantic segmentation, has been crucial to automated remote sensing image recognition tasks because of its end-to-end pixel-level classification capability. In this study, to mitigate the disadvantages of existing landslide detection methods, we propose a multiscale attention segment network (MsASNet) that acquires different scales of remote sensing image features, designs an encoder–decoder structure to strengthen the landslide boundary, and combines the channel attention mechanism to strengthen the feature extraction capability. The MsASNet model exhibited an average accuracy of 95.13% on the test set from Bijie’s landslide dataset, a mean accuracy of 91.45% on the test set from Chongqing’s landslide dataset, and a mean accuracy of 90.17% on the test set from Tianshui‘s landslide dataset, signifying its ability to extract landslide information efficiently and accurately in real time. Our proposed model may be used in efforts toward the prevention and control of geological disasters.

Reinforcement learning (RL)-based semantic segmentation and attention based backpropagation convolutional neural network (ABB-CNN) for breast cancer identification and classification using mammogram images

Reinforcement learning (RL)-based semantic segmentation and attention based backpropagation convolutional neural network (ABB-CNN) for breast cancer identification and classification using mammogram images

Multi-sensor Data Analysis for Aerial Image Semantic Segmentation and Vectorization

Abstract. One of the urgent and constantly in demand problems is updating maps. Maps, representing geo-information in vector form, have undoubted advantages in compactness and ”readability” compared to aerial photographs. The issue of maps actuality is critically important for rational urban planning, precision farming, the relevance of the cadastre and other geospatial applications. Various sources of data are used for maps updating, with aerial imagery being the main and rich source of information. Automatic processing of aerial photographs makes it possible to efficiently extract vector information, providing operational monitoring and accounting for changes that have appeared. The presented study addresses the problem of multi sensor information fusion in order to obtain accurate vector information. We use aerial images as a main data source and additionally the data of laser scanning and ground survey to increase performance of automatic image semantic segmentation and vectorization. The proposed framework is demonstrated on the task of forest monitoring.

Lumbar spine MRI annotation with intervertebral disc height and Pfirrmann grade predictions.

Many lumbar spine diseases are caused by defects or degeneration of lumbar intervertebral discs (IVD) and are usually diagnosed through inspection of the patient's lumbar spine MRI. Efficient and accurate assessments of the lumbar spine are essential but a challenge due to the size of the clinical radiologist workforce not keeping pace with the demand for radiology services. In this paper, we present a methodology to automatically annotate lumbar spine IVDs with their height and degenerative state which is quantified using the Pfirrmann grading system. The method starts with semantic segmentation of a mid-sagittal MRI image into six distinct non-overlapping regions, including the IVD and vertebrae regions. Each IVD region is then located and assigned with its label. Using geometry, a line segment bisecting the IVD is determined and its Euclidean distance is used as the IVD height. We then extract an image feature, called self-similar color correlogram, from the nucleus of the IVD region as a representation of the region's spatial pixel intensity distribution. We then use the IVD height data and machine learning classification process to predict the Pfirrmann grade of the IVD. We considered five different deep learning networks and six different machine learning algorithms in our experiment and found the ResNet-50 model and Ensemble of Decision Trees classifier to be the combination that gives the best results. When tested using a dataset containing 515 MRI studies, we achieved a mean accuracy of 88.1%.

Changes in the Water Area of an Inland River Terminal Lake (Taitma Lake) Driven by Climate Change and Human Activities, 2017–2022

Constructed from a dataset capturing the seasonal and annual water body distribution of the lower Qarqan River in the Taitma Lake area from 2017 to 2022, and combined with the meteorological and hydraulic engineering data, the spatial and temporal change patterns of the Taitma Lake watershed area were determined. Analyses were conducted using Planetscope (PS) satellite images and a deep learning model. The results revealed the following: ① Deep learning-based water body extraction provides significantly greater accuracy than the conventional water body index approach. With an impressive accuracy of up to 96.0%, UPerNet was found to provide the most effective extraction results among the three convolutional neural networks (U-Net, DeeplabV3+, and UPerNet) used for semantic segmentation; ② Between 2017 and 2022, Taitma Lake’s water area experienced a rapid decrease, with the distribution of water predominantly shifting towards the east–west direction more than the north–south. The shifts between 2017 and 2020 and between 2020 and 2022 were clearly discernible, with the latter stage (2020–2022) being more significant than the former (2017–2020); ③ According to observations, Taitma Lake’s changing water area has been primarily influenced by human activity over the last six years. Based on the research findings of this paper, it was observed that this study provides a valuable scientific basis for water resource allocation aiming to balance the development of water resources in the middle and upper reaches of the Tarim and Qarqan Rivers, as well as for the ecological protection of the downstream Taitma Lake.

An Ensemble Learning Framework for Anomaly Detection of Important Geographical Entities

Abstract. Due to the complex landforms and the limited resolution of remote sensing imagery, it is difficult to avoid the problem of incorrectly capturing geographical entities, such as buildings. Therefore, anomaly detection of important geographical entities is of great significance to ensure the authenticity and accuracy of geographical entity data. In this paper, we propose an ensemble learning framework for anomaly detection of geographical entity by aggregating the predicted labels generated by multiple deep learning models. In detail, we explore multiple change detection and semantic segmentation model and fully utilize the advantages of various deep learning neural network architectures. The proposed anomaly detection strategy of buildings has been performed on two benchmark datasets, including WHU Building change detection dataset and LEVIR building change detection dataset, the experimental results prove that the proposed method can achieve a more robust and better performance than using single change detection model in terms of quantitative performance and visual performance.

Crop and Weed Segmentation and Fractal Dimension Estimation Using Small Training Data in Heterogeneous Data Environment

The segmentation of crops and weeds from camera-captured images is a demanding research area for advancing agricultural and smart farming systems. Previously, the segmentation of crops and weeds was conducted within a homogeneous data environment where training and testing data were from the same database. However, in the real-world application of advancing agricultural and smart farming systems, it is often the case of a heterogeneous data environment where a system trained with one database should be used for testing with a different database without additional training. This study pioneers the use of heterogeneous data for crop and weed segmentation, addressing the issue of degraded accuracy. Through adjusting the mean and standard deviation, we minimize the variability in pixel value and contrast, enhancing segmentation robustness. Unlike previous methods relying on extensive training data, our approach achieves real-world applicability with just one training sample for deep learning-based semantic segmentation. Moreover, we seamlessly integrated a method for estimating fractal dimensions into our system, incorporating it as an end-to-end task to provide important information on the distributional characteristics of crops and weeds. We evaluated our framework using the BoniRob dataset and the CWFID. When trained with the BoniRob dataset and tested with the CWFID, we obtained a mean intersection of union (mIoU) of 62% and an F1-score of 75.2%. Furthermore, when trained with the CWFID and tested with the BoniRob dataset, we obtained an mIoU of 63.7% and an F1-score of 74.3%. We confirmed that these values are higher than those obtained by state-of-the-art methods.

Research on Satellite and Ground Multi-Sensor Collaborative Sensing Method for Floating Plants

Abstract. Dynamic monitoring of water environment is the basis of maintaining urban security and promoting urban sustainable development. To effectively solve the problems caused by the large-scale flooding of floating aquatic invasive plants, such as water environment destruction, clogging river, water quality pollution, water ecological balance destruction, etc., a satellite and ground multi-sensor collaborative sensing method for floating plants is proposed in this paper. For the monitoring of floating plants in macro watershed, the vegetation index and chlorophyll concentration were used to extract floating plants, analyzing, and calculating the distribution and coverage area of floating plants based on 4-band high-resolution satellite images. For the monitoring of floating plants in small watershed, combined with visible light video, multispectral images, and LiDAR data, optimized spectral feature variable processing, semantic segmentation network, and symmetric function neural network are utilized to obtain real-time information on the invasion location, distribution pattern, and coverage area of floating plants. In this paper, the upstream basin of the Huangpu River in Shanghai in 2023 was selected as the experimental area. The distribution of floating plants in different watercourse and different periods was successfully obtained by this method, and the coverage area was accurately calculated with an identification rate of more than 90%, which provided technical support for cross-regional management and efficient cleaning of floating plants.