Accurate Semantic Segmentation Research Articles

Hierarchical Transfer Learning with Transformers to Improve Semantic Segmentation in Remote Sensing Land Use

Land use classification remains a significant challenge in remote sensing semantic segmentation. While convolutional neural networks (CNNs) are widely used, their inherent limitations, such as restricted receptive fields, hinder their widespread application in remote sensing. Additionally, the scarcity of labeled remote sensing data and domain shift issues adversely impact deep learning model performance. This study proposes a hierarchical transfer learning framework for fine-category semantic segmentation tasks, leveraging the powerful global relationship modeling capabilities of Transformer models to classify land use in Dongpo District, Meishan City, in mainland China. Our framework represents multilevel transfer learning, progressing from non-remote sensing classification to coarse classification, then to the refined classification of remote sensing. We compared the performance of Transformer models with representative baseline CNNs like U-Net and DeepLab V3+. Results show that the Swin-Unet model outperforms the other models used in this study. It achieved the highest test mean intersection over union (MIoU) of 0.837 and 0.810 for residential and transportation in level 1 (coarse) classification, respectively, and 0.545 for irrigated land in level 2 (fine-grained) classification. Transfer learning from pre-trained models significantly enhanced semantic segmentation accuracy compared to random parameter initialization (ranging from 0.4% to 17.7%), with up to a 17.7% improvement in test MIoU for the public land category. The hierarchical transfer learning framework further improved segmentation accuracy for corresponding level 2 categories, leveraging pre-trained level 1 models. Our study shows the applicability of Transformer-based transfer learning in remote sensing land use classification.

Enhancing Remote Sensing Semantic Segmentation Accuracy and Efficiency Through Transformer and Knowledge Distillation

Enhancing Remote Sensing Semantic Segmentation Accuracy and Efficiency Through Transformer and Knowledge Distillation

ETiSeg-Net: edge-aware self attention to enhance tissue segmentation in histopathological images

AbstractDigital pathology employing Whole Slide Images (WSIs) plays a pivotal role in cancer detection. Nevertheless, the manual examination of WSIs for the identification of various tissue regions presents formidable challenges due to its labor-intensive nature and subjective interpretation. Convolutional Neural Network (CNN) based semantic segmentation algorithms have emerged as valuable tools for assisting in this task by automating ROI delineation. The incorporation of attention modules and carefully designed loss functions has shown promise in further augmenting the performance of these algorithms. However, there exists a notable gap in research regarding the utilization of attention modules specifically for tissue segmentation, thereby constraining our comprehension and application of these modules in this context. This study introduces ETiSeg-Net (Edge-aware self attention to enhance Tissue Segmentation), a CNN-based semantic segmentation model that uses a novel edge-based attention module to achieve effective delineation of class boundaries. In addition, an innovative iterative training strategy is devised to efficiently optimize the model parameters. The study also conducts a comprehensive investigation into the impact of attention modules and loss functions on the efficacy of semantic segmentation models. Qualitative and quantitative evaluations of these semantic segmentation models are conducted using publicly available datasets. The findings underscore the potential of attention modules in enhancing the accuracy and effectiveness of tissue semantic segmentation.

Pseudo Multi-Modal Approach to LiDAR Semantic Segmentation.

To improve the accuracy and reliability of LiDAR semantic segmentation, previous studies have introduced multi-modal approaches that utilize additional modalities, such as 2D RGB images, to provide complementary information. However, these methods increase the cost of data collection, sensor hardware requirements, power consumption, and computational complexity. We observed that multi-modal approaches improve the semantic alignment of 3D representations. Motivated by this observation, we propose a pseudo multi-modal approach. To this end, we introduce a novel class-label-driven artificial 2D image construction method. By leveraging the close semantic alignment between image and text features of vision-language models, artificial 2D images are synthesized by arranging LiDAR class label text features. During training, the semantic information encoded in the artificial 2D images enriches the 3D features through knowledge distillation. The proposed method significantly reduces the burden of training data collection and facilitates more effective learning of semantic relationships in the 3D backbone network. Extensive experiments on two benchmark datasets demonstrate that the proposed method improves performance by 2.2-3.5 mIoU over the baseline using only LiDAR data, achieving performance comparable to that of real multi-modal approaches.

Channel2DTransformer: A Multi-level Features Self-attention Fusion Module for Semantic Segmentation

Semantic segmentation is a crucial technology for intelligent vehicles, enabling scene understanding in complex driving environments. However, complex real-world scenarios often contain diverse multi-scale objects, which bring challenges to the accurate semantic segmentation. To address this challenge, we propose a multi-level features self-attention fusion module called Channel2DTransformer. The module utilizes self-attention mechanisms to dynamically fuse multi-level features by computing self-attention weights between their channels, resulting in a consistent and comprehensive representation of scene features. We perform the module on the Cityscapes and NYUDepthV2 datasets, which contain a large number of multi-scale objects. The experimental results validate the positive contributions of the module in enhancing the semantic segmentation accuracy of multi-scale objects and improving the performance of semantic segmentation in complex scenes.

Comparative Assessment of Different Architectures of Convolutional Neural Network for Semantic Segmentation of Forest Disturbances from Multi-Temporal Satellite Images

Algorithms based on convolutional neural networks are the most efficient for semantic segmentation of images, including segmentation of forest cover disturbances from satellite images. In this study, we consider the applicability of various modifications of the U-net architecture of convolutional neural network for recognizing logged, burnt and windthrow areas in forests from multi-temporal and multi-seasonal Sentinel-2 satellite images. The assessment was carried out on three test sites that differ significantly in the characteristics of forest stands and forest management. The highest accuracy (average F-measure of 0.59) was obtained from the U-net model, while the models that showed the best results during training (Attention U-Net and MobilNetv2 U-Net) did not improve segmentation of independent data. The resulting accuracy estimates are close to those previously published for forests with a substantial proportion of selective logged areas. Characteristics of logged areas and windthrows, namely their area and type are the main factor determining the accuracy of semantic segmentation. Substantial differences were also revealed between the images taken in different seasons of the year, with the maximum segmentation accuracy based on winter pairs of images. According to summertime and different-season pairs of images, the area of forest disturbances is substantially underestimated. Forest species composition has a less significant effect, although for two of the three test sites, the maximum accuracy was observed in dark coniferous forests, and the minimum in deciduous forests. There was no statistically significant effect of slope lighting factor calculated from digital elevation model on segmentation accuracy based for winter pairs of images. The accuracy of segmentation of burnt areas, which was assessed using the example of 14 large forest fires in 2021-2022, is unsatisfactory, which is probably due to the varying degrees of damage to the forest cover in the burnt areas.

Spatial Attention-Based Kernel Point Convolution Network for Semantic Segmentation of Transmission Corridor Scenarios in Airborne Laser Scanning Point Clouds

Accurate semantic segmentation in transmission corridor scenes is crucial for the maintenance and inspection of power infrastructure, facilitating the timely detection of potential hazards. In this study, we propose SA-KPConv, an advanced segmentation model specifically designed for transmission corridor scenarios. Traditional approaches, including Random Forest and point-based deep learning models such as PointNet++, demonstrate limitations in segmenting critical infrastructure components, particularly power lines and towers, primarily due to their inadequate capacity to capture complex spatial relationships and local geometric details. Our model effectively addresses these challenges by integrating a spatial attention module with kernel point convolution, enhancing both global context and local feature extraction. Experiments demonstrate that SA-KPConv outperforms state-of-the-art methods, achieving a mean Intersection over Union (mIoU) of 89.62%, particularly excelling in challenging terrains such as mountainous areas. Ablation studies further validate the significance of our model’s components in enhancing overall performance and effectively addressing class imbalance. This study presents a robust solution for semantic segmentation, with considerable potential for monitoring and maintaining power infrastructure.

Mamba-UAV-SegNet: A Multi-Scale Adaptive Feature Fusion Network for Real-Time Semantic Segmentation of UAV Aerial Imagery

Accurate semantic segmentation of high-resolution images captured by unmanned aerial vehicles (UAVs) is crucial for applications in environmental monitoring, urban planning, and precision agriculture. However, challenges such as class imbalance, small-object detection, and intricate boundary details complicate the analysis of UAV imagery. To address these issues, we propose Mamba-UAV-SegNet, a novel real-time semantic segmentation network specifically designed for UAV images. The network integrates a Multi-Head Mamba Block (MH-Mamba Block) for enhanced multi-scale feature representation, an Adaptive Boundary Enhancement Fusion Module (ABEFM) for improved boundary-aware feature fusion, and an edge-detail auxiliary training branch to capture fine-grained details. The practical utility of our method is demonstrated through its application to farmland segmentation. Extensive experiments on the UAV-City, VDD, and UAVid datasets show that our model outperforms state-of-the-art methods, achieving mean Intersection over Union (mIoU) scores of 71.2%, 77.5%, and 69.3%, respectively. Ablation studies confirm the effectiveness of each component and their combined contributions to overall performance. The proposed method balances segmentation accuracy and computational efficiency, maintaining real-time inference speeds suitable for practical UAV applications.

Deep-Sea Fauna Segmentation: A Comparative Analysis of Convolutional and Vision Transformer Architectures at Lucky Strike Vent Field

Abstract. Due to recent technological developments, the acquisition and availability of deep-sea imagery has increased exponentially in the last years, leading to an increasing backlog in image annotation and processing, attributable to limited specialized human resources. In this work, we investigate the performance of well-established convolutional neural networks and Vision Transformer (ViT) based architectures, namely, DeepLabv3+ and UNETR, for the segmentation of fauna in deep-sea images. The dataset consists of images captured at the Lucky Strike Vent field, located on the mid-Atlantic ridge, of three edifices named Montsegur, White Castle, and Eiffel Tower. Our experimental investigation reveals that the Vision Transformer consistently outperforms the fully convolutional deep learning architecture, by approximately 14% in terms of F1-Score, demonstrating the effectiveness of ViTs in capturing intricate patterns and long-range dependencies present in deep-sea imagery. Our findings highlight the potential of ViTs as a promising approach for accurate semantic segmentation in challenging environmental contexts, paving the way for improved understanding and analysis of deep-sea ecosystems.

Application of ground-penetrating radar based on U-Net in the statistics on the bonded area of external wall insulation

Abstract The insulation of external walls of buildings often suffers from various diseases. Therefore, it is necessary to accurately detect the bonding area of the insulation layer. Ground-penetrating radar is a non-destructive detection method that can effectively determine the bonding status. In this paper, FDTD forward simulation of the exterior wall insulation layer by ground-penetrating radar is carried out, and the GPR images of the 4GHZ and 6GHZ antennas are simulated, respectively, and field experiments are conducted to collect the stepped frequency vector net data of 2.5-8GHZ and 6-8GHZ. The data obtained are processed by using the improved SP-ICA clutter suppression algorithm, followed by the improved loss function of the U-Net and training. The F1_score of the obtained model can reach 0.8399, and the error of bonding area statistics is 0.23%, which realizes more accurate semantic segmentation and area statistics.

Establishing effective learning bridge cross multi-scale feature maps for object detection and semantic segmentation

ABSTRACT In the field of image object detection and semantic segmentation, improving the accuracy of object identification and segmentation is a primary goal. To achieve this, leveraging the potential of multi-scale information through feature map refinement and fusion has been widely recognized. However, existing feature fusion methods either design more complex feature pyramid networks, replace existing detectors, or incrementally introduce feature fusion modules, overlooking the effective approach of enhancing spatial information in deep feature maps. We propose a novel pluggable feature fusion paradigm termed ‘Effective Learning Bridge’. Our research introduces an efficient and adaptive learning mechanism that builds learning bridges between feature maps at different scales within the feature pyramid, thereby enhancing the spatial information of objects in deep feature maps. This mechanism is specifically designed for multi-scale feature maps and can be seamlessly integrated into any network incorporating feature maps. By altering the model’s backpropagation path, we successfully improve learning efficiency, which in turn enhances the accuracy of object detection and segmentation. Our proposed paradigm and method were extensively evaluated through experiments on SIMD, HRSID, and WHDLD datasets and benchmark models. The results unequivocally demonstrate the effectiveness of our approach in significantly improving the accuracy of object detection and semantic segmentation, as well as the overall learning efficiency of the model.

AEFF-SSC: an attention-enhanced feature fusion for 3D semantic scene completion

Abstract Three-dimensional (3D) occupancy perception technology aims to enable autonomous vehicles to observe and understand dense 3D environments. Estimating the complete geometry and semantics of a scene solely from visual images is challenging. However, humans can easily conceive the complete form of objects based on partial key information and their own experience. This ability is crucial for recognizing and interpreting the surrounding environment. To equip 3D occupancy perception systems with a similar capability, a 3D semantic scene completion method called AEFF-SSC is proposed. This method deeply explores boundary and multi-scale information in voxels, aiming to reconstruct 3D geometry more accurately. We have specifically designed an attention-enhanced feature fusion module that effectively fuses image feature information from different scales and focuses on feature boundary information, thereby more efficiently extracting voxel features. Additionally, we introduce a semantic segmentation module driven by a 3D attention-UNet network. This module combines a 3D U-Net network with a 3D attention mechanism. Through feature fusion and feature weighting, it aids in restoring 3D spatial information and significantly improves the accuracy of segmentation results. Experimental verification on the SemanticKITTI dataset demonstrates that AEFF-SSC significantly outperforms other existing methods in terms of both geometry and semantics. Specifically, within a 12.8 m × 12.8 m area ahead, our geometric occupancy accuracy has achieved a significant improvement of 71.58%, and at the same time, the semantic segmentation accuracy has also increased remarkably by 54.20%.

CNCAN: Contrast and normal channel attention network for super-resolution image reconstruction of crops and weeds

Numerous studies have been performed to apply camera vision technologies in robot-based agriculture and smart farms. In particular, to obtain high accuracy, it is essential to procure high-resolution (HR) images, which requires a high-performance camera. However, due to high costs it is difficult to widely apply the camera in agricultural robots. To overcome this limitation, we propose contrast and normal channel attention network (CNCAN) for super-resolution reconstruction (SR), which is the first research for the accurate semantic segmentation of crops and weeds even with low-resolution (LR) images captured by low-cost and LR camera. Attention block and activation function that considers high frequency and contrast information of images are used in CNCAN, and the residual connection method is applied to improve the learning stability.As a result of experimenting with three open datasets, namely, Bonirob, rice seedling and weed, and crop/weed field image (CWFID) datasets, the mean intersection of union (MIOU) results of semantic segmentation for crops and weeds with SR images through CNCAN were 0.7685, 0.6346, and 0.6931 in the Bonirob, rice seedling and weed, and CWFID datasets, respectively, confirming higher accuracy than other state-of-the-art methods for SR.

Attention-Guided Residual U-Net with SE Connection and ASPP for Watershed-Based Cell Segmentation in Microscopy Images.

Time-lapse microscopy imaging is a crucial technique in biomedical studies for observing cellular behavior over time, providing essential data on cell numbers, sizes, shapes, and interactions. Manual analysis of hundreds or thousands of cells is impractical, necessitating the development of automated cell segmentation approaches. Traditional image processing methods have made significant progress in this area, but the advent of deep learning methods, particularly those using U-Net-based networks, has further enhanced performance in medical and microscopy image segmentation. However, challenges remain, particularly in accurately segmenting touching cells in images with low signal-to-noise ratios. Existing methods often struggle with effectively integrating features across different levels of abstraction. This can lead to model confusion, particularly when important contextual information is lost or the features are not adequately distinguished. The challenge lies in appropriately combining these features to preserve critical details while ensuring robust and accurate segmentation. To address these issues, we propose a novel framework called RA-SE-ASPP-Net, which incorporates Residual Blocks, Attention Mechanism, Squeeze-and-Excitation connection, and Atrous Spatial Pyramid Pooling to achieve precise and robust cell segmentation. We evaluate our proposed architecture using an induced pluripotent stem cell reprogramming dataset, a challenging dataset that has received limited attention in this field. Additionally, we compare our model with different ablation experiments to demonstrate its robustness. The proposed architecture outperforms the baseline models in all evaluated metrics, providing the most accurate semantic segmentation results. Finally, we applied the watershed method to the semantic segmentation results to obtain precise segmentations with specific information for each cell.

Unsupervised Multi-Scale Hybrid Feature Extraction Network for Semantic Segmentation of High-Resolution Remote Sensing Images

Generating pixel-level annotations for semantic segmentation tasks of high-resolution remote sensing images is both time-consuming and labor-intensive, which has led to increased interest in unsupervised methods. Therefore, in this paper, we propose an unsupervised multi-scale hybrid feature extraction network based on the CNN-Transformer architecture, referred to as MSHFE-Net. The MSHFE-Net consists of three main modules: a Multi-Scale Pixel-Guided CNN Encoder, a Multi-Scale Aggregation Transformer Encoder, and a Parallel Attention Fusion Module. The Multi-Scale Pixel-Guided CNN Encoder is designed for multi-scale, fine-grained feature extraction in unsupervised tasks, efficiently recovering local spatial information in images. Meanwhile, the Multi-Scale Aggregation Transformer Encoder introduces a multi-scale aggregation module, which further enhances the unsupervised acquisition of multi-scale contextual information, obtaining global features with stronger feature representation. The Parallel Attention Fusion Module employs an attention mechanism to fuse global and local features in both channel and spatial dimensions in parallel, enriching the semantic relations extracted during unsupervised training and improving the performance of unsupervised semantic segmentation. K-means clustering is then performed on the fused features to achieve high-precision unsupervised semantic segmentation. Experiments with MSHFE-Net on the Potsdam and Vaihingen datasets demonstrate its effectiveness in significantly improving the accuracy of unsupervised semantic segmentation.

3D Point Cloud Semantic Segmentation based on Multi-scale Dense Nested Networks

Aiming at the problem that the relationship between geometric features and semantic features is ignored in the point cloud data downsampling process, which leads to inaccurate segmentation of object boundaries and structural details, this paper proposes a 3D point cloud semantic segmentation network based on multi-scale dense nested type. Firstly, a dense nested network architecture is constructed by nesting multiple multi-scale feature fusion modules to fuse multi-scale features of different directions between encoder-decoder paths, so as to effectively propagate local ge-ometric context information and enhance the ability of cross-scale information interaction. Secondly, a local feature aggregation unit is constructed in the multi-scale feature fusion module, which strengthens the structural awareness within the local point set based on graph convolution and attention mechanism, and promotes the complementarity of local geometric features and abstract semantic information. Then, the cross-layer multi-loss supervision module is combined to further optimize the multi-scale feature propagation, which makes the network training more stable and improves the point cloud segmentation accuracy. Finally, this paper verified the proposed network on the S3DIS dataset. The experimental results show that the proposed network has the mean intersection over Union of 71.2% and the overall accuracy of 88.7%, which is 1.2 and 0.7 percentage points higher than RandLA-Net, respectively, which proves that the proposed network can effectively improve the accuracy of 3D point cloud semantic segmentation.

Real-Time Semantic Segmentation Algorithm for Street Scenes Based on Attention Mechanism and Feature Fusion

In computer vision, the task of semantic segmentation is crucial for applications such as autonomous driving and intelligent surveillance. However, achieving a balance between real-time performance and segmentation accuracy remains a significant challenge. Although Fast-SCNN is favored for its efficiency and low computational complexity, it still faces difficulties when handling complex street scene images. To address this issue, this paper presents an improved Fast-SCNN, aiming to enhance the accuracy and efficiency of semantic segmentation by incorporating a novel attention mechanism and an enhanced feature extraction module. Firstly, the integrated SimAM (Simple, Parameter-Free Attention Module) increases the network’s sensitivity to critical regions of the image and effectively adjusts the feature space weights across channels. Additionally, the refined pyramid pooling module in the global feature extraction module captures a broader range of contextual information through refined pooling levels. During the feature fusion stage, the introduction of an enhanced DAB (Depthwise Asymmetric Bottleneck) block and SE (Squeeze-and-Excitation) attention optimizes the network’s ability to process multi-scale information. Furthermore, the classifier module is extended by incorporating deeper convolutions and more complex convolutional structures, leading to a further improvement in model performance. These enhancements significantly improve the model’s ability to capture details and overall segmentation performance. Experimental results demonstrate that the proposed method excels in processing complex street scene images, achieving a mean Intersection over Union (mIoU) of 71.7% and 69.4% on the Cityscapes and CamVid datasets, respectively, while maintaining inference speeds of 81.4 fps and 113.6 fps. These results indicate that the proposed model effectively improves segmentation quality in complex street scenes while ensuring real-time processing capabilities.

LESA-Net: Semantic segmentation of multi-type road point clouds in complex agroforestry environment

Point-cloud semantic segmentation is a visual task essential for agricultural robots to comprehend natural agroforestry environments. However, owing to the extremely large amount of point-cloud data in agroforestry environments, learning effective features for semantic segmentation from large-scale point clouds is challenging. Therefore, to address this issue and achieve accurate semantic segmentation of different types of road-surface point clouds in large-scale agroforestry environments, this study proposes a point-cloud semantic segmentation network framework based on double-distance self-attention. First, a point-cloud local feature enhancement module is proposed. This module primarily extends the receptive field and enhances the generalizability of multidimensional features by incorporating reflection intensity information and a spatial feature-encoding block that is enhanced with contextual semantic information. Second, we introduce a dual-distance attention pooling (DDAPS) block based on the self-attention mechanism. This block initially learns the feature representation of the local neighborhood of each point through the self-attention mechanism. Then, it uses the DDAPS block to aggregate more discriminative local neighborhood point features. Finally, extensive experimental results on large-scale point-cloud datasets, SemanticKITTI and RELLIS-3D, demonstrate that our algorithm outperforms similar algorithms in large-scale agroforestry environments.

Containment Control-Guided Boundary Information for Semantic Segmentation

Real-time semantic segmentation is a challenging task in computer vision, especially in complex scenes. In this study, a novel three-branch semantic segmentation model is designed, aiming to effectively use boundary information to improve the accuracy of semantic segmentation. The proposed model introduces the concept of containment control in a pioneering way, which treats image interior elements as well as image boundary elements as followers and leaders in containment control, respectively. Based on this, we utilize two learnable feature fusion matrices in the high-level semantic information stage of the model to quantify the fusion process of internal and boundary features. Further, we design a dedicated loss function to update the parameters of the feature fusion matrices based on the criterion of containment control, which enables fine-grained communication between target features. In addition, our model incorporates a Feature Enhancement Unit (FEU) to tackle the challenge of maximizing the utility of multi-scale features essential for semantic segmentation tasks through the meticulous reconstruction of these features. The proposed model proves effective on the publicly available Cityscapes and CamVid datasets, achieving a trade-off between effectiveness and speed.

Semantic segmentation based on multimodal information is used for object detection during autonomous driving at night

Currently, it is difficult for autonomous driving to detect objects accurately enough at night, mainly due to the poor light at night. The laser camera’s captured RGB image information is less effective than during the day, and the image content is seriously affected. The blurred outline of the object and the decreased accuracy of semantic segmentation lead to missed detections. In view of this situation, this study proposes the following solutions: introducing point cloud data collected by lidar during data acquisition, combining it with the image information from the camera, to constitute precise multimodal three-dimensional information. Then, using a dual adversarial network to preprocess the data and U-net semantic segmentation to segment the multimodal 3D information. The simulation experiment is then used to test and evaluate the segmentation effect. After the completion of the training, the object information detection is applied during the autonomous driving of the vehicle. This method, when compared with the general method, has accurate detection, is independent of the intensity of the light, and has the advantage of good universality.