Semantic Segmentation Network Research Articles

Efficient Multi-scale Network for Semantic Segmentation of fine-Resolution Remotely Sensed Images

Abstract Semantic segmentation of remote sensing urban scene images has diverse practical applications, including land cover mapping, urban change detection, environmental protection, and economic evaluation. However, classical semantic segmentation networks encounter challenges such as inadequate utilization of multi-scale semantic information and imprecise edge target segmentation in high-resolution remote sensing images. In response, this article introduces an Efficient Multi-scale Network (EMNet) tailored for semantic segmentation of common features in remote sensing images.To address these challenges, EMNet integrates several key components. Firstly, the Efficient Atrous Spatial Pyramid Pooling module(EASPP) is employed to enhance the relevance of multi-scale targets, facilitating improved extraction and processing of context information across different scales. Secondly, the Efficient Multi-Scale Attention mechanism (EMA)and multi-scale jump connections are utilized to fuse semantic features from various levels, thereby achieving precise segmentation boundaries and accurate position information. Finally, an encoder-decoder structure is incorporated to refine the segmentation results.The effectiveness of the proposed network is validated through experiments conducted on the publicly available DroneDeploy image dataset. Results indicate that EMNet achieves impressive performance metrics, with Mean Intersection over Union (MIoU), Mean Pixel Accuracy (MPA), Mean Precision (MPrecision), and Mean Recall (MRecall) reaching 75.99%, 85.08%, 86.76%, and 85.07%, respectively. Comparative analysis demonstrates that the network proposed in this article outperforms current mainstream semantic segmentation networks on the DroneDeploy dataset.

Lightweight railroad semantic segmentation network and distance estimation for railroad Unmanned aerial vehicle images

Derailments significantly harm railroads in terms of severity and fatality rates. Manually monitoring railway tracks is a tedious and often insufficient task to prevent derailment accidents. This necessitates the development of an automated monitoring system to oversee the condition of railway tracks. In the foreseeable future, Unmanned Aerial Vehicles (UAVs), artificial intelligence, and computer vision will play a crucial role in periodically assessing the railway environment to ensure passenger safety. This paper proposes a lightweight and efficient railroad semantic segmentation network (Lite-RSNet), which segments real-time aerial railroad images into rails, gauges, and backgrounds. The segmented regions then compute the distance between rails and obstacles, identifying potential derailment hazards in railroad applications. Lite-RSNet incorporates a multi-level feature fusion framework that includes a residual layer, a multi-head attention layer, a Concurrent Spatial and Channel Squeeze Excitation (CSSE) block, and a Convolutional Block Attention Module (CBAM). The designed approach enhances onboard processing efficiency by streamlining the model to use fewer parameters and reduce storage demands. Performance assessment of the model utilized the Rail Segmentation Dataset (RSD), which consists of high-resolution, low-altitude aerial photographs of Indian railroads. The Lite-RSNet model outperforms current leading methods, maintaining a lean structure with only 6 million parameters and achieving notable metrics of 0.971 for the Dice Coefficient (DC) and 0.947 for the Jaccard Index (JI) on the test dataset. Furthermore, distance estimation between an obstacle and rail has been implemented using image processing techniques as an additional work of this proposal to know about the caution of an obstacle.

Multimodality Image Segmentation Embedded with Metasurface Encoder

AbstractFeature dimensionality plays an important role for modern medical diagnosis and image processing. In this work, the study introduces an optoelectronic neural network for multimodal image segmentation, which dramatically improves computing speed and decreases imaging acquisition cost in brain tumor diagnostics. Multi‐layer metasurfaces are utilized as an image preprocessor that reduces image dimensionality at the physical layer. The low‐dimensional image is then processed to a U‐Net semantic segmentation network, to handle the complex and heterogeneous nature of brain image data. By using diffractive neural network, the metasurface encoder is optimized and physically constructed with high‐efficiency transmission metasurfaces. The entire optoelectronic network attains a structural similarity index measure (SSIM) of 96%, demonstrating its potential to revolutionize on‐site medical image processing with its high precision in segmenting brain imaging data.

Segmentation of ethnic clothing patterns with fusion of multiple attention mechanisms

AbstractTo address the challenges posed by diverse pattern-background elements, intricate details, and complex textures in the semantic segmentation of ethnic clothing patterns, this research introduces a novel semantic segmentation network model called MST-Unet (Mixed Swin Transformer U-net). The proposed model combines a U-shaped network structure with multiple attention mechanisms. The upper layers of the model employ classical convolutional operations, focusing on local relationships in the initial layers containing high-resolution details. In deeper layers, Swin Transformer modules are utilized, capable of efficient feature extraction with smaller spatial dimensions, maintaining performance while reducing computational burden. An attention gate mechanism is integrated into the decoder, contributing to enhanced performance in ethnic clothing pattern segmentation tasks by allowing the model to better capture crucial image features and achieve precise segmentation results. In visual comparisons of segmentation results, our proposed model demonstrates superior performance. The segmentation results exhibit more complete preservation of edge contours and fewer misclassifications in irrelevant regions within the images. In qualitative and quantitative experiments conducted on the ethnic clothing pattern dataset, our model achieves the highest Dice score for segmentation results in all four subclasses of ethnic clothing patterns. The average Dice score of our model reaches an impressive 89.80%, surpassing other algorithms in the same category. When compared to Deeplab_V3+, ResUnet, SwinUnet, and Unet networks, our model outperforms them by 7.72%, 5.09%, 5.05%, and 0.67%.

Unstructured Road Extraction in UAV Images based on Lightweight Model

There is no unified planning standard for unstructured roads, and the morphological structures of these roads are complex and varied. It is important to maintain a balance between accuracy and speed for unstructured road extraction models. Unstructured road extraction algorithms based on deep learning have problems such as high model complexity, high computational cost, and the inability to adapt to current edge computing devices. Therefore, it is best to use lightweight network models. Considering the need for lightweight models and the characteristics of unstructured roads with different pattern shapes, such as blocks and strips, a TMB (Triple Multi-Block) feature extraction module is proposed, and the overall structure of the TMBNet network is described. The TMB module was compared with SS-nbt, Non-bottleneck-1D, and other modules via experiments. The feasibility and effectiveness of the TMB module design were proven through experiments and visualizations. The comparison experiment, using multiple convolution kernel categories, proved that the TMB module can improve the segmentation accuracy of the network. The comparison with different semantic segmentation networks demonstrates that the TMBNet network has advantages in terms of unstructured road extraction.

Combined multi-level context aggregation and attention mechanism method for photovoltaic panel extraction from high resolution remote sensing images

ABSTRACT In the context of global carbon emission reduction, solar photovoltaic (PV) technology is experiencing rapid development. Using high-resolution remote sensing images to accurately obtain PV information over a large region, including location and size, has the advantages of high statistical efficiency and timely data update for the PV energy management. Due to the intra-class diversity of PV panels and the intricate variability in their deployment environments, existing semantic segmentation methods often have problems such as under-segmentation and mis-segmentation. To alleviate these problems, this paper proposes an improved DeepLabv3+ semantic segmentation network to more accurately extract PV panels from high-resolution remote sensing images. With the aim of alleviating under-segmentation, a multi-level context aggregation module is developed. This module can enhance the model’s ability to learn the characteristics of PV panels and their surrounding environment by aggregating rich contextual information from multi-scale and semantic levels. To alleviate the problem of mis-segmentation, a hybrid attention module is introduced. This module sequentially and adaptively adjusts the weight distribution in both the channel and spatial dimensions, thus enabling the model to focus more on the feature information and spatial positions of PV objects. Experiments conducted on a self-constructed Beijing PV segmentation dataset show that the method in this paper has advantages of completeness and accuracy in extracting PV panels compared to the baseline model and current mainstream semantic segmentation network. In addition, the results of experiments on extracting PV panels in real region show that our model also has good stability and generalization capability.

Building Extraction from LiDAR Point Clouds Based on Revised RandLA-Net

Abstract. 3D building models is crucial for applications in smart cities. Automatic reconstruction of 3D buildings has been investigated based on various data sources. Point clouds from airborne LiDAR scanners can be used to extract buildings data due to its high accuracy and point density. In this paper, we present a methodology to segment buildings and corresponding rooftop structure from point clouds. First, RandLA-Net, which is an efficient and lightweight neural network for semantic segmentation of large-scale point clouds, is revised and adopted for building segmentation. By implementing local feature aggregation of each point, RandLA-Net can effectively preserve geometric details in point clouds. Besides 3D coordinates of point clouds, we incorporated point attributes including pulse intensity and return numbers into the network as additional features. Feature normalizations are applied to the input features. To achieve a better result of the local feature aggregation, hyperparameters of the network are fine-tuned according to the density of points and building size. Based on the classified building point clouds, DBSCAN clustering algorithm is implemented for segmenting individual buildings. Elevation histogram analysis is conducted to determine optimal threshold values for delineating candidate rooftop point clouds of individual buildings. For the buildings with multiple rooftops, multiple elevation threshold values are necessary to extract corresponding rooftops or walls. Then DBSCAN is employed again for segmentation of individual rooftops and denoising of point clouds of each building. Finally, Alpha-shape analysis is applied based on adaptive threshold values to build the envelope of each rooftop. Experiments show that our implementation of building segmentation using RandLA-net achieves higher mean IoU (Intersection over Union) and better classification performance in building segmentation. ISPRS benchmark data was used in our experiment and our methodology produce results with accuracy of 90.79%.

Multi-Bottleneck progressive propulsion network for medical image semantic segmentation with integrated macro-micro dual-stage feature enhancement and refinement

Semantic segmentation can assist doctors to locate key lesion areas intelligently, rapidly and with high precision in medical imaging, which is of great clinical significance for the diagnosis and rehabilitation of patients in various medical fields such as oncology and neurology. Existing segmentation networks face accuracy bottlenecks because enriched information at the decoding end can’t disperse over multiple layers of semantic depth. This is compounded by challenges like complex anatomical structures and varying imaging modalities. In this study, we propose a Multi-Bottleneck Progressive Semantic Segmentation Network (MBP-SSNet), which is essentially a new Baseline based on our proposal: Multi-Bottleneck based Progressive Propulsion Feature Guidance Network (MB-PPFG Net). Its Multi-Bottleneck solves the problem of coding and decoding multiplexed transmission of enriched information, and the propulsive transmission alleviates the problem of insufficient semantic fusion. In addition, under the macro–micro processing idea, MacroFocus Pre-Bottleneck Feature Enhancer (MPBFE) is proposed at the codec intersection in order to better enhance the image contextual semantic dependency association; while MicroDetail Post- Decoding Feature Refiner (MPDFR) is proposed at the tail of the network for fine-grained polishing. In the experiment, the MBP-SSNet was compared with 17 other models across 7 datasets, including diverse medical imaging scenarios, involving metric exploration, image comparison, process validation, and statistical analysis. Concurrently, 22 types of ablation analyses were conducted on it, along with detailed discussions on two critical issues. The experimental results demonstrate that MBP-SSNet achieves an average performance improvement of 0.6809 % in medical image segmentation compared to the optimal benchmark model. The official web address of MBP-SSNet is at: https://github.com/YF-W/MBP-SSNet.

ABNet: An Aggregated Backbone Network Architecture for Fine Landcover Classification

High-precision landcover classification is a fundamental prerequisite for resource and environmental monitoring and land-use status surveys. Imbued with intricate spatial information and texture features, very high spatial resolution remote sensing images accentuate the divergence between features within the same category, thereby amplifying the complexity of landcover classification. Consequently, semantic segmentation models leveraging deep backbone networks have emerged as stalwarts in landcover classification tasks owing to their adeptness in feature representation. However, the classification efficacy of a solitary backbone network model fluctuates across diverse scenarios and datasets, posing a persistent challenge in the construction or selection of an appropriate backbone network for distinct classification tasks. To elevate the classification performance and bolster the generalization of semantic segmentation models, we propose a novel semantic segmentation network architecture, named the aggregated backbone network (ABNet), for the meticulous landcover classification. ABNet aggregates three prevailing backbone networks (ResNet, HRNet, and VoVNet), distinguished by significant structural disparities, using a same-stage fusion approach. Subsequently, it amalgamates these networks with the Deeplabv3+ head after integrating the convolutional block attention mechanism (CBAM). Notably, this amalgamation harmonizes distinct scale features extracted by the three backbone networks, thus enriching the model’s spatial contextual comprehension and expanding its receptive field, thereby facilitating more effective semantic feature extraction across different stages. The convolutional block attention mechanism primarily orchestrates channel adjustments and curtails redundant information within the aggregated feature layers. Ablation experiments demonstrate an enhancement of no less than 3% in the mean intersection over union (mIoU) of ABNet on both the LoveDA and GID15 datasets when compared with a single backbone network model. Furthermore, in contrast to seven classical or state-of-the-art models (UNet, FPN, PSPNet, DANet, CBNet, CCNet, and UPerNet), ABNet evinces excellent segmentation performance across the aforementioned datasets, underscoring the efficiency and robust generalization capabilities of the proposed approach.

Research on Satellite and Ground Multi-Sensor Collaborative Sensing Method for Floating Plants

Abstract. Dynamic monitoring of water environment is the basis of maintaining urban security and promoting urban sustainable development. To effectively solve the problems caused by the large-scale flooding of floating aquatic invasive plants, such as water environment destruction, clogging river, water quality pollution, water ecological balance destruction, etc., a satellite and ground multi-sensor collaborative sensing method for floating plants is proposed in this paper. For the monitoring of floating plants in macro watershed, the vegetation index and chlorophyll concentration were used to extract floating plants, analyzing, and calculating the distribution and coverage area of floating plants based on 4-band high-resolution satellite images. For the monitoring of floating plants in small watershed, combined with visible light video, multispectral images, and LiDAR data, optimized spectral feature variable processing, semantic segmentation network, and symmetric function neural network are utilized to obtain real-time information on the invasion location, distribution pattern, and coverage area of floating plants. In this paper, the upstream basin of the Huangpu River in Shanghai in 2023 was selected as the experimental area. The distribution of floating plants in different watercourse and different periods was successfully obtained by this method, and the coverage area was accurately calculated with an identification rate of more than 90%, which provided technical support for cross-regional management and efficient cleaning of floating plants.

Towards Sustainable Urban Energy: A Robust Deep Learning Framework for Solar Potential Estimation

Abstract. Rooftop photovoltaic is considered as a cost-effective and environmentally friendly solution to energy challenges in urban areas. To ensure photovoltaic efficiency, it is essential to accurately estimate rooftop solar potential and deploy solar panels wisely. During the past few years, deep learning-based estimation methods have emerged and mainly rely on inferring rooftop orientations from aerial imagery. However, we note that rooftops often appear diversely when images are taken at different solar azimuths, and this can lead to orientation misclassification. To address this, we propose a robust solar potential estimation framework, mainly composed of a rooftop orientation prediction network and a bilateral solar potential estimation module. Specifically, we first classify rooftops into five orientations, i.e., east, west, south, north towards, and flat with a semantic segmentation network. Afterward, opposing orientations are merged to alleviate misclassification caused by variant data acquisition time. Eventually, we compute solar potentials based on PVGIS and a weighting scheme. Experimental results on the RID dataset demonstrate the effectiveness of our approach in improving the accuracy of solar energy estimation.

Side-Scan Sonar Image Matching Method Based on Topology Representation

In the realm of underwater environment detection, achieving information matching stands as a pivotal step, forming an indispensable component for collaborative detection and research in areas such as distributed mapping. Nevertheless, the progress in studying the matching of underwater side-scan sonar images has been hindered by challenges including low image quality, intricate features, and susceptibility to distortion in commonly used side-scan sonar images. This article presents a comprehensive overview of the advancements in underwater sonar image processing. Building upon the novel SchemaNet image topological structure extraction model, we introduce a feature matching model grounded in side-scan sonar images. The proposed approach employs a semantic segmentation network as a teacher model to distill the DeiT model during training, extracting the attention matrix of intermediate layer outputs. This emulates SchemaNet’s transformation method, enabling the acquisition of high-dimensional topological structure features from the image. Subsequently, utilizing a real side-scan sonar dataset and augmenting data, we formulate a matching dataset and train the model using a graph neural network. The resulting model demonstrates effective performance in side-scan sonar image matching tasks. These research findings bear significance for underwater detection and target recognition and can offer valuable insights and references for image processing in diverse domains.

Distinguishing the Uterine Artery, the Ureter, and Nerves in Laparoscopic Surgical Images Using Ensembles of Binary Semantic Segmentation Networks.

Performing a minimally invasive surgery comes with a significant advantage regarding rehabilitating the patient after the operation. But it also causes difficulties, mainly for the surgeon or expert who performs the surgical intervention, since only visual information is available and they cannot use their tactile senses during keyhole surgeries. This is the case with laparoscopic hysterectomy since some organs are also difficult to distinguish based on visual information, making laparoscope-based hysterectomy challenging. In this paper, we propose a solution based on semantic segmentation, which can create pixel-accurate predictions of surgical images and differentiate the uterine arteries, ureters, and nerves. We trained three binary semantic segmentation models based on the U-Net architecture with the EfficientNet-b3 encoder; then, we developed two ensemble techniques that enhanced the segmentation performance. Our pixel-wise ensemble examines the segmentation map of the binary networks on the lowest level of pixels. The other algorithm developed is a region-based ensemble technique that takes this examination to a higher level and makes the ensemble based on every connected component detected by the binary segmentation networks. We also introduced and trained a classic multi-class semantic segmentation model as a reference and compared it to the ensemble-based approaches. We used 586 manually annotated images from 38 surgical videos for this research and published this dataset.

Parallel feature enhancement and adaptive weighted feature fusion for semantic segmentation

To tackle the challenges posed by the insensitivity of current multi-scale networks to image detailed information and their limited capacity to model contextual relationships, our paper proposes a novel semantic segmentation network called LEFNet. It is based on Parallel Feature Enhancement (PFE) and Adaptive Weighted Feature Fusion (AWFF). In the coding stage, the PFE module performs posterior enhancement of multi-scale features by Detail Sharpening Attention (DSA) and High-level Dilation Fusion (HDF) methods. The DSA guides the learning of detailed information in low-level features. The HDF broadens the perceptual field, enabling the acquisition of richer high-level features. In the decoding stage, the AWFF module supersedes the conventional feature fusion methods. The AWFF module constructs perceptual factors for each multi-scale feature map, enabling weighted learning of features. It emphasizes features with stronger semantic information, making them more decisive in pixel classification. It integrates features more reasonably based on the relevance of global contextual information, fully releasing the expressive potential of encoded features. Our method achieves mIoU scores of 82.8%, 49.3%, and 45.4% on the Cityscapes, ADE 20 K, and COCO-Stuff 164 K datasets, respectively, reaching an advanced level on popular benchmarks. The experimental results show that LEFNet alleviates the challenge of insensitivity of multi-scale networks to image detailed information, improves its ability to model contextual relationships, and significantly improves segmentation performance.

Residual spatial fusion network for RGB-thermal semantic segmentation

Semantic segmentation plays an important role in widespread applications such as autonomous driving and robotic sensing. This work focuses on RGB-Thermal (RGB-T) semantic segmentation, since RGB images are easily affected by lighting conditions, e.g., darkness, and thermal images are robust to the night scenario as a compensating modality. However, existing works either simply fuse RGB-T images or adopt the encoder with the same backbone for the two modalities, neglecting the semantic difference under varying lighting conditions. Therefore, we present a Residual Spatial Fusion Network (RSFNet) for RGB-T semantic segmentation. Specifically, RSFNet employs an asymmetric encoder to learn the compensating features of RGB and thermal images. To fuse the dual-modality features, it generates pseudo-labels by adopting the saliency detection skill to supervise feature learning, and develops the Residual Spatial Fusion (RSF) module with structural re-parameterization to learn more promising features by spatially fusing the cross-modality features. RSF adopts a hierarchical feature fusion to aggregate multi-level features, and applies the spatial weights with the residual connection to adaptively control the multi-spectral feature fusion by the confidence gate. Extensive experiments were carried out on two benchmarks, i.e., MFNet database and PST900 database. The results have shown the state-of-the-art segmentation performance of our method, which strikes a good balance between accuracy and speed.

TCFAP-Net: Transformer-based Cross-feature Fusion and Adaptive Perception Network for large-scale point cloud semantic segmentation

Point cloud semantic segmentation is an ingredient in understanding real-world scenes. Most existing approaches perform poorly on scene boundaries and struggle with recognizing objects of different scales. In this paper, we propose a novel framework that incorporates Transformer into the U-Net architecture for inferring pointwise semantics. Specifically, the Transformer-based cross-feature fusion module is designed first to employ geometric and semantic information to learn feature offsets to overcome the border ambiguity of segmentation results, and then it utilizes the Transformer to learn cross-feature enhanced and fused encoder features. Additionally, to facilitate the overall network’s structure-to-detail perception capabilities, the adaptive perception module is designed, which employs cross-attention to adaptively allocate weights to encoder features at varying resolutions, establishing long-range contextual dependencies. Ablation studies validate the individual contributions of our module design choices. Compared with the existing competitive methods, our approach achieves state-of-the-art performance and exhibits superior results on benchmarks. Code is available at https://github.com/xiluo-cug/TCFAP-Net.

Convolutional neural networks for urban green areas semantic segmentation on Sentinel-2 data

Urban green areas are essential components of any urban environment, providing a wide range of uses. However, there is currently a noticeable absence of an automated tool for their land use classification. The use of urban green areas depends on their size, shape, and relationship with their surroundings, all of which are fundamental features in convolutional neural networks. Various convolutional neural network architectures (FCN, U-Net, SegNet, DeepLabv3+) were evaluated on open and widely accessible Sentinel-2 data for semantic segmentation of land cover and land use in different levels of urban green areas nomenclature and band combinations. Moreover, we compared the CNNs with random forests model as a baseline to underline the CNNs’ strengths. The evaluation found that convolutional neural networks are capable of the land cover and land use semantic segmentation not only on the full-band Sentinel-2 scenes but also on limited subdatasets consisting only of RGB bands. U-Net is identified as the best-performing architecture, achieving an overall accuracy of almost 95% for a simple binary vegetation detection, 90% for the land-use task, and almost 88% for the land-use task enhanced by a distinction between high and low vegetation, while random forests reached 93%, 84%, and 81%, respectively. CNNs’ misclassifications were primarily identified at the edges of two neighboring competing classes where mixed pixels appear. Data augmentation improved the model’s performance in 94% of cases. However, dropout layers led to an overall accuracy decrease in more than half of the cases. Additionally, we compared the segmented urban green area with a pan-European dataset, the Coordination of Information on the Environment Land Cover - and found that the latter omits 74% of the total urban vegetation. This is mainly due to its minimal mapping unit specification. It is concluded that the most suitable approach for automated urban green areas land cover and land use semantic segmentation is the use of convolutional neural networks, from the tested architectures particularly U-Net.

GLIMS: Attention-guided lightweight multi-scale hybrid network for volumetric semantic segmentation

Convolutional Neural Networks (CNNs) have become widely adopted for medical image segmentation tasks, demonstrating promising performance. However, the inherent inductive biases in convolutional architectures limit their ability to model long-range dependencies and spatial correlations. While recent transformer-based architectures address these limitations by leveraging self-attention mechanisms to encode long-range dependencies and learn expressive representations, they often struggle to extract low-level features and are highly dependent on data availability. This motivated us for the development of GLIMS, a data-efficient attention-guided hybrid volumetric segmentation network. GLIMS utilizes Dilated Feature Aggregator Convolutional Blocks (DACB) to capture local–global feature correlations efficiently. Furthermore, the incorporated Swin Transformer-based bottleneck bridges the local and global features to improve the robustness of the model. Additionally, GLIMS employs an attention-guided segmentation approach through Channel and Spatial-Wise Attention Blocks (CSAB) to localize expressive features for fine-grained border segmentation. Quantitative and qualitative results on glioblastoma and multi-organ CT segmentation tasks demonstrate GLIMS’ effectiveness in terms of complexity and accuracy. GLIMS demonstrated outstanding performance on BraTS2021 and BTCV datasets, surpassing the performance of Swin UNETR. Notably, GLIMS achieved this high performance with a significantly reduced number of trainable parameters. Specifically, GLIMS has 47.16 M trainable parameters and 72.30G FLOPs, while Swin UNETR has 61.98 M trainable parameters and 394.84G FLOPs. The code is publicly available at https://github.com/yaziciz/GLIMS.

Efficient multi-task progressive learning for semantic segmentation and disparity estimation

Scene understanding is an important area in robotics and autonomous driving. To accomplish these tasks, the 3D structures in the scene have to be inferred to know what the objects and their locations are. To this end, semantic segmentation and disparity estimation networks are typically used, but running them individually is inefficient since they require high-performance resources. A possible solution is to learn both tasks together using a multi-task approach. Some current methods address this problem by learning semantic segmentation and monocular depth together. However, monocular depth estimation from single images is an ill-posed problem. A better solution is to estimate the disparity between two stereo images and take advantage of this additional information to improve the segmentation. This work proposes an efficient multi-task method that jointly learns disparity and semantic segmentation. Employing a Siamese backbone architecture for multi-scale feature extraction, the method integrates specialized branches for disparity estimation and coarse and refined segmentations, leveraging progressive task-specific feature sharing and attention mechanisms to enhance accuracy for solving both tasks concurrently. The proposal achieves state-of-the-art results for joint segmentation and disparity estimation on three distinct datasets: Cityscapes, TrimBot2020 Garden, and S-ROSeS, using only 1/3 of the parameters of previous approaches.

MSFA-Net: A Multiscale Feature Aggregation Network for Semantic Segmentation of Historical Building Point Clouds

In recent years, research on the preservation of historical architecture has gained significant attention, where the effectiveness of semantic segmentation is particularly crucial for subsequent repair, protection, and 3D reconstruction. Given the sparse and uneven nature of large-scale historical building point cloud scenes, most semantic segmentation methods opt to sample representative subsets of points, often leading to the loss of key features and insufficient segmentation accuracy of architectural components. Moreover, the geometric feature information at the junctions of components is cluttered and dense, resulting in poor edge segmentation. Based on this, this paper proposes a unique semantic segmentation network design called MSFA-Net. To obtain multiscale features and suppress irrelevant information, a double attention aggregation module is first introduced. Then, to enhance the model’s robustness and generalization capabilities, a contextual feature enhancement and edge interactive classifier module are proposed to train edge features and fuse the context data. Finally, to evaluate the performance of the proposed model, experiments were conducted on a self-curated ancient building dataset and the S3DIS dataset, achieving OA values of 95.2% and 88.7%, as well as mIoU values of 86.2% and 71.6%, respectively, further confirming the effectiveness and superiority of the proposed method.