Semantic Segmentation Research Articles

Research on Satellite and Ground Multi-Sensor Collaborative Sensing Method for Floating Plants

Abstract. Dynamic monitoring of water environment is the basis of maintaining urban security and promoting urban sustainable development. To effectively solve the problems caused by the large-scale flooding of floating aquatic invasive plants, such as water environment destruction, clogging river, water quality pollution, water ecological balance destruction, etc., a satellite and ground multi-sensor collaborative sensing method for floating plants is proposed in this paper. For the monitoring of floating plants in macro watershed, the vegetation index and chlorophyll concentration were used to extract floating plants, analyzing, and calculating the distribution and coverage area of floating plants based on 4-band high-resolution satellite images. For the monitoring of floating plants in small watershed, combined with visible light video, multispectral images, and LiDAR data, optimized spectral feature variable processing, semantic segmentation network, and symmetric function neural network are utilized to obtain real-time information on the invasion location, distribution pattern, and coverage area of floating plants. In this paper, the upstream basin of the Huangpu River in Shanghai in 2023 was selected as the experimental area. The distribution of floating plants in different watercourse and different periods was successfully obtained by this method, and the coverage area was accurately calculated with an identification rate of more than 90%, which provided technical support for cross-regional management and efficient cleaning of floating plants.

TEDNet: Twin Encoder Decoder Neural Network for 2D Camera and LiDAR Road Detection

Abstract Robust road surface estimation is required for autonomous ground vehicles to navigate safely. Despite it becoming one of the main targets for autonomous mobility researchers in recent years, it is still an open problem in which cameras and LiDAR sensors have demonstrated to be adequate to predict the position, size and shape of the road a vehicle is driving on in different environments. In this work, a novel Convolutional Neural Network model is proposed for the accurate estimation of the roadway surface. Furthermore, an ablation study has been conducted to investigate how different encoding strategies affect model performance, testing 6 slightly different neural network architectures. Our model is based on the use of a Twin Encoder–Decoder Neural Network (TEDNet) for independent camera and LiDAR feature extraction and has been trained and evaluated on the Kitti–Road dataset. Bird’s Eye View projections of the camera and LiDAR data are used in this model to perform semantic segmentation on whether each pixel belongs to the road surface. The proposed method performs among other state-of-the-art methods and operates at the same frame rate as the LiDAR and cameras, so it is adequate for its use in real-time applications.

Cascaded framework for earthquake building damage detection combining spatial and frequency domain feature integration

Abstract. Building collapse is a major cause of casualties after an earthquake, so accurately extracting building damage information is critical for post-earthquake assessment and rescue. Currently, most deep learning methods focus on the end-to-end detection of building collapse. However, in real-world earthquake scenarios, the end-to-end computational process often lacks flexibility and struggles to meet the requirements of rapid emergency response. To address this issue, this paper proposes a cascaded framework that combines pre-earthquake building extraction and post-earthquake building damage classification. The proposed framework includes two sections: (1) Progressive building semantic segmentation model in the joint frequency domain. This model is designed to accurately extract buildings prior to an earthquake, with the goal of minimizing error propagation throughout the cascading process. The model addresses the spatial similarity of buildings under complicated backgrounds, as well as the high internal heterogeneity of buildings, by utilizing frequency domain techniques. It compensates for the shortcomings of traditional models in terms of incomplete information extraction through the effective integration of global and local information. Finally, the model employs edge priors for edge regularization. (2) Rapid building damage classification process. Based on the accurate building extraction results, a fast and efficient classification process is developed. This process uses a simple and lightweight classification network to effectively extract building damage information caused by the earthquake. The superiority of the proposed framework is validated through comparison with traditional cascading architectures and end-to-end models. The results show that the cascading framework not only provides accurate pre-earthquake building extraction, but also enables efficient and accurate post-earthquake damage classification, which meets the requirements of rapid post-earthquake emergency response. This balance of accuracy and speed is essential for effective disaster management and recovery.

Towards Sustainable Urban Energy: A Robust Deep Learning Framework for Solar Potential Estimation

Abstract. Rooftop photovoltaic is considered as a cost-effective and environmentally friendly solution to energy challenges in urban areas. To ensure photovoltaic efficiency, it is essential to accurately estimate rooftop solar potential and deploy solar panels wisely. During the past few years, deep learning-based estimation methods have emerged and mainly rely on inferring rooftop orientations from aerial imagery. However, we note that rooftops often appear diversely when images are taken at different solar azimuths, and this can lead to orientation misclassification. To address this, we propose a robust solar potential estimation framework, mainly composed of a rooftop orientation prediction network and a bilateral solar potential estimation module. Specifically, we first classify rooftops into five orientations, i.e., east, west, south, north towards, and flat with a semantic segmentation network. Afterward, opposing orientations are merged to alleviate misclassification caused by variant data acquisition time. Eventually, we compute solar potentials based on PVGIS and a weighting scheme. Experimental results on the RID dataset demonstrate the effectiveness of our approach in improving the accuracy of solar energy estimation.

Leveraging Deep Learning for Automated Reconstruction of Indoor Unstructured Elements in Scan-to-BIM

Abstract. Achieving automatic 3D reconstruction for indoor scenes is extremely useful in the field of scene understanding. Building information modeling (BIM) models are essential for lowering project costs, assisting in building planning and renovations, as well as improving building management efficiency. However, nearly all current available scan-to-BIM approaches employ manual or semi-automatic methods. These approaches concentrate solely on significant structured objects, neglecting other unstructured elements such as furniture. The limitation arises from challenges in modeling incomplete point clouds of obstructed objects and capturing indoor scene details. Therefore, this research introduces an innovative and effective reconstruction framework based on deep learning semantic segmentation and model-driven techniques to address these limitations. The proposed framework utilizes wall segment recognition, feature extraction, opening detection, and automatic modeling to reconstruct 3D structured models of point clouds with different room layouts in both Manhattan and non-Manhattan architectures. Moreover, it provides 3D BIM models of actual unstructured elements by detecting objects, completing point clouds, establishing bounding boxes, determining type and orientation, and automatically generating 3D BIM models with a parametric algorithm implemented into the Revit software. We evaluated this framework using publicly available and locally generated point cloud datasets with varying furniture combinations and layout complexity. The results demonstrate the proposed framework's efficiency in reconstructing structured indoor elements, exhibiting completeness and geometric accuracy, and achieving precision and recall values greater than 98%. Furthermore, the generated unstructured 3D BIM models keep essential real-scene characteristics such as geometry, spatial locations, numerical aspects, various shapes, and orientations compared to literature methods.

Unsupervised domain adaptation for the semantic segmentation of remote sensing images via a class-aware Fourier transform and a fine-grained discriminator

The semantic segmentation of remote sensing images is vital for Earth observation purposes. However, its performance can decline significantly due to differences in dataset distributions between training (source) and deployment (target) settings. Unsupervised domain adaptation can be used to counter this problem by leveraging the knowledge acquired from the labelled source domain and by adapting it to the unlabelled target domain. Existing methods focus on either input-level or feature-level alignments, which can be sub-optimal for addressing large domain gaps. To this end, this paper introduces a new unsupervised domain adaptation method that employs concurrently two levels of alignment: first, at the input level, an adaptive Fourier-based image-to-image translation approach is utilised to generate target-styled source images with class-based low-amplitude changes. Then, at the feature level, an adaptive fine-grained domain discriminator is introduced that incorporates class information into two parallel discriminators, for source vs. target and target-styled source image vs. target settings. Experimental results indicate that the proposed method improves significantly cross-domain semantic segmentation performance with respect to the state-of-the-art.

Full-scale semantic segmentation of hyperspectral imaging based on spatial spatial-spectral joint network

Abstract. Hyperspectral images contain dozens or even hundreds of spectral bands, which contain rich spectral information and help distinguish different ground objects. Hyperspectral images have a wide range of applications in urban planning, environmental monitoring, and other fields. The semantic segmentation of hyperspectral images is one of the current research hotspots. The difficulty lies in the rich spectral information and strong correlation of hyperspectral images. Traditional semantic segmentation methods cannot fully extract information, which affects the accuracy of classification. This article utilizes an encoding decoding structure to simultaneously extract deep and shallow features of images. A REGCS convolution module was constructed using the idea of group convolution to extract spectral and spatial features of images. We compared the Salinas Valley dataset and MUUFL dataset with various classification algorithms. The experimental results show that compared with other classification models, the RESSU model has achieved stable and excellent results in hyperspectral image classification experiments. Among them, in the classification experiment of the Salinas Valley dataset, the accuracy of single class classification reached over 92%. In the effectiveness analysis experiment, we calculated different model parameter quantities to verify the performance of our method, and ultimately achieved good results.

Exploring the Impact of Campus Landscape Visual Elements Combination on Short-Term Stress Relief among College Students: A Case from China

Although the effect of campus landscape space on stress relief among college students has been confirmed, few existing studies have considered the impact on stress recovery from the perspective of factor combination, and the key visual elements and the most effective combination of visual elements to relieve stress are still unclear. This study attempts to conduct a natural experiment within Chinese campuses, measuring physiological indicators of stress such as heart rate (HR), frequency domain index of heart rate variability (LF/HF), skin conductance level (SCL), skin temperature (SKT), and respiratory rate (RESP) using physiological instruments. It explored the effects of visual elements and their combinations in campus landscape spaces on short-term stress relief among college students through semantic segmentation, multifactorial analysis of variance, and post hoc multiple comparison methods. Research results demonstrate that the presence of water elements in the field of vision can effectively improve the stress relief effects of landscape spaces. Reasonable combinations of natural landscape elements and artificial landscape elements in the design can also effectively promote stress relief among students. Building facade area and sky area, water area and sky area, and plant species and pavement area are three combinations of factors with the strongest interactive effects. “Natural water scenery” and “exquisite artificial” are two campus landscape design patterns most conducive to short-term stress relief.

Semantic scene segmentation for indoor autonomous vision systems: leveraging an enhanced and efficient U-NET architecture

Semantic scene segmentation for indoor autonomous vision systems: leveraging an enhanced and efficient U-NET architecture

CLGFormer: Cross-Level-Guided transformer for RGB-D semantic segmentation

CLGFormer: Cross-Level-Guided transformer for RGB-D semantic segmentation

Assessment of changes in vessel area during needle manipulation in microvascular anastomosis using a deep learning-based semantic segmentation algorithm: A pilot study.

Appropriate needle manipulation to avoid abrupt deformation of fragile vessels is a critical determinant of the success of microvascular anastomosis. However, no study has yet evaluated the area changes in surgical objects using surgical videos. The present study therefore aimed to develop a deep learning-based semantic segmentation algorithm to assess the area change of vessels during microvascular anastomosis for objective surgical skill assessment with regard to the "respect for tissue." The semantic segmentation algorithm was trained based on a ResNet-50 network using microvascular end-to-side anastomosis training videos with artificial blood vessels. Using the created model, video parameters during a single stitch completion task, including the coefficient of variation of vessel area (CV-VA), relative change in vessel area per unit time (ΔVA), and the number of tissue deformation errors (TDE), as defined by a ΔVA threshold, were compared between expert and novice surgeons. A high validation accuracy (99.1%) and Intersection over Union (0.93) were obtained for the auto-segmentation model. During the single-stitch task, the expert surgeons displayed lower values of CV-VA (p < 0.05) and ΔVA (p < 0.05). Additionally, experts committed significantly fewer TDEs than novices (p < 0.05), and completed the task in a shorter time (p < 0.01). Receiver operating curve analyses indicated relatively strong discriminative capabilities for each video parameter and task completion time, while the combined use of the task completion time and video parameters demonstrated complete discriminative power between experts and novices. In conclusion, the assessment of changes in the vessel area during microvascular anastomosis using a deep learning-based semantic segmentation algorithm is presented as a novel concept for evaluating microsurgical performance. This will be useful in future computer-aided devices to enhance surgical education and patient safety.

360-Degree Tri-Modal Scanning: Engineering a Modular Multi-Sensor Platform for Semantic Enrichment of BIM Models

Abstract. Point clouds, image data, and corresponding processing algorithms are intensively investigated to create and enrich Building Information Models (BIM) with as-is information and maintain their value across the building lifecycle. Point clouds can be captured using LiDAR and enriched with color information from images. Complementary to such dual-sensor systems, thermography captures the infrared light spectrum, giving insight into the temperature distribution on an object’s surface and allowing a diagnosis of the as-is energetic health of buildings beyond what humans can see. Although the three sensor modes are commonly used in pair-wise combinations, only a few systems leveraging the power of tri-modal sensor fusion have been proposed. This paper introduces a sensor system comprising LiDAR, RGB, and a radiometric thermal infrared sensor that can capture a 360-degree range through bi-axial rotation. The resulting tri-modal data is fused to a thermo-color point cloud from which temperature values are derived for a standard indoor building setting. Qualitative data analysis shows the potential for unlocking further object semantics in a state-of-the-art Scan-to-BIM pipeline. Furthermore, an outlook is provided on the cross-modal usage of semantic segmentation for automatic, accurate temperature calculations.

Vison Transformer-Based Automatic Crack Detection on Dam Surface

Dam is an essential structure in hydraulic engineering, and its surface cracks pose significant threats to its integrity, impermeability, and durability. Automated crack detection methods based on computer vision offer substantial advantages over manual approaches with regard to efficiency, objectivity and precision. However, current methods face challenges such as misidentification, discontinuity, and loss of details when analyzing real-world dam crack images. These images often exhibit characteristics such as low contrast, complex backgrounds, and diverse crack morphologies. To address the above challenges, this paper presents a pure Vision Transformer (ViT)-based dam crack segmentation network (DCST-net). The DCST-net utilizes an improved Swin Transformer (SwinT) block as the fundamental block for enhancing the long-range dependencies within a SegNet-like encoder–decoder structure. Additionally, we employ a weighted attention block to facilitate side fusion between the symmetric pair of encoder and decoder in each stage to sharpen the edge of crack. To demonstrate the superior performance of our proposed method, six semantic segmentation models have been trained and tested on both a self-built dam crack dataset and two publicly available datasets. Comparison results indicate that our proposed model outperforms the mainstream methods in terms of visualization and most evaluation metrics, highlighting its potential for practical application in dam safety inspection and maintenance.

Multiresolution semantic segmentation of biological structures in digital histopathology.

Semantic segmentation in high-resolution, histopathology whole slide images (WSIs) is an important fundamental task in various pathology applications. Convolutional neural networks (CNN) are the state-of-the-art approach for image segmentation. A patch-based CNN approach is often employed because of the large size of WSIs; however, segmentation performance is sensitive to the field-of-view and resolution of the input patches, and balancing the trade-offs is challenging when there are drastic size variations in the segmented structures. We propose a multiresolution semantic segmentation approach, which is capable of addressing the threefold trade-off between field-of-view, computational efficiency, and spatial resolution in histopathology WSIs. We propose a two-stage multiresolution approach for semantic segmentation of histopathology WSIs of mouse lung tissue and human placenta. In the first stage, we use four different CNNs to extract the contextual information from input patches at four different resolutions. In the second stage, we use another CNN to aggregate the extracted information in the first stage and generate the final segmentation masks. The proposed method reported 95.6%, 92.5%, and 97.1% in our single-class placenta dataset and 97.1%, 87.3%, and 83.3% in our multiclass lung dataset for pixel-wise accuracy, mean Dice similarity coefficient, and mean positive predictive value, respectively. The proposed multiresolution approach demonstrated high accuracy and consistency in the semantic segmentation of biological structures of different sizes in our single-class placenta and multiclass lung histopathology WSI datasets. Our study can potentially be used in automated analysis of biological structures, facilitating the clinical research in histopathology applications.

Smartphone-based straw incorporation: An improved convolutional neural network

In conservation tillage, the effective incorporation of straw into soil is a critical factor for optimizing decomposition rates and addressing challenges related to straw accumulation. Consequently, this study introduces an innovative straw semantic segmentation model to extract straw mixed with soil from images, and proposes a method to measure the quality of straw and soil mixing based on this model. We conducted field experiments based on this method, analyzed the quality of straw incorporation at different depths, and finally integrated the model into a mobile application to facilitate the automatic, accurate, and rapid assessment of the quality of straw and soil mixing. The methodology of this study as an improved convolutional neural network entailed the fusion of deep separable convolution with a multi-scale feature extraction module, utilizing convolution kernels of varying sizes to capture the diverse features of straw. Further refinement included halving the first convolution channel and implementing a direct connection structure. Attention gates were strategically deployed to enhance salient features, resulting in superior accuracy compared to the original U-shaped network, fully convolutional network, and Otsu algorithm. Notably, the proposed model achieved enhanced performance while reducing the number of parameters and model complexity to one-third and one-half of the original U-shaped network, respectively. For a quantitative description of the rate and uniformity of straw incorporation, this study employed image processing technology and a grid counting method. The synergistic integration of smartphone imagery and deep learning expedited the delivery of rapid and reliable results, promising practical applicability in future straw incorporation practices. A demonstration video of the application is accessible at: https://doi.org/10.6084/m9.figshare.14480745.v3. The findings of this research contribute significantly to the advancement of conservation tillage by establishing a robust framework for the assessment and enhancement of straw incorporation operations’ efficiency.

Deep learning-based semantic segmentation for morphological fractography

Fractographic analysis poses a significant challenge for field researchers without specialized training in fractography. To address this issue, this study introduces a comprehensive integrated workflow that encapsulates the entire process from dataset preparation and data enhancement to leveraging the SegFormer model for deep learning-driven semantic segmentation. An extensive collection of fractography images is formulated and augmented to train the SegFormer model, enabling precise semantic segmentation of morphological fracture regions including cleavage, ductile, dimple, fatigue striations, and others. To accommodate the demanding SEM imaging conditions which frequently include distortions, noise, and aberrations, we developed a two-stage method with diverse data augmentation strategies. This method resulted in a robust model demonstrating exceptional performance, as evidenced by a high mean Intersection over Union (mIOU) score of 59.7 and other metrics. The findings validate the potential of deep learning techniques, particularly the SegFormer model's efficacy in morphological fractography image segmentation for the first time. Our work offers a cost-effective, and efficient alternative deep learning approach to traditional experimental fracture analysis, thereby expanding opportunities for a broader range of professionals in the engineering domain.

MRASFusion: A multi-scale residual attention infrared and visible image fusion network based on semantic segmentation guidance

To address the challenges of inadequate preservation of prominent targets, poor retention of texture details, and unsatisfactory reconstruction of image backgrounds in image fusion. In this paper, a multi-scale residual attention fusion network based on semantic segmentation guidance is proposed, termed as MRASFusion. First of all, Swin Transformer segmentation mask with high precision, and strong scalability is adopted to avoid the inefficiency and error of manual segmentation mask. The mask generated by semantic segmentation is used to construct a loss function to guide the image fusion process. Secondly, in order to maintain the integrity of contextual information and texture details, a new feature extraction module is proposed to fully extract the meaningful features. Finally, the fused image is obtained by reconstructing the extracted features. To verify the effectiveness of the method, MRASFusion is qualitatively and quantitatively compared with nine state-of-the-art fusion methods on TNO and RoadScene datasets. Experimental results indicate that our method has demonstrated satisfactory performance in image fusion tasks, exhibiting superior capabilities in preserving target information and retaining texture details. Furthermore, our fusion results have brought some performance improvements for advanced vision tasks, i.e., improved accuracy for the object detection, which provides a better foundation for solving real-world problems.

Spectral–Spatial transformer-based semantic segmentation for large-scale mapping of individual date palm trees using very high-resolution satellite data

Date palm plantations in the United Arab Emirates (UAE) are under threat from soil salinity, drought, and date palm weevils. Accordingly, monitoring and conserving date palms are crucial to preserving a vital component of the country’s agricultural heritage, economy, food security, and ecological balance. Previous studies have effectively identified date palm trees using RGB-based aerial and UAV imagery utilizing diverse deep learning methods. However, the utilization of very high-resolution satellite data for delineating individual date palm crowns remains unexplored due to the limited spatial resolution capabilities of existing satellite systems. This study primarily aimed to achieve precise and comprehensive mapping of date palm trees using WorldView-3 (WV-3) satellite data by leveraging the high representational power of the state-of-the-art vision transformers (ViT) in capturing global information from the input data. First, an in-depth analysis assessment of the various transformer-based semantic segmentation architectures, including UperNet with vision transformer and Swin transformer, SegFormer, Mask2Former, and UniFormer, was conducted. Second, the integration of spectral data on the performance of ViTs was evaluated. Moreover, the models’ generalizability and complexity effect on the segmentation effectiveness were assessed. Accordingly, a postprocessing strategy was developed to aid in delineating and counting date palm trees from semantic segmentation outputs. Results demonstrated that integration of WV-3 spectral data into the analysis resulted in a marked improvement in segmentation quality. The UniFormer, UperNet-Swin, and Mask2Former models demonstrated considerable improvements in multispectral data analysis, with increases in mean intersection over union (mIoU) of 2.17% (77.88% mIoU, 86.01% mean F-score [mF-score]), 2% (78.10% mIoU, 86.18% mF-score), and 1.15% (77.36% mIoU, 85.59% mF-score), respectively, compared with their RGB-based results. Evaluations of model transferability also indicated that Mask2Former, UniFormer, and UperNet-Swin transformers efficiently adapted to multispectral data in the Dibba region. These models achieved mIoU scores of 84.36%, 84.25%, and 83.17% and mF-scores of 90.95%, 90.87%, and 90.13%, highlighting their effectiveness and potential for broader regional application. This research highlights the efficacy and feasibility of using ViTs with WV-3 multispectral data for accurate and comprehensive surveying of date palm plantations, enabling the development of palm tree inventories and continuously updating geospatial databases.

A novel MO-YOLOv4 for segmentation of multi-class bridge damages

Most studies of multi-class damage segmentation overlook two prominent and critical challenges: the substantial variations in damage scale and the susceptibility of network training to background interferences. As a consequence, achieving the desired performance in multi-class damage segmentation becomes a difficult task. To tackle these challenges, a novel multi-class damage segmentation method, MO-YOLOv4, is proposed based on the widely used YOLOv4 structure. MO-YOLOv4 incorporates the Multi-Scale Attention (MSA) module to greatly enhance the feature extraction ability for multi-class damages, and employs the Oriented Bounding Box (OBB)-based segmentation strategy for effectively reducing the influence of background interferences during the network training phase. The effectiveness of the MSA module and OBB-based segmentation strategy is validated through verification and ablation experiments. In the comparative study, the Mean Precision rate (MP), Mean Recall rate (MR), Mean F1-score (MF) and Mean Intersection over Union rate (MIoU) of the proposed MO-YOLOv4 reach up to 82.49%, 72.73%, 77.04% and 64.28%, respectively, indicating the superior level of segmentation performance offered by MO-YOLOv4 in comparison to the mainstream semantic segmentation methods for multi-class damages.

A New Dataset and Comparative Study for Aphid Cluster Detection and Segmentation in Sorghum Fields.

Aphid infestations are one of the primary causes of extensive damage to wheat and sorghum fields and are one of the most common vectors for plant viruses, resulting in significant agricultural yield losses. To address this problem, farmers often employ the inefficient use of harmful chemical pesticides that have negative health and environmental impacts. As a result, a large amount of pesticide is wasted on areas without significant pest infestation. This brings to attention the urgent need for an intelligent autonomous system that can locate and spray sufficiently large infestations selectively within the complex crop canopies. We have developed a large multi-scale dataset for aphid cluster detection and segmentation, collected from actual sorghum fields and meticulously annotated to include clusters of aphids. Our dataset comprises a total of 54,742 image patches, showcasing a variety of viewpoints, diverse lighting conditions, and multiple scales, highlighting its effectiveness for real-world applications. In this study, we trained and evaluated four real-time semantic segmentation models and three object detection models specifically for aphid cluster segmentation and detection. Considering the balance between accuracy and efficiency, Fast-SCNN delivered the most effective segmentation results, achieving 80.46% mean precision, 81.21% mean recall, and 91.66 frames per second (FPS). For object detection, RT-DETR exhibited the best overall performance with a 61.63% mean average precision (mAP), 92.6% mean recall, and 72.55 on an NVIDIA V100 GPU. Our experiments further indicate that aphid cluster segmentation is more suitable for assessing aphid infestations than using detection models.