Semantic Segmentation Algorithms Research Articles

Art Pattern Design Based on Visual Semantic Segmentation and Evolutionary Algorithm Optimization

This paper aims to address the issue of traditional art patterns not fully leveraging contextual information in image semantic segmentation. It proposes a multi-level feature aggregation method for image semantic segmentation based on visual Transformer and coordinate attention adjustment (CA Adjustment). Initially, the input image is segmented into slices for linear projection, incorporating learnable positional embeddings to derive an encoded input sequence. Subsequently, employing a visual Transformer-based encoder, the image is transformed into patches to capture global context across the network. CA Adjustment is then introduced to adaptively merge fusion features and visual backbone features using weighted adjustments. Lastly, a multi-objective grasshopper optimization algorithm is designed based on a clustering evolution mechanism to further enhance algorithm performance. Extensive experimental results demonstrate that our proposed algorithm effectively models global contextual information for image feature extraction. Compared to existing advanced algorithms, our method achieves high segmentation accuracy in semantic segmentation tasks, consistently exceeding 95%. Ablation experiments further validate the efficacy of our approach, underscoring its broad applicability in high-precision image semantic segmentation.

Evaluation of Tree Object Segmentation Performance for Individual Tree Recognition Using Remote Sensing Techniques Based on Urban Forest Green Structures

This study evaluated whether tree object segmentation using remote sensing techniques could be effectively conducted according to the green structures of urban forests. The remote sensing techniques used were handheld LiDAR and UAV-based photogrammetry. The data collected from both methods were merged to complement each other’s limitations. The green structures of the study area were classified into three types based on the distance between canopy trees and the presence of shrubs. The ability to individually classify trees within each of the three types of green structures was then evaluated. The evaluation method was to assess the success rate by comparing the actual number of trees, which were visually counted in the field, with the number of tree objects classified in the study. To perform semantic segmentation of tree objects, a preprocessing step was conducted to extract only the data related to tree structures from the data collected through remote sensing techniques. The preprocessing steps included data merging, noise removal, separation of DTM and DSM, and separation of green areas and structures. The analysis results showed that tree object recognition was not efficient when the green structures were complex and mixed, and the recognition rate was highest when only canopy trees were present, and the canopies did not overlap. Therefore, when observing in high-density areas, the semantic segmentation algorithm’s variables should be adjusted to narrow the object recognition range, and additional observations in winter are needed to compensate for areas obscured by leaves. By improving data collection methods and systematizing the analysis methods according to the green structures, the object recognition process can be enhanced.

L-DeeplabV3+: a lightweight semantic segmentation algorithm for complex scene perception

L-DeeplabV3+: a lightweight semantic segmentation algorithm for complex scene perception

Automating hock wound detection in dairy cattle

Hock scoring in dairy cattle is a crucial welfare assessment tool used to evaluate the condition of a cow's hocks, particularly for signs of injury, swelling, or lesions. These scores provide insight into the overall well-being of the animals and are essential for ensuring proper management and housing conditions. Accurate hock scoring is vital because it can indicate issues such as poor bedding quality or inadequate space, which directly affect the health and productivity of the herd. Traditionally, hock scoring is performed manually by trained observers. However, consistency in scoring can be a challenge. Two studies were conducted to quantify inconsistency in hock scoring. In one study, manual scoring repeatability was measured. In the second study, manual and video scoring repeatability was measured. Repeatability was quantified with a weighted Cowen's kappa metric. Manual scoring was found to be inconsistent but more consistent than video scoring. This variability highlights the need for a more reliable, objective method of scoring. To address this, we explored the automation of hock score detection using artificial intelligence. Specifically, we employed a simple U-net semantic segmentation algorithm to detect wounds on the hocks without classifying them into specific categories. Automating the detection process can reduce observer bias, improve consistency, and allow for continuous monitoring of large herds. This approach holds promise for enhancing animal welfare by providing a more efficient and accurate method of assessing hock health in dairy cattle.

Barrier-free tomato fruit selection and location based on optimized semantic segmentation and obstacle perception algorithm.

With the advancement of computer vision technology, vision-based target perception has emerged as a predominant approach for harvesting robots to identify and locate fruits. However, little attention has been paid to the fact that fruits may be obscured by stems or other objects. In order to improve the vision detection ability of fruit harvesting robot, a fruit target selection and location approach considering obstacle perception was proposed. To enrich the dataset for tomato harvesting, synthetic data were generated by rendering a 3D simulated model of the tomato greenhouse environment, and automatically producing corresponding pixel-level semantic segmentation labels. An attention-based spatial-relationship feature extraction module (SFM) with lower computational complexity was designed to enhance the ability of semantic segmentation network DeepLab v3+ in accurately segmenting linear-structured obstructions such as stems and wires. An adaptive K-means clustering method was developed to distinguish individual instances of fruits. Furthermore, a barrier-free fruit selection algorithm that integrates information of obstacles and fruit instances was proposed to identify the closest and largest non-occluded fruit as the optimal picking target. The improved semantic segmentation network exhibited enhanced performance, achieving an accuracy of 96.75%. Notably, the Intersection-over-Union (IoU) of wire and stem classes was improved by 5.0% and 2.3%, respectively. Our target selection method demonstrated accurate identification of obstacle types (96.15%) and effectively excluding fruits obstructed by strongly resistant objects (86.67%). Compared to the fruit detection method without visual obstacle avoidance (Yolo v5), our approach exhibited an 18.9% increase in selection precision and a 1.3% reduction in location error. The improved semantic segmentation algorithm significantly increased the segmentation accuracy of linear-structured obstacles, and the obstacle perception algorithm effectively avoided occluded fruits. The proposed method demonstrated an appreciable ability in precisely selecting and locating barrier-free fruits within non-structural environments, especially avoiding fruits obscured by stems or wires. This approach provides a more reliable and practical solution for fruit selection and localization for harvesting robots, while also being applicable to other fruits and vegetables such as sweet peppers and kiwis.

Digital rock modeling of deformed multi-scale media in deep hydrocarbon reservoirs based on in-situ stress-loading CT imaging and U-Net deep learning

Deep and ultra-deep hydrocarbon reservoirs are regarded as an attractive target for onshore oil and gas exploration and development in China. The multi-scale fractures are widely distributed in this type of reservoir. The fluid-solid coupling effect is strong, which can greatly affect the fluid flow, resulting in a low hydrocarbon recovery with the average value less than 15%. To explore the underlying flow dynamics during efficient development of deep and ultra-deep hydrocarbon reservoirs, it is of great importance to characterize the multi-scale fracture-pore structure evolution of rock affected by strong geo-stress field variation. To tackle the above issues, an actual rock drilled from a typical ultra-deep reservoir in Tarim Basin are selected to conduct in-situ stress-loading computed tomography (CT) scanning experiments, and the CT gray images of rock microstructure under different dynamic loading conditions are obtained. A fully convolutional neural network (U-Net) deep learning segmentation algorithm is introduced to accurately distinguish the rock skeleton, pore space and fractures in deep rock. The three-dimensional (3-D) digital rock model of deformed multi-scale fracture-porous media under different dynamic loading conditions is ultimately established to investigate the evolution of fracture morphology and deep rock microstructure as the in-situ stress is gradually loaded. It indicates that, the U-Net deep learning semantic segmentation algorithm can accurately segment CT images of deep rocks, and the established 3D digital rock model of fractured porous media can accurately represent the pore-throat distribution, fracture morphology, and pore-scale topological connectivity. At the initial stage of stress loading, there are few micro-fractures inside the rock, and the fracture connectivity is poor. Due to effect of rock compaction, the average pore throat radius increases while pore throat ratio, coordination number and tortuosity greatly decrease. As the effective stress is gradually loaded, the micro-fractures begin to propagate, leading to stronger heterogeneity of micro-fractures’ distribution and better topological connectivity, and both the coordination number and tortuosity gradually increase. After the deep rock is fractured, micro-fractures run through a fracture network and all the above topological parameters increase greatly.

An One-step Triple Enhanced weakly supervised semantic segmentation using image-level labels.

Weakly supervised semantic segmentation, based on image-level labels, abandons the pixel-level labels relied upon by traditional semantic segmentation algorithms. It only utilizes images as supervision information, thereby reducing the time cost and human resources required for marking pixel data. The prevailing approach in weakly supervised segmentation involves two-step method, introducing an additional network and numerous parameters, thereby complicating the model structure. Furthermore, image-level labels typically furnishes only category information for the entire image, lacking specific location details and accurate target boundaries during model training. We propose an innovative One-Step Triple Enhanced weakly supervised semantic segmentation network(OSTE). OSTE streamlines the model structure, which can accomplish both pseudo-labels generation and semantic segmentation tasks in just one step. Furthermore, we augment the weakly supervised semantic segmentation network in three key aspects based on the class activation map construction method, thereby enhancing segmentation accuracy: Firstly, by integrating local information from the activation map with the image, we can enhance the network's localization and expansion capabilities to obtain more accurate and rich location information. Then, we refine the seed areas of the class activation map by exploiting the correlation between multi-level feature. Finally, we incorporate conditional random field theory to generate pseudo-labels with higher confidence and richer boundary information. In comparison to the prevailing two-step weakly supervised semantic segmentation schemes, the segmentation network proposed in this paper achieves a more competitive mean Intersection over Union (mIoU) score of 58.47% on Pascal VOC. Additionally, it enhances the mIoU score by at least 5.03% when compared to existing end-to-end schemes.

Semantic Segmentation Uncertainty Assessment of Different U-net Architectures for Extracting Building Footprints

Abstract. Automatic extraction of building footprints from aerial and space imageries has been found ever increasing importance in urban planning, disaster management, and environmental monitoring. However, achieving accurate building footprint extraction poses significant challenges due to diverse building characteristics and their similarities to their background elements. While conventional methods in building footprint extraction have mainly relied on image processing techniques, recent advancements in deep learning, particularly semantic segmentation algorithms like U-Net, have shown promise in addressing these challenges through machine learning. This study explores different depths of the U-Net model for building footprint extraction, aiming to identify the optimum architecture while investigating the semantic uncertainty of the building footprint extraction. Utilizing aerial imagery from cities including Berlin, Paris, Chicago, and Zurich, collected from Google Maps and OpenStreetMap (OSM) data, five U-Net models have been compared with varying depths. In addition, the impact of dataset sizes and learning rates on model performance has been investigated. Results confirmed that the U-Net-32-1024 model achieves the highest intersection over union (IoU), Accuracy, and F1-score. Moreover, increasing the training dataset size leads to significant improvements in model performance with IoU, Accuracy and F1-score reaching their values of 73.73%, 88.65% and 88.53%. However, challenges remain in accurately delineating buildings in dense urban areas. Nonetheless, our findings demonstrated the effectiveness of U-Net models in building footprint extraction.

Key technology for digital twins in the architecture, engineering, and construction industry: new advances in point cloud semantic segmentation algorithms for buildings

Key technology for digital twins in the architecture, engineering, and construction industry: new advances in point cloud semantic segmentation algorithms for buildings

Competing for Pixels: A Self-play Algorithm for Weakly-supervised Semantic Segmentation.

Weakly-supervised semantic segmentation (WSSS) methods, reliant on image-level labels indicating object presence, lack explicit correspondence between labels and regions of interest (ROIs), posing a significant challenge. Despite this, WSSS methods have attracted attention due to their much lower annotation costs compared to fully-supervised segmentation. Leveraging reinforcement learning (RL) self-play, we propose a novel WSSS method that gamifies image segmentation of a ROI. We formulate segmentation as a competition between two agents that compete to select ROI-containing patches until exhaustion of all such patches. The score at each time-step, used to compute the reward for agent training, represents likelihood of object presence within the selection, determined by an object presence detector pre-trained using only image-level binary classification labels of object presence. Additionally, we propose a game termination condition that can be called by either side upon exhaustion of all ROI-containing patches, followed by the selection of a final patch from each. Upon termination, the agent is incentivised if ROI-containing patches are exhausted or disincentivised if a ROI-containing patch is found by the competitor. This competitive setup ensures minimisation of over- or under-segmentation, a common problem with WSSS methods. Extensive experimentation across four datasets demonstrates significant performance improvements over recent state-of-the-art methods. Code: https://github.com/s-sd/spurl/tree/main/wss.

Image‐based quality control of fresh concrete based on semantic segmentation algorithms

AbstractIn practice, various empirical methods such as the flow table test or the slump test are in use worldwide for assessing the workability of fresh concrete on the construction site. The majority of these tests has in common, that fresh concrete is subjected to some kind of defined flow process on a standardized table‐like platform and that the geometrical properties of the material after the flow has ceased is determined by simple means such as measuring the flow cake diameter or its sag. The paper at hand proposes a novel image‐based approach for an automatic derivation of concrete properties as part of the flow table test. The image‐based method enables a digital evaluation of concrete properties. By combining digital image analysis and deep learning methods, not only the consistency but also an abundance of additional concrete properties can be derived from image data. In this way, the quality control of the fresh concrete can be expanded to include a large number of additional parameters, currently not available to the producer nor to the construction site. This data can be integrated into a digital control loop, with which communication between the concrete producer and the construction site can be automated using highly precise real‐time data.

RS-SLAM: real time semantic slam with driverless car using LiDAR-camera-IMU sensing

Abstract Accurate and robust Simultaneous Localization and Mapping (SLAM) technology is a critical component of driverless cars, and semantic information plays a vital role in their analysis and understanding of the scene. In the actual scene, the moving object will produce the shadow phenomenon in the mapping process. The positioning accuracy and mapping effect will be affected. Therefore, we propose a semantic SLAM framework combining LiDAR, IMU, and camera, which includes a semantic fusion front-end odometry module and a closed-loop back-end optimization module based on semantic information. An improved image semantic segmentation algorithm based on Deeplabv3+ is designed to enhance the performance of the image semantic segmentation model by replacing the backbone network and introducing an attention mechanism to ensure the accuracy of point cloud segmentation. Dynamic objects are detected and eliminated by calculating the similarity score of semantic labels. A loop closure detection method based on semantic information is proposed to detect key semantic features and use threshold range detection and point cloud re-matching to establish the correct loop closure detection, and finally reduce the global cumulative error and improve the global trajectory accuracy using graph optimization to ultimately obtain the global motion trajectory and realize the construction of 3D semantic maps. We evaluated it on the KITTI dataset and collected a dataset for evaluation by ourselves, which includes four different sequences. The results show that the proposed framework has good positioning accuracy and mapping effect in large-scale urban road environments.

RAF-Unet: A Remote Sensing Identification Method for Forest Land Information with Modified Unet

Abstract Carrying out remote sensing refinement identification of forest land in complex environment is of great significance for timely mapping of forest distribution. Aiming at the problem that remote sensing images have bias in the extraction of forest land information data, based on the semantic segmentation algorithm Unet, combining the ResNet50 deep learning network, the attention mechanism module and the feature pyramid structure, we construct RAF-Unet (ResNet+Attention+FPN+Unet) to improve the extraction of forest land information data. The ResNet50 classification network is used as the encoder of the Unet network to extract the feature maps at five different scales; then, the attention mechanism module is introduced in the decoder stage of the Unet network to extract the key task goal information by learning the weight values of the features; finally, the feature pyramid structure is used in the output stage of the encoder to fuse the information from the shallow network and the deep network to extract the remote sensing forest land information in the image. The results show that the RAF-Unet algorithm outperforms the Unet algorithm in all the indexes, with a precision of 95.24%, a recall of 91.80%, an F1-score value of 93.49%, an intersection over union of 87.63%, and an accuracy of 93.68%; the validity of the modules is verified by the ablation experiments, and the ResNet network, the attention mechanism, and the feature pyramid structure are all effective in improve the classification effect. It helps the forestry department to better manage and dynamically monitor forestry information, which is of great significance to the scientific development, utilization and protection of forest land resources.

Development of an Algorithm for Semantic Segmentation of Earth Remote Sensing Data to Determine Phytoplankton Populations

Introduction. Computer vision is widely used for semantic segmentation of Earth remote sensing (ERS) data. The method allows monitoring ecosystems, including aquatic ones. Algorithms that maintain the quality of semantic segmentation of ERS images are in demand, specifically, to identify areas with phytoplankton, where water blooms— the cause of suffocation — are possible. The objective of the study is to create an algorithm that processes satellite data as input information for the formation and checking of mathematical models of hydrodynamics, which are used to monitor the state of water bodies. Various algorithms for semantic segmentation are described in the literature. New research focuses on enhancing the reliability of recognition — often using neural networks. This approach is modified in the presented work. To develop the direction, a new set of information from open sources and synthetic data are proposed. They are aimed at improving the generalization ability of the model. For the first time, the contour area of the phytoplankton population is compared to the database — and thus the boundary conditions are formed for the implementation of mathematical models and the construction of boundary-adaptive grids.Materials and Methods. The set of remote sensing images was supplemented with the author's augmentation algorithm in Python. Computer vision segmented areas of phytoplankton populations in the images. The U-Net convolutional neural network (CNN) was trained on the basis of NVIDIA Tesla T4 computing accelerators.Results. To automate the detection of phytoplankton distribution areas, a computer vision algorithm based on the U-Net CNN was developed. The model was evaluated by the calculated values of the main quality metrics related to segmentation tasks. The following metric values were obtained: Precision = 0.89, Recall = 0.88, F1 = 0.87, Dice = 0.87, and IoU = 0.79. Graphical visualization of the results of CNN learning on the training and validation sets showed good quality of model learning. This is evidenced by small changes in the loss function at the end of training. The segmentation performed by the model turned out to be close to manual marking, which indicated the high quality of the proposed solution. The area of the segmented region of the phytoplankton population was calculated by the area of one pixel. The result obtained for the original image was 51202.5 (based on information about the number of pixels related to the bloom of blue-green algae). The corresponding result of the modeling was 51312.Discussion and Conclusion. The study expands theoretical and practical knowledge on the use of convolutional neural networks for semantic segmentation of space imagery data. Given the results of the work, it is possible to assess the potential for automating the process of semantic segmentation of remote sensing data to determine the boundaries of phytoplankton populations using artificial intelligence. The use of the proposed computer vision model to obtain contours of water bloom due to phytoplankton will provide for the creation of databases — the basis for environmental monitoring of water resources and predictive modeling of hydrobiological processes.

SMLS-YOLO: an extremely lightweight pathological myopia instance segmentation method.

Pathological myopia is a major cause of blindness among people under 50 years old and can result in severe vision loss in extreme cases. Currently, its detection primarily relies on manual methods, which are slow and heavily dependent on the expertise of physicians, making them impractical for large-scale screening. To tackle these challenges, we propose SMLS-YOLO, an instance segmentation method based on YOLOv8n-seg. Designed for efficiency in large-scale screenings, SMLS-YOLO employs an extremely lightweight model. First, StarNet is introduced as the backbone of SMLS-YOLO to extract image features. Subsequently, the StarBlock from StarNet is utilized to enhance the C2f, resulting in the creation of the C2f-Star feature extraction module. Furthermore, shared convolution and scale reduction strategies are employed to optimize the segmentation head for a more lightweight design. Lastly, the model incorporates the Multi-Head Self-Attention (MHSA) mechanism following the backbone to further refine the feature extraction process. Experimental results on the pathological myopia dataset demonstrate that SMLS-YOLO outperforms the baseline YOLOv8n-seg by reducing model parameters by 46.9%, increasing Box mAP@0.5 by 2.4%, and enhancing Mask mAP@0.5 by 4%. Furthermore, when compared to other advanced instance segmentation and semantic segmentation algorithms, SMLS-YOLO also maintains a leading position, suggesting that SMLS-YOLO has promising applications in the segmentation of pathological myopia images.

Real-Time Semantic Segmentation Algorithm for Street Scenes Based on Attention Mechanism and Feature Fusion

In computer vision, the task of semantic segmentation is crucial for applications such as autonomous driving and intelligent surveillance. However, achieving a balance between real-time performance and segmentation accuracy remains a significant challenge. Although Fast-SCNN is favored for its efficiency and low computational complexity, it still faces difficulties when handling complex street scene images. To address this issue, this paper presents an improved Fast-SCNN, aiming to enhance the accuracy and efficiency of semantic segmentation by incorporating a novel attention mechanism and an enhanced feature extraction module. Firstly, the integrated SimAM (Simple, Parameter-Free Attention Module) increases the network’s sensitivity to critical regions of the image and effectively adjusts the feature space weights across channels. Additionally, the refined pyramid pooling module in the global feature extraction module captures a broader range of contextual information through refined pooling levels. During the feature fusion stage, the introduction of an enhanced DAB (Depthwise Asymmetric Bottleneck) block and SE (Squeeze-and-Excitation) attention optimizes the network’s ability to process multi-scale information. Furthermore, the classifier module is extended by incorporating deeper convolutions and more complex convolutional structures, leading to a further improvement in model performance. These enhancements significantly improve the model’s ability to capture details and overall segmentation performance. Experimental results demonstrate that the proposed method excels in processing complex street scene images, achieving a mean Intersection over Union (mIoU) of 71.7% and 69.4% on the Cityscapes and CamVid datasets, respectively, while maintaining inference speeds of 81.4 fps and 113.6 fps. These results indicate that the proposed model effectively improves segmentation quality in complex street scenes while ensuring real-time processing capabilities.

YLS-SLAM: a real-time dynamic visual SLAM based on semantic segmentation

Purpose Traditional visual simultaneous localization and mapping (SLAM) systems are primarily based on the assumption that the environment is static, which makes them struggle with the interference caused by dynamic objects in complex industrial production environments. This paper aims to improve the stability of visual SLAM in complex dynamic environments through semantic segmentation and its optimization. Design/methodology/approach This paper proposes a real-time visual SLAM system for complex dynamic environments based on YOLOv5s semantic segmentation, named YLS-SLAM. The system combines semantic segmentation results and the boundary semantic enhancement algorithm. By recognizing and completing the semantic masks of dynamic objects from coarse to fine, it effectively eliminates the interference of dynamic feature points on the pose estimation and enhances the retention and extraction of prominent features in the background, thereby achieving stable operation of the system in complex dynamic environments. Findings Experiments on the Technische Universität München and Bonn data sets show that, under monocular and Red, Green, Blue - Depth modes, the localization accuracy of YLS-SLAM is significantly better than existing advanced dynamic SLAM methods, effectively improving the robustness of visual SLAM. Additionally, the authors also conducted tests using a monocular camera in a real industrial production environment, successfully validating its effectiveness and application potential in complex dynamic environment. Originality/value This paper combines semantic segmentation algorithms with boundary semantic enhancement algorithms to effectively achieve precise removal of dynamic objects and their edges, while ensuring the system's real-time performance, offering significant application value.

Decoupling foreground and background with Siamese ViT networks for weakly-supervised semantic segmentation

Due to the coarse granularity of information extraction in image-level annotation-based weakly supervised semantic segmentation algorithms, there exists a significant gap between the generated pseudo-labels and the real pixel-level labels. In this paper, we propose the DeFB-SV framework, which consists of a dual-branch Siamese network structure. This framework separates the foreground and background of images by generating unified resolution and mixed resolution class activation maps, which are then fused to obtain pseudo-labels. The mixed-resolution class activation maps are produced by a new mixed-resolution patch partition method, where we introduce a semantically heuristic patch scorer to divide the image into patches of different sizes based on semantics. Additionally, a novel multi-confidence region division mechanism is proposed to enable the adaptive extraction of the effective parts of pseudo-labels, further enhancing the accuracy of weakly supervised semantic segmentation algorithms. The proposed semantic segmentation framework, DeFB-SV, is evaluated on the PASCAL VOC 2012 and MS COCO 2014 datasets, demonstrating comparable segmentation performance with state-of-the-art methods.

Semantic segmentation algorithm for pantograph based on multi-scale strip pooling attention mechanism and application research

Semantic segmentation algorithm for pantograph based on multi-scale strip pooling attention mechanism and application research

Automatic crack detection method for high-speed railway box girder based on deep learning techniques and inspection robot

Box girders serve as crucial upper-level load-bearing components in high-speed railway simply-supported bridges, requiring sufficient structural rigidity during operation. The occurrence of cracks compromises the overall stiffness of the structure, posing significant safety risks and potentially leading to substantial loss of life and property. Therefore, it is essential to rapidly and accurately detect cracks within the girder structure, particularly in the interior of box girders where access for maintenance by personnel is inconvenient. To address this issue, this paper proposes a robot-based framework for crack detection in high-speed railway box girder, and accurately evaluate the damage status of structures. This comprehensive framework includes an image generation network for generating high-quality crack images, a lightweight object detection algorithm for rapidly identifying crack targets, and a high-precision semantic segmentation algorithm for accurately extracting crack pixels. Comparative analysis with mainstream algorithms validates the superiority of the proposed methods. Moreover, preliminary validation through simulated tests highlights the feasibility of the proposed framework, offering novel method and theoretical support for the intelligent operation and maintenance of high-speed railway bridge structures.