Real-time Semantic Segmentation Research Articles

Research on Efficient Asymmetric Attention Module for Real-Time Semantic Segmentation Networks in Urban Scenes

Currently, numerous high-precision models have been proposed for semantic segmentation, but the model parameters are large and the segmentation speed is slow. Real-time semantic segmentation for urban scenes necessitates a balance between accuracy, inference speed, and model size. In this paper, we present an efficient solution to this challenge, efficient asymmetric attention module net (EAAMNet) for the semantic segmentation of urban scenes, which adopts an asymmetric encoder–decoder structure. The encoder part of the network utilizes an efficient asymmetric attention module to form the network backbone. In the decoding part, we propose a lightweight multi-feature fusion decoder that can maintain good segmentation accuracy with a small number of parameters. Our extensive evaluations demonstrate that EAAMNet achieves a favorable equilibrium between segmentation efficiency, model parameters, and segmentation accuracy, rendering it highly suitable for real-time semantic segmentation in urban scenes. Remarkably, EAAMNet attains a 73.31% mIoU at 128 fps on Cityscapes and a 69.32% mIoU at 141 fps on CamVid without any pre-training. Compared to state-of-the-art models, our approach not only matches their model parameters but also enhances accuracy and increases speed.

Correction to: FBRNet: a feature fusion and border refinement network for real-time semantic segmentation

Correction to: FBRNet: a feature fusion and border refinement network for real-time semantic segmentation

Real-time semantic segmentation with dual interaction fusion network

Real-time semantic segmentation with dual interaction fusion network

Real-time tunnel lining leakage image semantic segmentation via multiple attention mechanisms

One of the key objectives in tunnel illness detection is identifying tunnel lining leakage, and deep learning-based image semantic segmentation approaches can automatically locate tunnel lining leakage. However, in order to meet the real-time processing needs of professional mobile inspection equipment, existing leakage image segmentation approaches have difficulties in identifying real-time, dealing with voids, and dealing with edge discontinuities in the leaking zone. To address the aforementioned issues, this study introduces the PP-LiteSeg-Attn model, which takes the real-time semantic segmentation model PP-LiteSeg-B as baseline model, and combines the multi-layer CBAM attention mechanism and the CoT attention mechanism. Using the publically available dataset Water-Leakage, we trained and validated the PP-LiteSeg-Attn model, and attained IoU and F1 values of 88.18% and 93.72%, respectively, outperforming similar models in both measures. Extensive experiments show that the segmentation speed of the PP-LiteSeg-Attn model reaches 112.28 FPS, which meets real-time requirements, and that the model can effectively solve problems such as the appearance of voids in the seepage area, discontinuity, and fuzzy segmentation of seepage edges. The PP-LiteSeg-Attn model is better applicable to complicated tunnel settings, offering technical references for real-time diagnosis of tunnel illnesses.

MAFNet: dual-branch fusion network with multiscale atrous pyramid pooling aggregate contextual features for real-time semantic segmentation

MAFNet: dual-branch fusion network with multiscale atrous pyramid pooling aggregate contextual features for real-time semantic segmentation

Real-time semantic segmentation network based on parallel atrous convolution for short-term dense concatenate and attention feature fusion

Real-time semantic segmentation network based on parallel atrous convolution for short-term dense concatenate and attention feature fusion

Fast-DSAGCN: Enhancing semantic segmentation with multifaceted attention mechanisms

Real-time semantic segmentation provides precise insights into dynamic street environments for autonomous driving, traffic control, and urban planning. However, state-of-the-art models following attention mechanisms and deep convolutional neural networks have improved semantic segmentation at the cost of complex architectures and high computation complexity. The study aims to mitigate the presence of gridding artifacts and enhance semantic segmentation performance. In addition, we propose a multi-level downsampling approach before employing the depth-wise split separable global convolution with the bottleneck to achieve a trade-off between accuracy and inference time. The spatial attention module used in this study effectively keeps low-level spatial characteristics, enhancing the accuracy of localization, robustness against disturbances, processing efficiency, and the ability to handle occlusions. Thorough tests of the Cityscapes and CamVid datasets available for public access indicate that the model presented is capable of efficiently processing high-resolution photos in real time, resulting in exceptional performance. The model has achieved an accuracy of 72.3% on the cityscapes dataset and 72.7% on the CamVid dataset.

BENet: boundary-enhanced network for real-time semantic segmentation

BENet: boundary-enhanced network for real-time semantic segmentation

Real-time semantic segmentation based on BiSeNetV2 for wild road

Abstract State-of-the-art segmentation models have shown great performance in structured road segmentation. However, these models are not suitable for the wild roads, which are highly unstructured. To tackle the problem of real-time semantic segmentation of wild roads, we propose a Multi-Information Concatenate Network based on BiSeNetV2 and construct a segmentation dataset Dalle Molle institute for artificial intelligence feature segmentation (IDSIAFS) based on Dalle Molle institute for artificial intelligence. The proposed model removes structural redundancy and optimizes the semantic branch based on BiSeNetV2. Moreover, the Dual-Path Semantic Inference Layer (TPSIL) reduces computation by designing the channel dimension of the semantic branch feature map and aggregates feature maps of different depths. Finally, the segmentation results are achieved by fusing both shallow detail information and deep semantic information. Experiments on the IDSIAFS dataset demonstrate that our proposed model achieves an 89.5% Intersection over Union. The comparative experiments on Cityscapes and India driving dataset benchmarks show that proposed model achieves good inference accuracy and faster inference speed.

Lightweight convolutional neural networks for real-time lane segmentation

Autonomous driving depends on reliable perception systems that involve various perception modules and advanced computer vision techniques. One crucial component of these systems is lane detection, which traditional methods often rely on basic features like color or edges that are sensitive to lighting and perspective changes. Recently, convolutional neural networks (CNNs), have revolutionized lane detection. Nevertheless, existing methods still have some limitations, such as the need for pixel-level labeling and computational inefficiency for real-time applications. To address these challenges, this work leverages PP-LiteSeg for real-time semantic segmentation. PP-LiteSegs key elements are its Simple Pyramid Pooling Module (SPPM), Unified Attention Fusion Module (UAFM), and Flexible and Lightweight Decoder (FLD), which optimize lane detection efficiency. The FLD flexibly adjusts computational costs between the encoder and decoder, balancing efficiency and accuracy. The UAFM enhances feature representations using attention mechanisms, increasing segmentation accuracy. The SPPM efficiently aggregates contextual information while reducing computational complexity. The comprehensive method for lane segmentation achieves competitive results on popular lane detection datasets. The proposed model can adapt to different computational capabilities and significantly enhances lane detection efficiency for real-time applications.

SCTNet: Single-Branch CNN with Transformer Semantic Information for Real-Time Segmentation

Recent real-time semantic segmentation methods usually adopt an additional semantic branch to pursue rich long-range context. However, the additional branch incurs undesirable computational overhead and slows inference speed. To eliminate this dilemma, we propose SCTNet, a single branch CNN with transformer semantic information for real-time segmentation. SCTNet enjoys the rich semantic representations of an inference-free semantic branch while retaining the high efficiency of lightweight single branch CNN. SCTNet utilizes a transformer as the training-only semantic branch considering its superb ability to extract long-range context. With the help of the proposed transformer-like CNN block CFBlock and the semantic information alignment module, SCTNet could capture the rich semantic information from the transformer branch in training. During the inference, only the single branch CNN needs to be deployed. We conduct extensive experiments on Cityscapes, ADE20K, and COCO-Stuff-10K, and the results show that our method achieves the new state-of-the-art performance. The code and model is available at https://github.com/xzz777/SCTNet.

Real-time semantic segmentation for underground mine tunnel

Semantic segmentation is the underlying technology for many intelligent applications in underground mines. Unlike the ordinary scenario, the lighting of underground mines varies drastically, and there is heavy dust and mist, which results in poor image quality and greatly hinders the application of image semantic segmentation in underground mines. Because there is currently no datasets for underground mine tunnels available, a dataset named Underground Mine Tunnel Semantic Segmentation Dataset (UMTSSD) have to be constructed to support our research. UMTSSD consists of 3461 meticulously annotated images and 17 annotated categories. A real-time semantic segmentation algorithm named Fast Adaptive Deep Dual-resolution Network (FA-DDRNet) which uses Deep Dual-resolution Network (DDRNet) as the backbone is proposed for underground mines. To enhance the semantic segmentation accuracy in underground environment, FA-DDRNet introduces two modules: Fast Adaptive Input Normalization Module (FAINM) and Scale-wise Residual Cascade Module (SRCM). FAINM can autonomously and quickly adjust normal lighting images, weak lighting images, and overexposed images to improve the robustness of semantic segmentation algorithms. SRCM is integrated into the backbone to swiftly fuse multi-scale features in a cascade fashion, resulting in enhanced detection of objects with diverse shapes in underground environment. Finally, our method achieves exceptional performance with a superior inference speed compared to other semantic segmentation algorithms in UMTSSD. The method can realize running in real-time on low computational power embedded devices, which is well adapted to underground environment.

Tripartite real-time semantic segmentation network with scene commonality

Tripartite real-time semantic segmentation network with scene commonality

Real-time semantic segmentation via mutual optimization of spatial details and semantic information

Semantic segmentation is a pivotal task in the field of computer vision, encompassing diverse applications and undergoing continuous development. Despite the growing dominance of deep learning methods in this field, many existing network models suffer from trade-offs between accuracy and computational cost, or between speed and accuracy. In essence, semantic segmentation aims to extract semantic information from deep features and optimize them before upsampling output. However, shallow features tend to contain more detailed information but also more noise, while deep features have strong semantic information but lose some spatial information. To address this issue, we propose a novel mutual optimization strategy based on shallow spatial information and deep semantic information, and construct a details and semantic mutual optimization network (DSMONet). This effectively reduces the noise in the shallow features and guides the deep features to reconstruct the lost spatial information, avoiding cumbersome side auxiliary or complex decoders. The Mutual Optimization Module (MOM) includes Semantic Adjustment Details Module (SADM) and Detail Guided Semantic Module (DGSM), which enables mutual optimization of shallow spatial information and deep semantic information. Comparative evaluations against other methods demonstrate that DSMONet achieves a favorable balance between accuracy and speed. On the Cityscapes dataset, DSMONet achieves performances of 79.3% mean of class-wise intersection-over-union (mIoU)/44.6 frames per second (FPS) and 78.0% mIoU/102 FPS. The code is available at https://github.com/m828/DSMONet.

LFFNet: lightweight feature-enhanced fusion network for real-time semantic segmentation of road scenes

LFFNet: lightweight feature-enhanced fusion network for real-time semantic segmentation of road scenes

Semantic segmentation of raw multispectral laser scanning data from urban environments with deep neural networks

Real-time semantic segmentation of point clouds has increasing importance in applications related to 3D city modelling and mapping, automated inventory of forests, autonomous driving and mobile robotics. Current state-of-the-art point cloud semantic segmentation methods rely heavily on the availability of 3D laser scanning data. This is problematic in regards of low-latency, real-time applications that use data from high-precision mobile laser scanners, as those are typically 2D line scanning devices. In this study, we experiment with real-time semantic segmentation of high-density multispectral point clouds collected from 2D line scanners in urban environments using encoder - decoder convolutional neural network architectures. We introduce a rasterized multi-scan input format that can be constructed exclusively from the raw (non-georeferenced profiles) 2D laser scanner measurement stream without odometry information. In addition, we investigate the impact of multispectral data on the segmentation accuracy. The dataset used for training, validation and testing was collected with multispectral FGI AkhkaR4-DW backpack laser scanning system operating at the wavelengths of 905 nm and 1550 nm, and consists in total of 228 million points (39 583 scans). The data was divided into 13 classes that represent various targets in urban environments. The results show that the increased spatial context of the multi-scan format improves the segmentation performance on the single-wavelength lidar dataset from 45.4 mIoU (a single scan) to 62.1 mIoU (24 consecutive scans). In the multispectral point cloud experiments we achieved a 71 % and 28 % relative increase in the segmentation mIoU (43.5 mIoU) as compared to the purely single-wavelength reference experiments, in which we achieved 25.4 mIoU (905 nm) and 34.1 mIoU (1550 nm). Our findings show that it is possible to semantically segment 2D line scanner data with good results by combining consecutive scans without the need for odometry information. The results also serve as motivation for developing multispectral mobile laser scanning systems that can be used in challenging urban surveys.

Real-time semantic segmentation network with an enhanced backbone based on Atrous spatial pyramid pooling module

Real-time semantic segmentation methods demand high levels of both speed and accuracy. However, recent methods in semantic segmentation generally use the models pre-trained on ImageNet as their backbone to increase the field-of-view quickly. Due to the low resolution and plenty of image categories, the existing methods often suffer from insufficient receptive field and channel redundancy. To tackle these problems, we propose a novel block structure named Atrous block (A block) in this paper. By repeating this block structure, we have developed our dedicated backbone tailored for the semantic segmentation task, eliminating the need for additional modules. According to the different output channels of the backbone, we obtain our own architecture called Atrous Network (ANet), Furthermore, we propose a lightweight decoder to recover the spatial details lost during encoding. To demonstrate the effectiveness of our methods, we have validated our methods on CityScapes and CamVid datasets, using mixed precision during the training process. Compared with the state-of-the-art methods, our models achieve competitive results. Among them, our small size model achieves 75.2 % mean intersection over union (mIOU) on CityScapes dataset and 78.3 % mIOU on CamVid dataset, both without ImageNet pretraining.

KDP-Net: An Efficient Semantic Segmentation Network for Emergency Landing of Unmanned Aerial Vehicles

As the application of UAVs becomes more and more widespread, accidents such as accidental injuries to personnel, property damage, and loss and destruction of UAVs due to accidental UAV crashes also occur in daily use scenarios. To reduce the occurrence of such accidents, UAVs need to have the ability to autonomously choose a safe area to land in an accidental situation, and the key lies in realizing on-board real-time semantic segmentation processing. In this paper, we propose an efficient semantic segmentation method called KDP-Net for characteristics such as large feature scale changes and high real-time processing requirements during the emergency landing process. The proposed KDP module can effectively improve the accuracy and performance of the semantic segmentation backbone network; the proposed Bilateral Segmentation Network improves the extraction accuracy and processing speed of important feature categories in the training phase; and the proposed edge extraction module improves the classification accuracy of fine features. The experimental results on the UDD6 and SDD show that the processing speed of this method reaches 85.25 fps and 108.11 fps while the mIoU reaches 76.9% and 67.14%, respectively. The processing speed reaches 53.72 fps and 38.79 fps when measured on Jetson Orin, which can meet the requirements of airborne real-time segmentation for emergency landing.

ULAF-Net: Ultra lightweight attention fusion network for real-time semantic segmentation

ULAF-Net: Ultra lightweight attention fusion network for real-time semantic segmentation

FBRNet: a feature fusion and border refinement network for real-time semantic segmentation

FBRNet: a feature fusion and border refinement network for real-time semantic segmentation