Cityscapes Dataset Research Articles

ETFT: Equiangular Tight Frame Transformer for Imbalanced Semantic Segmentation

Semantic segmentation often suffers from class imbalance, where the label ratio for each class in the dataset is not uniform. Recent studies have addressed the issue of class imbalance in semantic segmentation by leveraging the neural collapse phenomenon in conjunction with an Equiangular Tight Frame (ETF). While the use of ETF aids in enhancing the discriminability of minor classes, class correlation is another crucial factor that must be taken into account. However, managing the balance between class correlation and discrimination through neural collapse remains challenging, as these properties inherently conflict with one another. Moreover, this control is established during the training stage, resulting in a fixed classifier. There is no guarantee that this classifier will consistently perform well with different input images. To address this problem, we propose an Equiangular Tight Frame Transformer (ETFT), a transformer-based model that jointly processes the features and classifier using ETF structure, and dynamically generates the classifier as a function of the input for imbalanced semantic segmentation. Specifically, the classifier initialized with the ETF structure is jointly processed with the input patch tokens during the attention process. As a result, the transformed patch tokens, aided by the ETF structure, achieve discriminability between classes while preserving contextual correlation. The classifier, initially structured as an ETF, is adjusted to incorporate the correlation information, benefiting from the attention mechanism. Furthermore, the learned classifier is combined with the fixed ETF classifier, leveraging the advantages of both. Extensive experiments demonstrate that the proposed method outperforms state-of-the-art methods for imbalanced semantic segmentation on both the ADE20K and Cityscapes datasets.

Cross-domain autonomous driving visual segmentation based on enhanced target data learning

Within the broader context of Information and Communications Technology (ICT), the quest for reliable and scalable visual segmentation methods poses significant challenges, particularly in autonomous driving, where real-world scene complexity requires advanced solutions. To address data scarcity and improve segmentation performance, we propose a novel unsupervised domain adaptation (UDA) approach that enhances target domain learning. Our method introduces multiple perturbations consistency, leveraging spatial context within the target domain to improve recognition. By applying perturbations at input and feature levels and using a consistency loss, we enhance contextual learning. Additionally, a weight mapping technique reduces the impact of detrimental source domain information. Experimental results demonstrate that our approach outperforms baseline methods on the GTAV→Cityscapes and SYNTHIA→Cityscapes datasets.

Road surface semantic segmentation for autonomous driving

Although semantic segmentation is widely employed in autonomous driving, its performance in segmenting road surfaces falls short in complex traffic environments. This study proposes a frequency-based semantic segmentation with a transformer (FSSFormer) based on the sensitivity of semantic segmentation to frequency information. Specifically, we propose a weight-sharing factorized attention to select important frequency features that can improve the segmentation performance of overlapping targets. Moreover, to address boundary information loss, we used a cross-attention method combining spatial and frequency features to obtain further detailed pixel information. To improve the segmentation accuracy in complex road scenarios, we adopted a parallel-gated feedforward network segmentation method to encode the position information. Extensive experiments demonstrate that the mIoU of FSSFormer increased by 2% compared with existing segmentation methods on the Cityscapes dataset.

MEDANet: More Efficient Dual Attention Network for Scene Segmentation

The dual attention module is a potent semantic segmentation technique renowned for its capabilities, yet it often faces significant computational demands and GPU memory usage. To tackle these challenges, we introduce an advanced dual perception network comprising two modules: A streamlined Multi-scale Efficient Position Attention Module (MEPAM) and an optimized Efficient Channel Attention Module (MECAM). MEPAM incorporates multi-scale global average pooling into the Position Attention Module (PAM), substantially cutting computational overhead and memory consumption without compromising performance. Meanwhile, MECAM integrates compressed convolutions into the Channel Attention Module (CAM), improving segmentation accuracy and inference speed compared to conventional methods like DANet. Our approach underwent comprehensive evaluation on a semantic segmentation benchmark dataset, showcasing superior performance. For instance, on the Cityscapes dataset, our method achieves an IoU of 82.2%. In terms of efficiency gains, MEPAM operates nearly 1.97 times faster than the standard PAM module on GPU, while requiring 7.55 times less memory with a [Formula: see text] input. Similarly, MECAM achieves approximately 2.2 times faster processing than CAM, while cutting GPU memory usage by 7.53 times. This innovative dual perception network not only enhances segmentation accuracy and speed but also addresses the computational challenges associated with traditional dual attention modules.

Enhancing the utilization of uncertain pixels in semi-supervised semantic segmentation

In semi-supervised semantic segmentation, determining the correct label for uncertain pixels is crucial yet challenging. The recently proposed virtual category (VC) learning achieves excellent performance by creating a potential category (PC) set to exploit the valuable part of the uncertain pixels effectively. However, the potential category set in the existing method is not accurate enough at the beginning of iterations, which could lead to less accurate segmentation results. To better utilize uncertain pixels, we propose to improve the strategy of constructing the potential category set. Specifically, we adjust the threshold based on the model’s optimization status and split the pseudo-labels into low/high-confidence areas to create a more appropriate PC set. At the same time, due to uncertainty in boundary pixels, it is challenging to achieve precise object segmentation, and there is currently no method available to optimize these areas in semi-supervised segmentation tasks. Therefore, we propose a entropy-based method to identify boundary areas and design a novel boundary refinement network to process labeled and unlabeled data separately to optimize segmentation. The experimental results demonstrate that our proposed method excels in accurately segmenting the boundary areas of the targets. Furthermore, it achieves state-of-the-art semi-supervised segmentation on the Pascal VOC and Cityscapes datasets.

Lightweight multi-scale feature dense cascade neural network for scene understanding of intelligent autonomous platform

Understanding the surrounding environment is an essential part of intelligent autonomous platform to complete autonomous driving or other autonomous tasks. Thereinto, scene understanding is a key technology to understand environmental information. For intelligent autonomous platform, it not only requires high accuracy, but also has a strict requirement for the inference speed. Moreover, it should be able to perform normally in complex environments such as target occlusion, the influence of illumination, and objects at different sizes. For tackling these problems, in this study, a lightweight multi-scale feature dense cascade neural network (LMFDCNet) is proposed in real-time scene understanding in complex scenes. Furthermore, trilateral fusion module is adopted to enhance low-level detailed features or high-level semantic features of each branch to describe the object better in different environments. A multi-scale upsampling module that fuses contextual information at different scales allows objects to be represented at different size. The results indicate that LMFDCNet produces 99.1 % classification accuracy on the tobacco dataset. On the Cityscapes dataset, we achieve 74.2 % Mean Intersection over Union (MIoU) with 13.6 M parameters at the speed of 38.2 FPS on a single 1070Ti card. To test the performance on the real-time application in the real world, LMFDCNet is deployed on the autonomous platform for real-time vision task such as classification of tobacco leaf state in curing process and semantic segmentation in the driving process of autonomous platform. The results reveal that LMFDCNet perform well on the AI embedded device and could be applied into intelligent autonomous platform.

Style Optimization Networks for real-time semantic segmentation of rainy and foggy weather

Semantic segmentation is an essential task in the field of computer vision. Existing semantic segmentation models can achieve good results under good weather and lighting conditions. However, when the external environment changes, the effectiveness of these models are seriously affected. Therefore, we focus on the task of semantic segmentation in rainy and foggy weather. Fog is a common phenomenon in rainy weather conditions and has a negative impact on image visibility. Besides, to make the algorithm satisfy the application requirements of mobile devices, the computational cost and the real-time requirement of the model have become one of the major points of our research. In this paper, we propose a novel Style Optimization Network (SONet) architecture, containing a Style Optimization Module (SOM) that can dynamically learn style information, and a Key information Extraction Module (KEM) that extracts important spatial and contextual information. This can improve the learning ability and robustness of the model for rainy and foggy conditions. Meanwhile, we achieve real-time performance by using lightweight modules and a backbone network with low computational complexity. To validate the effectiveness of our SONet, we synthesized CityScapes dataset for rainy and foggy weather and evaluated the accuracy and complexity of our model. Our model achieves a segmentation accuracy of 75.29% MIoU and 83.62% MPA on a NVIDIA TITAN Xp GPU. Several comparative experiments have shown that our SONet can achieve good performance in semantic segmentation tasks under rainy and foggy weather, and due to the lightweight design of the model we have a good advantage in both accuracy and model complexity.

An Active Learning Semantic Segmentation Model Based on an Improved Double Deep Q-Network

Image semantic segmentation is essential in fields such as computer vision, autonomous driving, and human-computer interaction due to its ability to accurately identify and classify each pixel in an image. However, this task is fraught with challenges, including the difficulty of obtaining detailed pixel labels and the problem of class imbalance in segmentation datasets. These challenges can hinder the effectiveness and efficiency of segmentation models. To address these issues, we propose an active learning semantic segmentation model named CG_D3QN, which is designed and implemented based on an enhanced Double Deep Q-Network (D3QN). The proposed CG_D3QN model incorporates a hybrid network structure that combines a dueling network architecture with Gated Recurrent Units (GRUs). This novel approach improves policy evaluation accuracy and computational efficiency by mitigating a Q-value overestimation and making better use of historical state information. Our experiments, conducted on the CamVid and Cityscapes datasets, reveal that the CG_D3QN model significantly reduces the number of required sample annotations by 65.0% compared to traditional methods. Additionally, it enhances the mean Intersection over Union (IoU) for underrepresented categories by approximately 1% to 3%. These results highlight the model’s effectiveness in lowering annotation costs, addressing class imbalance, and its versatility across different segmentation networks.

Generative Denoise Distillation: Simple stochastic noises induce efficient knowledge transfer for dense prediction

Knowledge distillation is the process of transferring knowledge from a more powerful large model (teacher) to a simpler counterpart (student). Numerous current approaches involve the student imitating the knowledge of the teacher directly, which tend to learn each spatial location’s features indiscriminately. Real-world datasets frequently exhibit noise, motivating models to acquire compact and representative features instead of memorizing every details in the training data. To derive a more compact representation (concept feature) from the teacher, we propose an innovative method, termed Generative Denoise Distillation (GDD), where stochastic noises are added to the concept feature of the student to embed them into the generated instance feature from a shallow network. Then, the generated instance feature is aligned with the knowledge of the instance from the teacher. This process diminishes model redundancy while concurrently augmenting its capacity for generalization. We extensively experiment with object detection, instance segmentation, and semantic segmentation to demonstrate the versatility and effectiveness of our method. Notably, GDD achieves new state-of-the-art performance in the tasks mentioned above. We have achieved substantial improvements in semantic segmentation by enhancing PspNet and DeepLabV3, both of which are based on ResNet-18, resulting in mIoU scores of 74.67 and 77.69, respectively, surpassing their previous scores of 69.85 and 73.20 on the Cityscapes dataset of 20 categories. The code is available at https://github.com/ZhgLiu/GDD.

HRDLNet: a semantic segmentation network with high resolution representation for urban street view images

Semantic segmentation of urban street scenes has attracted much attention in the field of autonomous driving, which not only helps vehicles perceive the environment in real time, but also significantly improves the decision-making ability of autonomous driving systems. However, most of the current methods based on Convolutional Neural Network (CNN) mainly use coding the input image to a low resolution and then try to recover the high resolution, which leads to problems such as loss of spatial information, accumulation of errors, and difficulty in dealing with large-scale changes. To address these problems, in this paper, we propose a new semantic segmentation network (HRDLNet) for urban street scene images with high-resolution representation, which improves the accuracy of segmentation by always maintaining a high-resolution representation of the image. Specifically, we propose a feature extraction module (FHR) with high-resolution representation, which efficiently handles multi-scale targets and high-resolution image information by efficiently fusing high-resolution information and multi-scale features. Secondly, we design a multi-scale feature extraction enhancement (MFE) module, which significantly expands the sensory field of the network, thus enhancing the ability to capture correlations between image details and global contextual information. In addition, we introduce a dual-attention mechanism module (CSD), which dynamically adjusts the network to more accurately capture subtle features and rich semantic information in images. We trained and evaluated HRDLNet on the Cityscapes Dataset and the PASCAL VOC 2012 Augmented Dataset, and verified the model’s excellent performance in the field of urban streetscape image segmentation. The unique advantages of our proposed HRDLNet in the field of semantic segmentation of urban streetscapes are also verified by comparing it with the state-of-the-art methods.

Attention-based fusion network for RGB-D semantic segmentation

RGB-D semantic segmentation can realize a profound comprehension of scenes, which is crucial in various computer vision tasks. However, due to the inherent modal variances and image noise, achieving superior segmentation using existing methods remains challenging. In this paper, we propose an attention-based fusion network for RGB-D semantic segmentation. Specifically, our network employs a forward multi-step propagation strategy and a backward progressive bootstrap fusion strategy based on the encoder–decoder architecture. By aggregating feature maps at different scales, we effectively diminish the uncertainty in the final prediction. Meanwhile, we introduce a Channel and Spatial Rectification Module (CSRM) to enable multi-dimensional interactions and noise removal. In order to achieve comprehensive integration between RGB and depth images, we put the rectified features into the Cross-Attention Fusion Module(CAFM). Extensive experiments show that our network can adeptly manage a diverse array of complex scenarios, demonstrating its innovative strength with superior performance and robust effectiveness across indoor NYU Depth V2 and SUN-RGBD datasets, and extending its capabilities to the outdoor Cityscapes dataset.

Improving real-time object detection in Internet-of-Things smart city traffic with YOLOv8-DSAF method

With the rise of global smart city construction, target detection technology plays a crucial role in optimizing urban functions and improving the quality of life. However, existing target detection technologies still have shortcomings in terms of accuracy, real-time performance, and adaptability. To address this challenge, this study proposes an innovative target detection model. Our model adopts the structure of YOLOv8-DSAF, comprising three key modules: depthwise separable convolution (DSConv), dual-path attention gate module (DPAG), and feature enhancement module (FEM). Firstly, DSConv technology optimizes computational complexity, enabling real-time target detection within limited hardware resources. Secondly, the DPAG module introduces a dual-channel attention mechanism, allowing the model to selectively focus on crucial areas, thereby improving detection accuracy in high-dynamic traffic scenarios. Finally, the FEM module highlights crucial features to prevent their loss, further enhancing detection accuracy. Additionally, we propose an Internet of Things smart city framework consisting of four main layers: the application domain, the Internet of Things infrastructure layer, the edge layer, and the cloud layer. The proposed algorithm utilizes the Internet of Things infrastructure layer, edge layer, and cloud layer to collect and process data in real-time, achieving faster response times. Experimental results on the KITTI V and Cityscapes datasets indicate that our model outperforms the YOLOv8 model. This suggests that in complex urban traffic scenarios, our model exhibits superior performance with higher detection accuracy and adaptability. We believe that this innovative model will significantly propel the development of smart cities and advance target detection technology.

DECNet: Dense embedding contrast for unsupervised semantic segmentation

Unsupervised semantic segmentation is important for understanding that each pixel belongs to known categories without annotation. Recent studies have demonstrated promising outcomes by employing a vision transformer backbone pre-trained on an image-level dataset in a self-supervised manner. However, those methods always depend on complex architectures or meticulously designed inputs. Naturally, we are attempting to explore the investment with a straightforward approach. To prevent over-complication, we introduce a simple Dense Embedding Contrast network (DECNet) for unsupervised semantic segmentation in this paper. Specifically, we propose a Nearest Neighbor Similarity strategy (NNS) to establish well-defined positive and negative pairs for dense contrastive learning. Meanwhile, we optimize a contrastive objective named Ortho-InfoNCE to alleviate the false negative problem inherent in contrastive learning for further enhancing dense representations. Finally, extensive experiments conducted on COCO-Stuff and Cityscapes datasets demonstrate that our approach outperforms state-of-the-art methods.

Feature boosting with efficient attention for scene parsing

The complexity of scene parsing grows with the number of object and scene classes, which is higher in unrestricted open scenes. The biggest challenge is to model the spatial relation between scene elements while succeeding in identifying objects at smaller scales. This paper presents a novel feature-boosting network that gathers spatial context from multiple levels of feature extraction and computes the attention weights for each level of representation to generate the final class labels. A novel ‘channel attention module’ is designed to compute the attention weights, ensuring that features from the relevant extraction stages are boosted while the others are attenuated. The model also learns spatial context information at low resolution to preserve the abstract spatial relationships among scene elements and reduce computational cost. Spatial attention is subsequently concatenated into a final feature set before applying feature boosting. Low-resolution spatial attention features are trained using an auxiliary task that help to learn a coarse global scene structure. The proposed model outperforms all state-of-the-art models on both the ADE20K and the Cityscapes datasets.

GC-YOLOv9: Innovative smart city traffic monitoring solution

In urban smart city environments, traffic hazards can lead to catastrophic outcomes, including significant property losses and severe threats to public safety. Conventional traffic monitoring systems are limited in terms of accuracy and speed, presenting significant challenges for real-time traffic surveillance. To tackle these challenges, this paper introduces the GC-YOLOv9 algorithm. Specifically, we have enhanced the YOLOv9 model by incorporating Ghost Convolution, markedly improving the model’s perceptual abilities and detection accuracy. Furthermore, this study designed an integrated smart city framework that includes layers for service applications, the Internet of Things, edge processing, and data centers. By deploying the enhanced YOLOv9 model within this framework, our method achieved mAP@0.5 scores of 77.15 and 74.95 on the BDD100K and Cityscapes datasets, respectively, surpassing existing technologies. Additionally, the potential applications of this method in public area fire safety management, forest fire monitoring, and intelligent security systems further underscore its significant value in improving the safety and efficiency of smart cities.

Road Anomaly Detection with Unknown Scenes Using DifferNet-Based Automatic Labeling Segmentation

Obstacle avoidance is essential for the effective operation of autonomous mobile robots, enabling them to detect and navigate around obstacles in their environment. While deep learning provides significant benefits for autonomous navigation, it typically requires large, accurately labeled datasets, making the data’s preparation and processing time-consuming and labor-intensive. To address this challenge, this study introduces a transfer learning (TL)-based automatic labeling segmentation (ALS) framework. This framework utilizes a pretrained attention-based network, DifferNet, to efficiently perform semantic segmentation tasks on new, unlabeled datasets. DifferNet leverages prior knowledge from the Cityscapes dataset to identify high-entropy areas as road obstacles by analyzing differences between the input and resynthesized images. The resulting road anomaly map was refined using depth information to produce a robust drivable area and map of road anomalies. Several off-the-shelf RGB-D semantic segmentation neural networks were trained using pseudo-labels generated by the ALS framework, with validation conducted on the GMRPD dataset. Experimental results demonstrated that the proposed ALS framework achieved mean precision, mean recall, and mean intersection over union (IoU) rates of 80.31%, 84.42%, and 71.99%, respectively. The ALS framework, through the use of transfer learning and the DifferNet network, offers an efficient solution for semantic segmentation of new, unlabeled datasets, underscoring its potential for improving obstacle avoidance in autonomous mobile robots.

Evaluating the Effectiveness of Panoptic Segmentation Through Comparative Analysis

Image segmentation method is extensively used in the fields of computer vision, machine learning, and artificial intelligence. The task of segmentation is to distinguish objects in images either by their boundaries or as entire objects from the entire image. Image segmentation methods are implemented as instance, semantic, and panoptic segmentation. In this article, the panoptic segmentation method, seen as an advanced stage of instance and semantic segmentation, has been applied to three datasets and compared with the instance segmentation method. Experimental results are presented visually. Numerical results have been analyzed with the Panoptic Quality (PQ) and Semantic Quality (SQ) metrics. It has been observed that the segmentation outcome was best for the CityScapes dataset for panoptic segmentation.

LACTNet: A Lightweight Real-Time Semantic Segmentation Network Based on an Aggregated Convolutional Neural Network and Transformer

Transformers have demonstrated a significant advantage over CNNs in modeling long-range dependencies, leading to increasing attention being paid towards their application in semantic segmentation tasks. In the present work, a novel semantic segmentation model, LACTNet, is introduced, which synergistically combines Transformer and CNN architectures for the real-time processing of local and global contextual features. LACTNet is designed with a lightweight Transformer, which integrates a specially designed gated convolutional feedforward network, to establish feature dependencies across distant regions. A Lightweight Average Feature Bottleneck (LAFB) module is designed to effectively capture spatial detail information within the features, thereby enhancing segmentation accuracy. To address the issue of spatial feature loss in the decoder, a long skip-connection approach is employed through the designed Feature Fusion Enhancement Module (FFEM), which enhances the integrity of spatial features and the feature interaction capability in the decoder. LACTNet is evaluated on two datasets, achieving a segmentation accuracy of 74.8% mIoU and a frame rate of 90 FPS on the Cityscapes dataset, and a segmentation accuracy of 71.8% mIoU with a frame rate of 126 FPS on the CamVid dataset.

Depth-Aware Panoptic Segmentation

Abstract. Panoptic segmentation unifies semantic and instance segmentation and thus delivers a semantic class label and, for so-called thing classes, also an instance label per pixel. The differentiation of distinct objects of the same class with a similar appearance is particularly challenging and frequently causes such objects to be incorrectly assigned to a single instance. In the present work, we demonstrate that information on the 3D geometry of the observed scene can be used to mitigate this issue: We present a novel CNN-based method for panoptic segmentation which processes RGB images and depth maps given as input in separate network branches and fuses the resulting feature maps in a late fusion manner. Moreover, we propose a new depth-aware dice loss term which penalises the assignment of pixels to the same thing instance based on the difference between their associated distances to the camera. Experiments carried out on the Cityscapes dataset show that the proposed method reduces the number of objects that are erroneously merged into one thing instance and outperforms the method used as basis by +2.2% in terms of panoptic quality.

IIMT-net: Poly-1 weights balanced multi-task network for semantic segmentation and depth estimation using interactive information

Semantic segmentation and depth estimation are two basic researchable problems in computer vision. In common, we explore the two tasks separately. However, in some scenes, such as autonomous driving, they need be done at the same time. Meanwhile, there exists interconnected information between two tasks, which can jointly promote the performances of them. Thus, we explore the two tasks based on multi-task learning to jointly train the tasks and gain predictions together. In this paper, we build Interactive Information Multi-Task Network (IIMT-Net) incorporating the information interactive modules, trained with proposed task-balancing strategy. To be specific, we construct the principal part of encoder and decoder based on Transformer to well capture the global information. For better utilization of the task interaction between two tasks, we also add information fusion modules in two sub-decoders. In addition, the task-balancing strategy, Poly-1 weights, is designed as the balance among samples with different degrees of difficulty to ensure the network won't be biased towards any task severely. The proposed approach's exceptional performance has been extensively showcased through experimental results on the NYU Depth V2 dataset, the Cityscapes dataset, and the SUN RGB-D dataset. Our model can complete the predictions of semantic segmentation task and depth estimation task together and obtain mIoU values of 46.66% on the NYU Depth V2 dataset, 66.37% on the Cityscapes dataset, and 49.89% on the SUN RGB-D dataset, respectively with rmse values of 0.648, 6.630 and 0.401 for depth estimation task, which outperform most existing methods in multi-task learning.