Cityscapes Dataset Research Articles

A feature enhancement FCOS algorithm for dynamic traffic object detection

The development of object detection plays an important role in the realisation of fully autonomous driving, and the feature extraction is the key step for object detection. There has been significant difference and scale variation of object features for different road traffic participants (RTPs), meanwhile traditional Convolutional Neural Networks (CNNs) was difficult to extract object features efficiently for small targets. In order to improve the ability of feature extraction, a RTP object detection method combining dynamic convolution and feature enhancement was proposed. The Fully Convolutional One-Stage (FCOS) object detection algorithm was used as baseline. First, the dynamic convolution module was designed in the backbone network to identify different object features to the maximum extent. Second, a dual attention module was designed to filter object feature information while reducing the amount of computation. Finally, in the detection part, the feature expression ability of shallow network was further enhanced by multi-scale feature fusion module, and the effectiveness of the proposed algorithm was verified using Cityscapes dataset. The experimental result indicated that mAP increased by 2.3% compared with baseline. This study can improve the efficiency of RTP detection and contribute to the industrialisation of intelligent connected vehicles.

AM‐MulFSNet: A fast semantic segmentation network combining attention mechanism and multi‐branch

AbstractIn order to balance accuracy and real‐time performance in semantic segmentation, this paper proposes a real‐time semantic segmentation algorithm model based on attention mechanism and multi‐branch feature fusion using Fast convolutional neural network model (Fast‐SCNN). In this method, the spatial detail feature enhancement branch is introduced to enhance spatial detail features firstly. Then, through rational design of fusion module, the feature information of each branch is optimized to achieve better fusion of deep and shallow features. At the end of the feature fusion module, an adaptive feature enhancement focus module is introduced to capture the interdependence between remote pixels. The experimental results show that the proposed algorithm achieves 71.55% segmentation accuracy on Cityscapes dataset, the reasoning speed FPS is 97.6 frames/s, and the number of parameters is 1.39 M, which verifies the effectiveness of the network model constructed by the algorithm. Code is available at https://github.com/ccchhheeennn/model.

FLODCAST: Flow and depth forecasting via multimodal recurrent architectures

Forecasting motion and spatial positions of objects is of fundamental importance, especially in safety-critical settings such as autonomous driving. In this work, we address the issue by forecasting two different modalities that carry complementary information, namely optical flow and depth. To this end we propose FLODCAST a flow and depth forecasting model that leverages a multitask recurrent architecture, trained to jointly forecast both modalities at once. We stress the importance of training using flows and depth maps together, demonstrating that both tasks improve when the model is informed of the other modality. We train the proposed model to also perform predictions for several timesteps in the future. This provides better supervision and leads to more precise predictions, retaining the capability of the model to yield outputs autoregressively for any future time horizon. We test our model on the challenging Cityscapes dataset, obtaining state of the art results for both flow and depth forecasting. Thanks to the high quality of the generated flows, we also report benefits on the downstream task of segmentation forecasting, injecting our predictions in a flow-based mask-warping framework.

Explainer on GNN-based segmentation networks

Graph Neural Networks (GNN) are powerful tools for deep learning. Similar to other neural networks, GNNs are complex models, in which humans can’t understand the decision-making procedures of the models. Therefore, it brings the need to explainability of GNNs. Explainability is critical for deep learning to support its predictions. In this paper, we will investigate the Grad-Cam and Integrated-Gradients explaining methods. The Grad-Cam applies a global average pooling over the feature activation mapping, and then which was followed by a ReLU activation to obtain an attribution. The Integrated-Gradients explains models by taking a line integral between the baseline image (a black image) and the source image. We demonstrate how Grad-Cam and the Integrated-Gradients methods explain the graph-deep model in semantic segmentation tasks over the Cityscapes dataset. FCN and LRASSP-MobileNet are used as a comparison to the DualGCN in the experiment to show the explaining effect.

Exploring the spatial attributes of streets in Lu Xun’s hometown of Shaoxing, China, through image semantic segmentation

Image semantic segmentation, a deep learning algorithm, enables the recognition of pixel collections that form distinct categories, allowing for the identification of vehicles, pedestrians, traffic signs, pavement, and other road features. In urban and architectural design domains, image semantic segmentation and related techniques empower practitioners and researchers to efficiently analyze the distribution of public spaces. This application facilitates a better understanding of how people interact with urban environments, ultimately improving the design of functional and inviting spaces. This paper presents an analysis of images of different streets within the Lu Xun Heritage Area in Shaoxing, Zhejiang Province, China, which were obtained through onsite photography. The images were sampled, segmented, and compared to assess the spatial characteristics of distinct street types. A self-trained semantic segmentation model based on the Cityscapes dataset and the PaddlePaddle framework was employed to statistically analyze space variations across various dimensions. This analysis contributes to a better understanding of historical street structure and provides insights into the integration of artificial intelligence in urban planning and design.

ISTR: Mask-Embedding-Based Instance Segmentation Transformer.

Transformer-based instance-level recognition has attracted increasing research attention recently due to the superior performance. However, although attempts have been made to encode masks as embeddings into Transformer-based frameworks, how to combine mask embeddings and spatial information for a transformer-based approach is still not fully explored. In this paper, we revisit the design of mask-embedding-based pipelines and propose an Instance Segmentation TRansformer (ISTR) with Mask Meta-Embeddings (MME), leveraging the strengths of transformer models in encoding embedding information and incorporating spatial information from mask embeddings. ISTR incorporates a recurrent refining head that consists of a Dynamic Box Predictor (DBP), a Mask Information Generator (MIG), and a Mask Meta-Decoder (MMD). To improve the quality of mask embeddings, MME interprets the mask encoding-decoding processes as a mutual information maximization problem, which unifies the objective functions of different decoding schemes such as Principal Component Analysis (PCA) and Discrete Cosine Transform (DCT) with a meta-formulation. Under the meta-formulation, a learnable Spatial Mask Tuner (SMT) is further proposed, which fuses the spatial and embedding information produced from MIG and can significantly boost the segmentation performance. The resulting varieties, i.e., ISTR-PCA, ISTR-DCT, and ISTR-SMT, demonstrate the effectiveness and efficiency of incorporating mask embeddings with the query-based instance segmentation pipelines. On the COCO dataset, ISTR surpasses all predominant mask-embedding-based models by a large margin, and achieves competitive performance compared to concurrent state-of-the-art models. On the Cityscapes dataset, ISTR also outperforms several strong baselines. Our code has been made available at: https://github.com/hujiecpp/ISTR.

Stereo Superpixel Segmentation via Decoupled Dynamic Spatial-Embedding Fusion Network

Stereo superpixel segmentation aims at grouping the discretizing pixels into perceptual regions through left and right views more collaboratively and efficiently. Existing superpixel segmentation algorithms mostly utilize color and spatial features as input, which may impose strong constraints on spatial information while utilizing the disparity information in terms of stereo image pairs. To alleviate this issue, we propose a stereo superpixel segmentation method with a decoupling mechanism of spatial information in this work. To decouple stereo disparity information and spatial information, the spatial information is temporarily removed before fusing the features of stereo image pairs, and a decoupled stereo fusion module (DSFM) is designed to handle the stereo features alignment as well as occlusion problems. Moreover, since the spatial information is vital to superpixel segmentation, we further design a dynamic spatiality embedding module (DSEM) to re-add spatial information, and the weights of spatial information will be adaptively adjusted through the dynamic fusion (DF) mechanism in DSEM for achieving a finer segmentation. Comprehensive experimental results demonstrate that our method can achieve the state-of-the-art performance on the KITTI2015 and Cityscapes datasets, and also verify the efficiency when applied in salient object detection on NJU2K dataset. The source code will be available publicly after paper is accepted.

Improving Prediction Accuracy of Semantic Segmentation Methods Using Convolutional Autoencoder Based Pre-processing Layers

In this paper, we propose a method to improve prediction accuracy of semantic segmentation methods as follows: (1) construct a neural network that has pre-processing layers based on a convolutional autoencoder ahead of a semantic segmentation network, and (2) train the entire network initialized by the weights of the pre-trained autoencoder. We applied this method to the fully convolutional network (FCN) and experimentally compared its prediction accuracy on the cityscapes dataset. The Mean IoU of the proposed target model with the He normal initialization is 18.7% higher than that of FCN with the He normal initialization. In addition, those of the modified models of the target model are significantly higher than that of FCN with the He normal initialization. The accuracy and loss curves during the training showed that these are resulting from the improvement of the generalization ability. All of these results provide strong evidence that the proposed method is significantly effective in improving the prediction accuracy of FCN. The proposed method has the following features: it is comparatively simple, whereas the effect on improving the generalization ability and prediction accuracy of FCN is significant; the increase in the number of parameters by using it is very small, and that in the computation time is substantially large. In principle, the proposed method can be applied to other semantic segmentation methods. For semantic segmentation, at present, there is no effective way to improve the prediction accuracy of existing methods. None have published a method which is the same as or similar to our method and none have used such a method in practice. Therefore, we believe that our method is useful in practice and worthy of being widely known and used.

Fully Convolutional Network-Based Self-Supervised Learning for Semantic Segmentation.

Although deep learning has achieved great success in many computer vision tasks, its performance relies on the availability of large datasets with densely annotated samples. Such datasets are difficult and expensive to obtain. In this article, we focus on the problem of learning representation from unlabeled data for semantic segmentation. Inspired by two patch-based methods, we develop a novel self-supervised learning framework by formulating the jigsaw puzzle problem as a patch-wise classification problem and solving it with a fully convolutional network. By learning to solve a jigsaw puzzle comprising 25 patches and transferring the learned features to semantic segmentation task, we achieve a 5.8% point improvement on the Cityscapes dataset over the baseline model initialized from random values. It is noted that we use only about 1/6 training images of Cityscapes in our experiment, which is designed to imitate the real cases where fully annotated images are usually limited to a small number. We also show that our self-supervised learning method can be applied to different datasets and models. In particular, we achieved competitive performance with the state-of-the-art methods on the PASCAL VOC2012 dataset using significantly fewer time costs on pretraining.

Real-Time Semantic Segmentation via Spatial-Detail Guided Context Propagation.

Nowadays, vision-based computing tasks play an important role in various real-world applications. However, many vision computing tasks, e.g., semantic segmentation, are usually computationally expensive, posing a challenge to the computing systems that are resource-constrained but require fast response speed. Therefore, it is valuable to develop accurate and real-time vision processing models that only require limited computational resources. To this end, we propose the spatial-detail guided context propagation network (SGCPNet) for achieving real-time semantic segmentation. In SGCPNet, we propose the strategy of spatial-detail guided context propagation. It uses the spatial details of shallow layers to guide the propagation of the low-resolution global contexts, in which the lost spatial information can be effectively reconstructed. In this way, the need for maintaining high-resolution features along the network is freed, therefore largely improving the model efficiency. On the other hand, due to the effective reconstruction of spatial details, the segmentation accuracy can be still preserved. In the experiments, we validate the effectiveness and efficiency of the proposed SGCPNet model. On the Cityscapes dataset, for example, our SGCPNet achieves 69.5% mIoU segmentation accuracy, while its speed reaches 178.5 FPS on 768 x 1536 images on a GeForce GTX 1080 Ti GPU card. In addition, SGCPNet is very lightweight and only contains 0.61 M parameters. The code will be released at https://github.com/zhouyuan888888/SGCPNet.

Boosting Neural Image Compression for Machines Using Latent Space Masking

Today, many image coding scenarios do not have a human as final intended user, but rather a machine fulfilling computer vision tasks on the decoded image. Thereby, the primary goal is not to keep visual quality but maintain the task accuracy of the machine for a given bitrate. Due to the tremendous progress of deep neural networks setting benchmarking results, mostly neural networks are employed to solve the analysis tasks at the decoder side. Moreover, neural networks have also found their way into the field of image compression recently. These two developments allow for an end-to-end training of the neural compression network for an analysis network as information sink. Therefore, we first roll out such a training with a task-specific loss to enhance the coding performance of neural compression networks. Compared to the standard VVC, 41.4% of bitrate are saved by this method for Mask R-CNN as analysis network on the uncompressed Cityscapes dataset. As a main contribution, we propose LSMnet, a network that runs in parallel to the encoder network and masks out elements of the latent space that are presumably not required for the analysis network. By this approach, additional 27.3% of bitrate are saved compared to the basic neural compression network optimized with the task loss. In addition, we are the first to utilize a feature-based distortion in the training loss within the context of machine-to-machine communication, which allows for a training without annotated data. We provide extensive analyses on the Cityscapes dataset including cross-evaluation with different analysis networks and present exemplary visual results. Inference code and pre-trained models are published at https://github.com/FAU-LMS/NCN_for_M2M.

IterDepth: Iterative Residual Refinement for Outdoor Self-Supervised Multi-Frame Monocular Depth Estimation

Self-supervised monocular depth estimation has been a challenging task in computer vision for a long time, and it relies on only monocular or stereo video for its supervision. To address the challenge, we propose a novel multi-frame monocular depth estimation method called IterDepth, which is based on an iterative residual refinement network. IterDepth extracts depth features from consecutive frames and computes a 3D cost volume measuring the difference between current and previous features transformed by PoseCNN (pose estimation convolutional neural network). We reformulate depth prediction as a residual learning problem, revamping the dominating depth regression to enable high-accuracy multi-frame monocular depth estimation. Specifically, we design a gated recurrent depth fusion unit that seamlessly blends depth features from the cost volume, image features, and the depth prediction. The unit updates the hidden states and refines the depth map through iterative refinement, achieving more accurate predictions than existing methods. Our experiments on the KITTI dataset demonstrate that IterDepth is 7× faster in terms of FPS (frames per second) than the recent state-of-the-art DepthFormer model with competitive performance. We also test IterDepth on the Cityscapes dataset to showcase its generalization capability in other real-world environments. Moreover, IterDepth can balance accuracy and computational efficiency by adjusting the number of refinement iterations and performs competitively with other CNN-based monocular depth estimation approaches. Code will be made publicly available.

Binocular Image Dehazing via a Plain Network Without Disparity Estimation

Heavy haze leads to severely degraded visual quality for images, and thus the performance of high level image-based tasks such as object detection and semantic segmentation is deteriorated. It is necessary and important to design an effective dehazing method for the computer vision system. It is well known that image haze is a function of depth and binocular images can predict the depth. Existing binocular dehazing methods conduct disparity estimation and dehazing jointly to enhance each other. However, a small error in disparity gives rise to a large variation in depth and in the estimation of haze-free images. To alleviate the problem, we propose a plain binocular image dehazing network in this paper, called BidNet, to dehaze both the left and right images simultaneously. BidNet does not explicitly perform disparity estimation that is time-consuming and well-known to be challenging. Instead, we design a stereo transformation module to mine the relationship and correlation between binocular images, making the best of varying information of cross views. Additionally, we design a Stereo Foggy Cityscapes dataset extended from the Foggy Cityscapes dataset for training the proposed BidNet. Extensive experimental results demonstrate that BidNet significantly outperforms the SOTA dehazing methods on the synthetic stereo foggy datasets as well as in real stereo foggy scenes. Experimental results show that jointly dehazing binocular image pairs is mutually beneficial, which is better than only dehazing left images. Furthermore, when applying BidNet to preprocess foggy inputs, large improvements are obtained in the performance of object detection, instance segmentation, semantic segmentation, and stereo-based 3D object detection.

Quantifying morphologies of developing neuronal cells using deep learning with imperfect annotations

The functionality of human intelligence relies on the interaction and health of neurons, hence, quantifying neuronal morphologies can be crucial for investigating the functionality of the human brain. This paper proposes a deep learning (DL) based method for segmenting and quantifying neuronal structures in fluorescence microscopy images of developing neuronal cells cultured in vitro. Compared to the majority of supervised DL-based segmentation methods that heavily rely on creating exact corresponding masks of neuronal structures for the preparation of training samples, the proposed approach allows for imperfect annotation of neurons, as it only requires tracing the centrelines of the neurites. This ability accelerates the preparation of training data by several folds. Our proposed framework is built on a modified version of PSPNet with an EfficientNet backbone pre-trained on the CityScapes dataset. To handle the imperfectness of training samples, we incorporated a weighted combination of two loss functions, namely the Dice loss and Lovász loss functions, into our network. We evaluated the proposed framework and several other state-of-the-art methods on a published dataset of approximately 900 manually quantified cultured mouse neurons. Our results indicate a close correlation between the proposed method and manual quantification in terms of neuron length and the number of branches while demonstrating improved analysis speed. Furthermore, the proposed method achieved high accuracy in neuron segmentation, as evidenced by the evaluation of the neurons’ length and number of branches.

Image Recognition Technology Applied to the Design of Mobile Platform for Warehouse Logistics Robots

Abstract This paper first studies the processing flow of image processing technology that preprocesses the image and adopts the method of polygonal approximation to identify the shape and localize the moving target. Then, the mobile platform of the warehouse logistics robot is designed. Then, the vision system of the robot was designed using image recognition technology to realize obstacle collision prediction and route planning. Finally, the robot’s localization and grasping abilities, trajectory following performance, and semantic segmentation abilities are analyzed using comparative experiments. The successful localization and grasping rates of the warehouse robots are all higher than 93%, and the trajectory following the straight line road section is better, with a maximum error of less than 21 mm. The mIoU of this paper’s method on the Cityscapes dataset is 78.85%, MPA is 86.05%, and PA is 96.89%, with good image segmentation performance. This study is of great significance for the development of the intelligent logistics field.

MFSNet: Enhancing Semantic Segmentation of Urban Scenes with a Multi-Scale Feature Shuffle Network

The complexity of urban scenes presents a challenge for semantic segmentation models. Existing models are constrained by factors such as the scale, color, and shape of urban objects, which limit their ability to achieve more accurate segmentation results. To address these limitations, this paper proposes a novel Multi-Scale Feature Shuffle NetWork (MFSNet), which is an improvement upon the existing Deeplabv3+ model. Specifically, MFSNet integrates a novel Pyramid Shuffle Module (PSM) to extract discriminative features and feature correlations, with the objective of improving the accuracy of classifying insignificant objects. Additionally, we propose an efficient feature aggregation module (EFAM) to effectively expand the receptive field and aggregate contextual information, which is integrated as a branch within the network architecture to mitigate the information loss resulting from downsampling operations. Moreover, in order to augment the precision of segmentation boundary delineation and object localization, we employ a progressive upsampling strategy for reinstating spatial information in the feature maps. The experimental results show that the proposed model achieves competitive performance, achieving 80.4% MIoU on the Pascal VOC 2012 dataset, 79.4% MIoU on the Cityscapes dataset, and 40.1% MIoU on the Coco-Stuff dataset.

Dynamic context modeling based lightweight high-resolution network for dense prediction

Recent research shows that high-resolution networks can provide representative multi-scale features for vision tasks. However, high-resolution-based architectures are weak in capturing spatial long-range information, and they are computationally expensive. To alleviate these problems and encourage the engineering applications of dense prediction, based on vanilla high-resolution network, this paper presents a novel efficient architecture for dense prediction, namely Dynamic context modeling based lightweight High-Resolution Network (Dynamic-HRNet). In particular, the network consists of two key components: (i) dynamic split convolution, a more efficient and flexible convolution compared to the standard convolution, which could be conveniently embedded into other networks, and (ii) adaptive context modeling, which adaptively captures local and global contextual information to enrich the representation in parallel. Using the above two components, two lightweight blocks are designed that serve as the basic building units of Dynamic-HRNet, termed dynamic multi-scale context block and dynamic global context block, respectively. Extensive experiments demonstrate that the proposed Dynamic-HRNet, as a backbone, achieves significant superiority in popular benchmarks for human pose estimation (70.6% AP on COCO dataset and 87.6% PCKh on MPII dataset) and semantic segmentation (75.3% MIoU on Cityscapes dataset). Considering both efficiency and accuracy, it outperforms the most advanced lightweight architectures.

HexNet: An Orientation-Aware Deep Learning Framework for Omni-Directional Input.

While omni-directional sensors provide holistic representations typical deep learning frameworks reduce the benefits by introducing distortions and discontinuities as spherical data is supplied as planar input. On the other hand, recent spherical convolutional neural networks (CNNs) often require significant memory and parameters, thus enabling execution only at very low resolutions and shallow architectures. We propose HexNet, an orientation-aware deep learning framework for spherical signals, that allows for fast computation as we exploit standard planar network operations on an efficiently arranged projection of the sphere. Furthermore, we introduce a graph-based version for partial spheres, allowing us to compete at high-resolution with planar CNNs using residual network architectures. Our kernels operate on the tangent of the sphere and thus standard feature weights, pretrained on perspective data, can be transferred, enabling spherical pretraining on ImageNet. As our design is free of distortions and discontinuity, our orientation-aware CNN becomes a new state of the art for semantic segmentation on the recent 2D3DS dataset, and the omni-directional version of SYNTHIA introduced in this work. Moreover, we experimentally show the benefit of our spherical representation over standard images on the Cityscapes dataset by reducing distortion effects of planar CNNs. We implement object detection for the spherical domain. Rotation invariant classification and segmentation tasks are additionally presented for comparison to prior art.

A hybrid Cycle GAN-based lightweight road perception pipeline for road dataset generation for Urban mobility.

One of the major problems that cause continual trouble in deep learning networks is that training a large network requires massive labelled datasets. The preparation of a massive labelled dataset is a cumbersome task and requires lot of human interventions. This paper proposes a novel generator network 'Sim2Real' transfer is a recent and fast-developing field in machine learning used to bridge the gap between simulated and real data. Training with simulated datasets often converges due to its size but fails to generalize real-world applications. Simulated datasets can be used to train and test deep learning models, enables the development and evaluation of new algorithms and architectures. By simulating road dataset, researchers can generate large amounts of realistic road-traffic dataset that can be used to study and understand several problems such as vehicular object tracking and classification, traffic situation analysis etc. The main advantage of such a transfer algorithm is to use the abundance of a simulated dataset to generate huge realistic-looking datasets to solve data-intense tasks. This work presents a novel, robust sim2real algorithm that converts the labels of a semantic segmentation map to a realistic-looking street view using the Cityscapes dataset and aims to achieve robust urban mobility for smart cities. Further, the generalizability of the Cycle Generative Adversarial Network (CycleGAN) architecture was tested by using an origami robot dataset for sim2real transfer. We show that the results were found to be qualitatively satisfactory for different traffic analysis applications. In addition, road perception was done using a lightweight SVM pipeline and evaluated on the KITTI dataset. We have incorporated Cycle Consistency Loss and Identity Loss as the metrics to evaluate the performance of the proposed Cycle GAN model. We inferred that the proposed Cycle GAN model provides an Identity loss of less than 0.2 in both the Cityscapes dataset and KITTI datasets. Also, we understand that the super-pixel resolution has a good impact on the quantitative results of the proposed Cycle GAN models.

A lightweight vision transformer with symmetric modules for vision tasks

Transformer-based networks have demonstrated their powerful performance in various vision tasks. However, these transformer-based networks are heavyweight and cannot be applied to edge computing (mobile) devices. Despite that the lightweight transformer network has emerged, several problems remain, i.e., weak feature extraction ability, feature redundancy, and lack of convolutional inductive bias. To address these three problems, we propose a lightweight visual transformer (Symmetric Former, SFormer), which contains two novel modules (Symmetric Block and Symmetric FFN). Specifically, we design Symmetric Block to expand feature capacity inside the module and enhance the long-range modeling capability of attention mechanism. To increase the compactness of the model and introduce inductive bias, we introduce convolutional cheap operations to design Symmetric FFN. We compared the SFormer with existing lightweight transformers on several vision tasks. Remarkably, on the image recognition task of ImageNet [13], SFormer gains 1.2% and 1.6% accuracy improvements compared to PVTv2-b0 and Swin Transformer, respectively. On the semantic segmentation task of ADE20K [64], SFormer delivers performance improvements of 0.2% and 0.7% compared to PVTv2-b0 and Swin Transformer, respectively. On the cityscapes dataset [11], SFormer delivers performance improvements of 2.5% and 4.2% compared to PVTv2-b0 and Swin Transformer, respectively. The code is open-source and available at: https://github.com/ISCLab-Bistu/Symmetric_Former.git.