Semantic Segmentation Map Research Articles

Architectural Space Recognition from Blueprints Using Machine Learning-Based Semantic Segmentation

Architectural blueprints have been providing fundamental processes for urban planning, construction, and interior design for professionals. It contains a visual representation of building plans and specifications in the construction and design industry. However, for the non-expert, it is difficult to understand these complex technical drawings. This is because of the specialized language, symbols, and technical knowledge required to understand architectural blueprints. This challenge often leads to misunderstandings which can extend the construction process and escalate costs. To bridge this knowledge gap, in this research paper, I propose a machine-based blueprint interpretation method using semantic segmentation. The proposed method takes blueprints as input and generates semantic segmentation maps. These maps categorize and isolate distinct architectural areas into predefined classes, including rooms, kitchens, and bathrooms. The proposed machine learning model is tested on a publicly available dataset of architectural blueprints. Through comprehensive quantitative and qualitative assessments, it is shown that the proposed method achieves state-of-the-art performance.

A Unified Framework for Jointly Compressing Visual and Semantic Data

The rapid advancement of multimedia and imaging technologies has resulted in increasingly diverse visual and semantic data. A large range of applications such as remote-assisted driving requires the amalgamated storage and transmission of various visual and semantic data. However, existing works suffer from the limitation of insufficiently exploiting the redundancy between different types of data. In this article, we propose a unified framework to jointly compress a diverse spectrum of visual and semantic data, including images, point clouds, segmentation maps, object attributes, and relations. We develop a unifying process that embeds the representations of these data into a joint embedding graph according to their categories, which enables flexible handling of joint compression tasks for various visual and semantic data. To fully leverage the redundancy between different data types, we further introduce an embedding-based adaptive joint encoding process and a Semantic Adaptation Module to efficiently encode diverse data based on the learned embeddings in the joint embedding graph. Experiments on the Cityscapes, MSCOCO, and KITTI datasets demonstrate the superiority of our framework, highlighting promising steps toward scalable multimedia processing.

A three in one bottom-up framework for simultaneous semantic segmentation, instance segmentation and classification of multi-organ nuclei in digital cancer histology

Simultaneous segmentation and classification of nuclei in digital histology remains challenging. The highest achieved Panoptic Quality (PQ) remains low due to overlapping nuclei, higher staining and tissue variability, and rough clinical conditions. The generic deep-learning methods usually rely on end-to-end models, which fail to address these problems associated explicitly with digital histology. We resolve these issues using a dual attention-based model combined with post-processing in a bottom-up fashion. We use three attention decoder heads, which produce semantic segmentation, edge proposals, and classification maps. We use these outputs to apply post-processing, including controlled watershed and pixel grouping, to produce instance segmentation and classification. Our multi-stage approach utilizes edge proposals and semantic segmentations compared to direct segmentation and classification strategies followed by most generic state-of-the-art methods. Due to this, we demonstrate a significant performance improvement in producing high-quality instance segmentation and nuclei classification. We have achieved a 0.841 Dice score for semantic segmentation, 0.713 bPQ scores for instance segmentation, and 0.633 mPQ for nuclei classification. Furthermore, the framework is less complex compared to the state-of-the-art.

Estimating daily semantic segmentation maps of classified ocean eddies using sea level anomaly data from along-track altimetry.

Mesoscale eddies, which are fast-moving rotating water bodies in the ocean with horizontal scales ranging from 10 km to 100 km and above, are considered to be the weather of the oceans. They are of interest to marine biologists, oceanographers, and geodesists for their impact on water mass, heat, and nutrient transport. Typically, gridded sea level anomaly maps processed from multiple radar altimetry missions are used to detect eddies. However, multi-mission sea level anomaly maps obtained by the operational processors have a lower effective spatiotemporal resolution than their grid spacing and temporal resolution, leading to inaccurate eddy detection. In this study, we investigate the use of higher-resolution along-track sea level anomaly data to infer daily two-dimensional segmentation maps of cyclonic, anticyclonic, or non-eddy areas with greater accuracy than using processed sea level anomaly grid map products. To tackle this challenge, we propose a deep neural network that uses spatiotemporal contextual information within the modality of along-track data. This network is capable of producing a two-dimensional segmentation map from data with varying sparsity. We have developed an architecture called Teddy, which uses a Transformer module to encode and process spatiotemporal information, and a sparsity invariant CNN to infer a two-dimensional segmentation map of classified eddies from the ground tracks of varying sparsity on the considered region. Our results show that Teddy creates two-dimensional maps of classified eddies from along-track data with higher accuracy and timeliness when compared to commonly used methods that work with less accurate preprocessed sea level anomaly grid maps. We train and test our method with a carefully curated and independent dataset, which can be made available upon request.

Semantic and Geometric-Aware Day-to-Night Image Translation Network.

Autonomous driving systems heavily depend on perception tasks for optimal performance. However, the prevailing datasets are primarily focused on scenarios with clear visibility (i.e., sunny and daytime). This concentration poses challenges in training deep-learning-based perception models for environments with adverse conditions (e.g., rainy and nighttime). In this paper, we propose an unsupervised network designed for the translation of images from day-to-night to solve the ill-posed problem of learning the mapping between domains with unpaired data. The proposed method involves extracting both semantic and geometric information from input images in the form of attention maps. We assume that the multi-task network can extract semantic and geometric information during the estimation of semantic segmentation and depth maps, respectively. The image-to-image translation network integrates the two distinct types of extracted information, employing them as spatial attention maps. We compare our method with related works both qualitatively and quantitatively. The proposed method shows both qualitative and qualitative improvements in visual presentation over related work.

Semantic surrounding projection for object height estimation using single-line lidar and image

Indoor environment perception requires surroundings to facilitate the path planning and object management using three dimensional lidar or visual sensors,which addresses the need for accurate height estimation in tasks like estimating cabinet height for grabbing things and judging passageway passing height. The proposed method combines single-line lidar and image to extract features, such as line segmentation, objects detection and semantic segmentation map, using neural network. Height estimation is achieved by referencing single image with single-line lidar data. The multi-source data approach enhances the accuracy especially in the areas where data from a single sensor are unreliable. A set of analysis of errors and uncertainties introduced by the method are conducted to facilitate the indoor perception system through the integration of muti-source data and neural network-base feature extraction for height estimation, and results demonstrate the performance and efficiency of the method.

A hybrid Cycle GAN-based lightweight road perception pipeline for road dataset generation for Urban mobility.

One of the major problems that cause continual trouble in deep learning networks is that training a large network requires massive labelled datasets. The preparation of a massive labelled dataset is a cumbersome task and requires lot of human interventions. This paper proposes a novel generator network 'Sim2Real' transfer is a recent and fast-developing field in machine learning used to bridge the gap between simulated and real data. Training with simulated datasets often converges due to its size but fails to generalize real-world applications. Simulated datasets can be used to train and test deep learning models, enables the development and evaluation of new algorithms and architectures. By simulating road dataset, researchers can generate large amounts of realistic road-traffic dataset that can be used to study and understand several problems such as vehicular object tracking and classification, traffic situation analysis etc. The main advantage of such a transfer algorithm is to use the abundance of a simulated dataset to generate huge realistic-looking datasets to solve data-intense tasks. This work presents a novel, robust sim2real algorithm that converts the labels of a semantic segmentation map to a realistic-looking street view using the Cityscapes dataset and aims to achieve robust urban mobility for smart cities. Further, the generalizability of the Cycle Generative Adversarial Network (CycleGAN) architecture was tested by using an origami robot dataset for sim2real transfer. We show that the results were found to be qualitatively satisfactory for different traffic analysis applications. In addition, road perception was done using a lightweight SVM pipeline and evaluated on the KITTI dataset. We have incorporated Cycle Consistency Loss and Identity Loss as the metrics to evaluate the performance of the proposed Cycle GAN model. We inferred that the proposed Cycle GAN model provides an Identity loss of less than 0.2 in both the Cityscapes dataset and KITTI datasets. Also, we understand that the super-pixel resolution has a good impact on the quantitative results of the proposed Cycle GAN models.

UPDExplainer: An interpretable transformer-based framework for urban physical disorder detection using street view imagery

Urban Physical Disorder (UPD), such as old or abandoned buildings, broken sidewalks, litter, and graffiti, has a negative impact on residents’ quality of life. They can also increase crime rates, cause social disorder, and pose a public health risk. Currently, there is a lack of efficient and reliable methods for detecting and understanding UPD. To bridge this gap, we propose UPDExplainer, an interpretable transformer-based framework for UPD detection. We first develop a UPD detection model based on the Swin Transformer architecture, which leverages readily accessible street view images to learn discriminative representations. To provide clear and comprehensible evidence and analysis, we subsequently introduce a UPD factor identification and ranking module that combines visual explanation maps with semantic segmentation maps. This novel integrated approach enables us to identify the exact objects within street view images that are responsible for physical disorders and gain insight into the underlying causes. Experimental results on the re-annotated Place Pulse 2.0 dataset demonstrate promising detection performance of the proposed method, surpassing the state-of-the-art with an increase of 6.54% in Accuracy and 7.05% in F1 score. For a comprehensive evaluation of the method’s ranking performance, we provide both quantitative and qualitative results, highlighting the model’s precise identification of one, two, three, and four UPD factors, respectively. We also present a case study of detecting and ranking physical disorders in the southern region of downtown Los Angeles, California, to demonstrate the practicality and effectiveness of our framework.

Point Cloud Scene Completion with Joint Color and Semantic Estimation from Single RGB-D Image.

We present a deep reinforcement learning method of progressive view inpainting for colored semantic point cloud scene completion under volume guidance, achieving high-quality scene reconstruction from only a single RGB-D image with severe occlusion. Our approach is end-to-end, consisting of three modules: 3D scene volume reconstruction, 2D RGB-D and segmentation image inpainting, and multi-view selection for completion. Given a single RGB-D image, our method first predicts its semantic segmentation map and goes through the 3D volume branch to obtain a volumetric scene reconstruction as a guide to the next view inpainting step, which attempts to make up the missing information; the third step involves projecting the volume under the same view of the input, concatenating them to complete the current view RGB-D and segmentation map, and integrating all RGB-D and segmentation maps into the point cloud. Since the occluded areas are unavailable, we resort to a A3C network to glance around and pick the next best view for large hole completion progressively until a scene is adequately reconstructed while guaranteeing validity. All steps are learned jointly to achieve robust and consistent results. We perform qualitative and quantitative evaluations with extensive experiments on the 3D-FUTURE data, obtaining better results than state-of-the-arts.

SeTGAN: Semantic‐text guided face image generation

AbstractIn this article, we propose a method to jointly control face image generation through semantic segmentation maps and text. Existing semantic segmentation maps lack detailed face attributes such as beards, and it is difficult to explicitly represent the gender of the target person by virtue of the semantic maps. State‐of‐the‐art face image generation methods guided by semantic segmentation maps mostly solved this by introducing the original image for supervision, which cannot accurately control the detailed attributes of the target face. At the same time, the text‐guided image generation method perform poorly in controlling the front and side of the face pose. Therefore, we propose an idea that the semantic segmentation map controls the coarse content of the target image and the text controls the fine details of the target image. Through the well‐designed mapping network and content mixing mechanism, the model in this article flexibly draws on the advantages of the two modes, and can generate high‐resolution images that are diverse, high‐quality, and more faithful to the target in detail attributes than most of previous methods. Extensive experiments demonstrate the superior performance of the proposed method in terms of accuracy and fidelity.

A multi‐stage model for bird's eye view prediction based on stereo‐matching model and RGB‐D semantic segmentation

AbstractBird's‐Eye‐View (BEV) map is a powerful and detailed scene representation for intelligent vehicles that provides both the location and semantic information about nearby objects from a top‐down perspective. BEV map generation is a complex multi‐stage task, and the existing methods typically perform poorly for distant scenes. Thus, the authors introduce a novel multi‐stage model that infers to obtain more accurate BEV map. First, the authors propose the Adaptive Aggregation with Stereo Mixture Density (AA‐SMD) model, which is an improved stereo matching model that eliminates bleeding artefacts and provides more accurate depth estimation. Next, the authors employ the RGB‐Depth (RGB‐D) semantic segmentation model to improve the semantic segmentation performance and connectivity of their model. The depth map and semantic segmentation maps are then combined to create an incomplete BEV map. Finally, the authors propose a Multi Strip Pooling Unet (MSP‐Unet) model with a hierarchical multi‐scale (HMS) attention and strip pooling (SP) module to improve prediction with BEV generation. The authors evaluate their model with a Car Learn to Act (CARLA)‐generated synthetic dataset. The experiment results demonstrate that the authors’ model generates a highly accurate representation of the surrounding environment achieving a state‐of‐the‐art result of 61.50% Mean Intersection‐over‐Union (MIoU) across eight classes.

3D layout estimation of general rooms based on ordinal semantic segmentation

AbstractRoom layout estimation aims to predict the location and range of layout planes of interior spaces. Previous works treat each layout plane as an independent individual without considering the ordinal relation between walls, resulting the loss of the wall planes and the lack of integrity. This paper proposes a novel two‐branch neural networks model to estimate 3D layouts of cuboid and non‐cuboid room types. The model embeds the ordinal relation between layout planes into the layout segmentation branch through an proposed ordinal classification loss function, and outputs both pixel‐level layout segmentation maps and layout plane parameter maps. Then, the instance‐level plane parameters of each layout plane are determined by using an instance‐aware pooling layer. Finally, the sharpness of layout edges of the 2D layout semantic segmentation map is optimized by using an improved depth map intersection algorithm. Furthermore, we annotate a large‐scale 3D room layout estimation dataset, InteriorNet‐Layout, to obtain a steady model. Experiments on synthesized real‐world datasets show that the proposed method achieves faster calculation while maintaining high accuracy. Code is available at https://github.com/Hui‐Yao/3D‐ordinal‐layout‐estimation.

U-shape Transformer for Underwater Image Enhancement.

The light absorption and scattering of underwater impurities lead to poor underwater imaging quality. The existing data-driven based underwater image enhancement (UIE) techniques suffer from the lack of a large-scale dataset containing various underwater scenes and high-fidelity reference images. Besides, the inconsistent attenuation in different color channels and space areas is not fully considered for boosted enhancement. In this work, we built a large scale underwater image (LSUI) dataset, which covers more abundant underwater scenes and better visual quality reference images than existing underwater datasets. The dataset contains 4279 real-world underwater image groups, in which each raw image's clear reference images, semantic segmentation map and medium transmission map are paired correspondingly. We also reported an U-shape Transformer network where the transformer model is for the first time introduced to the UIE task. The U-shape Transformer is integrated with a channel-wise multi-scale feature fusion transformer (CMSFFT) module and a spatial-wise global feature modeling transformer (SGFMT) module specially designed for UIE task, which reinforce the network's attention to the color channels and space areas with more serious attenuation. Meanwhile, in order to further improve the contrast and saturation, a novel loss function combining RGB, LAB and LCH color spaces is designed following the human vision principle. The extensive experiments on available datasets validate the state-of-the-art performance of the reported technique with more than 2dB superiority. The dataset and demo code are available on https://bianlab.github.io/.

Self-Ensembling GAN for Cross-Domain Semantic Segmentation

Deep neural networks (DNNs) have greatly contributed to the performance gains in semantic segmentation. Nevertheless, training DNNs generally requires large amounts of pixel-level labeled data, which is expensive and time-consuming to collect in practice. To mitigate the annotation burden, this paper proposes a self-ensembling generative adversarial network (SE-GAN) exploiting cross-domain data for semantic segmentation. In SE-GAN, a teacher network and a student network constitute a self-ensembling model for generating semantic segmentation maps, which together with a discriminator, forms a GAN. Despite its simplicity, we find SE-GAN can significantly boost the performance of adversarial training and enhance the stability of the model, the latter of which is a common barrier shared by most adversarial training-based methods. We theoretically analyze SE-GAN and provide an $\mathcal O(1/\sqrt{N})$ generalization bound ($N$ is the training sample size), which suggests controlling the discriminator's hypothesis complexity to enhance the generalizability. Accordingly, we choose a simple network as the discriminator. Extensive and systematic experiments in two standard settings demonstrate that the proposed method significantly outperforms current state-of-the-art approaches. The source code of our model is available online (https://github.com/YonghaoXu/SE-GAN).

Wireless End-to-End Image Transmission System Using Semantic Communications

Semantic communication is considered the future of mobile communication, which aims to transmit data beyond Shannon’s theorem of communications by transmitting the semantic meaning of the data rather than the bit-by-bit reconstruction of the data at the receiver’s end. The semantic communication paradigm aims to bridge the gap of limited bandwidth problems in modern high-volume multimedia application content transmission. Integrating AI technologies with the 6G communications networks paved the way to develop semantic communication-based end-to-end communication systems. In this study, we have implemented a semantic communication-based end-to-end image transmission system, and we discuss potential design considerations in developing semantic communication systems in conjunction with physical channel characteristics. A Pre-trained GAN network is used at the receiver as the transmission task to reconstruct the realistic image based on the Semantic segmented image at the receiver input. The semantic segmentation task at the transmitter (encoder) and the GAN network at the receiver (decoder) is trained on a common knowledge base, the COCO-Stuff dataset. The research shows that the resource gain in the form of bandwidth saving is immense when transmitting the semantic segmentation map through the physical channel instead of the ground truth image in contrast to conventional communication systems. Furthermore, the research studies the effect of physical channel distortions and quantization noise on semantic communication-based multimedia content transmission.

A Deep Detection Model based on Multi-task Learning for Appearance Defect of Solid Propellants

Solid propellants (SPs), as a high-energy material, are commonly used in military and industrial power systems, such as solid rocket and missiles. The SPs, however, confronts severe difficulties of inevitable defects while being made, thus bringing about the significance of inspection. However, the literatures before typically tackled this problem separately, which subsequently combines different models for the variety of defect patterns. Despite of the effectiveness, this act of matters usually brings excessive complexity and additionally computational burden. In this article, we managed to solve this problem in an integrated framework, which unite both the size detection task and shape detection task at the same time, but with different training strategies. To be specific, our framework is mostly consisted of two stage. Firstly, the SPs region is output using a semantic segmentation network, and size measurements are completed with traditional image processing to determine the size defects of the SPs. Then, the depth features of the segmentation network are combined with the semantic segmentation map to make a spatial attention mechanism, which is input to the deep classifier to complete the shape defect detection. The focus of model is gradually shifted from the segmentation task to the classification task as the number of training sessions increases by introducing dynamic balancing factors. The experimental results show that the multi-task learning approach can greatly improve the generalization and robustness of the model, and the accuracy and speed are improved for appearance defect detection of SPs.

DESC: Domain Adaptation for Depth Estimation via Semantic Consistency

Accurate real depth annotations are difficult to acquire, needing the use of special devices such as a LiDAR sensor. Self-supervised methods try to overcome this problem by processing video or stereo sequences, which may not always be available. Instead, in this paper, we propose a domain adaptation approach to train a monocular depth estimation model using a fully-annotated source dataset and a non-annotated target dataset. We bridge the domain gap by leveraging semantic predictions and low-level edge features to provide guidance for the target domain. We enforce consistency between the main model and a second model trained with semantic segmentation and edge maps, and introduce priors in the form of instance heights. Our approach is evaluated on standard domain adaptation benchmarks for monocular depth estimation and show consistent improvement upon the state-of-the-art. Code available at https://github.com/alopezgit/DESC.

White blood cell detection, classification and analysis using phase imaging with computational specificity (PICS)

Treatment of blood smears with Wright’s stain is one of the most helpful tools in detecting white blood cell abnormalities. However, to diagnose leukocyte disorders, a clinical pathologist must perform a tedious, manual process of locating and identifying individual cells. Furthermore, the staining procedure requires considerable preparation time and clinical infrastructure, which is incompatible with point-of-care diagnosis. Thus, rapid and automated evaluations of unlabeled blood smears are highly desirable. In this study, we used color spatial light interference microcopy (cSLIM), a highly sensitive quantitative phase imaging (QPI) technique, coupled with deep learning tools, to localize, classify and segment white blood cells (WBCs) in blood smears. The concept of combining QPI label-free data with AI for the purpose of extracting cellular specificity has recently been introduced in the context of fluorescence imaging as phase imaging with computational specificity (PICS). We employed AI models to first translate SLIM images into brightfield micrographs, then ran parallel tasks of locating and labelling cells using EfficientNet, which is an object detection model. Next, WBC binary masks were created using U-net, a convolutional neural network that performs precise segmentation. After training on digitally stained brightfield images of blood smears with WBCs, we achieved a mean average precision of 75% for localizing and classifying neutrophils, eosinophils, lymphocytes, and monocytes, and an average pixel-wise majority-voting F1 score of 80% for determining the cell class from semantic segmentation maps. Therefore, PICS renders and analyzes synthetically stained blood smears rapidly, at a reduced cost of sample preparation, providing quantitative clinical information.

Exploration of Semantic Label Decomposition and Dataset Size in Semantic Indoor Scenes Synthesis via Optimized Residual Generative Adversarial Networks.

In this paper, we revisit the paired image-to-image translation using the conditional generative adversarial network, the so-called "Pix2Pix", and propose efficient optimization techniques for the architecture and the training method to maximize the architecture's performance to boost the realism of the generated images. We propose a generative adversarial network-based technique to create new artificial indoor scenes using a user-defined semantic segmentation map as an input to define the location, shape, and category of each object in the scene, exactly similar to Pix2Pix. We train different residual connections-based architectures of the generator and discriminator on the NYU depth-v2 dataset and a selected indoor subset from the ADE20K dataset, showing that the proposed models have fewer parameters, less computational complexity, and can generate better quality images than the state of the art methods following the same technique to generate realistic indoor images. We also prove that using extra specific labels and more training samples increases the quality of the generated images; however, the proposed residual connections-based models can learn better from small datasets (i.e., NYU depth-v2) and can improve the realism of the generated images in training on bigger datasets (i.e., ADE20K indoor subset) in comparison to Pix2Pix. The proposed method achieves an LPIPS value of 0.505 and an FID value of 81.067, generating better quality images than that produced by Pix2Pix and other recent paired Image-to-image translation methods and outperforming them in terms of LPIPS and FID.

DeepImageTranslator V2: Analysis of Multimodal Medical Images using Semantic Segmentation Maps Generated through Deep Learning

Introduction: Analysis of multimodal medical images often requires the selection of one or many anatomical regions of interest (ROIs) for extraction of useful statistics. This task can prove laborious when a manual approach is used. We have previously developed a user-friendly software tool for image-to-image translation using deep learning. Therefore, we present herein an update to the DeepImageTranslator V2 software with the addition of a tool for multimodal medical image segmentation analysis (hereby referred to as the MMMISA). Methods: The MMMISA was implemented using the Tkinter library; backend computations were implemented using the Pydicom, Numpy, and OpenCV libraries. We tested our software using 4188 slices from whole-body axial 2-deoxy-2-[18F]-fluoroglucose-position emission tomography/ computed tomography scans ([¹⁸F]-FDG-PET/CT) of 10 patients from the American College of Radiology Imaging Network-Head and Neck Squamous Cell Carcinoma (ACRIN-HNSCC) database. Using the deep learning software DeepImageTranslator, a model was trained with 36 randomly selected CT slices and manually labelled semantic segmentation maps. Utilizing the trained model, all the CT scans of the 10 HNSCC patients were segmented with high accuracy. Segmentation maps generated using the deep convolutional network were then used to measure organ specific [¹⁸F]-FDG uptake. We also compared measurements performed using the MMMISA and those made with manually selected ROIs. Results: The MMMISA is a tool that allows user to select ROIs based on deep learning-generated segmentation maps and to compute accurate statistics for these ROIs based on coregistered multimodal images. We found that organ-specific [¹⁸F]-FDG uptake measured using multiple manually selected ROIs is concordant with whole-tissue measurements made with segmentation maps using the MMMISA tool. Doi: 10.28991/HIJ-2022-03-03-07 Full Text: PDF