Neural Network For Semantic Segmentation Research Articles

A Multi-Task Convolutional Neural Network for Semantic Segmentation and Event Detection in Laparoscopic Surgery.

The current study presents a multi-task end-to-end deep learning model for real-time blood accumulation detection and tools semantic segmentation from a laparoscopic surgery video. Intraoperative bleeding is one of the most problematic aspects of laparoscopic surgery. It is challenging to control and limits the visibility of the surgical site. Consequently, prompt treatment is required to avoid undesirable outcomes. This system exploits a shared backbone based on the encoder of the U-Net architecture and two separate branches to classify the blood accumulation event and output the segmentation map, respectively. Our main contribution is an efficient multi-task approach that achieved satisfactory results during the test on surgical videos, although trained with only RGB images and no other additional information. The proposed multi-tasking convolutional neural network did not employ any pre- or postprocessing step. It achieved a Dice Score equal to 81.89% for the semantic segmentation task and an accuracy of 90.63% for the event detection task. The results demonstrated that the concurrent tasks were properly combined since the common backbone extracted features proved beneficial for tool segmentation and event detection. Indeed, active bleeding usually happens when one of the instruments closes or interacts with anatomical tissues, and it decreases when the aspirator begins to remove the accumulated blood. Even if different aspects of the presented methodology could be improved, this work represents a preliminary attempt toward an end-to-end multi-task deep learning model for real-time video understanding.

Semantic segmentation of surface defects of smooth parts based on deep convolutional neural networks

Machine vision plays an increasingly important role in industrial product quality detection. During processing, scratches, dents and other defects are inevitable on the surface of a smooth part. Although surface defects do not affect the overall performance of the product, their existence is unacceptable when a perfect product is required. The surface defect detection method based on machine vision and deep convolutional neural networks overcomes, to a certain extent, the problem of low detection efficiency, high false detection and missing detection rates in the traditional detection method. In this paper, a multistream semantic segmentation neural network is proposed to identify defects on smooth parts. Taking a seatbelt buckle as an example, the scratch and crush defects on the surface are classified. The network takes DeepLabV3+ as the framework and three types of image stream as the input of the network. In the backbone feature extraction network, the Xception structure is improved to MobilenetV2 and the convolutional block attention module (CBAM) is introduced into the decoding network, which improves the operational efficiency and accuracy. Compared with other classical networks, this network demonstrates good performance in the image dataset of the seatbelt buckle and realises fast and accurate semantic segmentation and classification of surface defects. The evaluation results of the network model have been significantly improved.

An Improved U-Net Model for Infrared Image Segmentation of Wind Turbine Blade

Infrared image segmentation of the wind turbine blade is an important link of wind turbine condition monitoring. Nevertheless, there remain plenty of challenges when the traditional semantic segmentation networks are applied directly to the infrared image segmentation of wind turbine blade, including the blurred boundaries caused by similar thermal radiation of background and blade, the irregularity of edge identification caused by uneven heating, and the hub misidentified as blade. To address these issues, a semantic segmentation neural network based on the basic U-Net is improved for infrared image segmentation of the wind turbine blade. The hierarchical-split depthwise separable convolution block is integrated into the constructed network to have a higher segmentation accuracy. Also, the fusion of convolution layer and batch normalization layer is performed in inference to get a better speed–accuracy tradeoff. The experimental results show that the proposed approach outperforms all the comparing methods, which further demonstrates that the trained network can achieve superior runtime performance and accurate segmentation with excellent anti-interference performance.

Semantic segmentation of water bodies in very high-resolution satellite and aerial images

This study evaluates the performance of convolutional neural networks for semantic segmentation of water bodies in very high-resolution satellite and aerial images from multiple sensors with particular focus on flood emergency response applications. Different model architectures (U-Net and DeepLab-V3+) are combined with encoder backbones (MobileNet-V3, ResNet-50 and EfficientNet-B4) and tested for their ability to delineate inundated areas under varying environmental conditions and data availability scenarios. An unprecedented reference dataset of 1120 globally sampled images with quality checked binary water masks is introduced and used to train, validate and test the models for water body segmentation. Furthermore, independent test datasets are developed to test the generalization ability of the trained models across regions, sensors (IKONOS, GeoEye-1, WorldView-2, WorldView-3 and four different airborne camera systems) and tasks (normal water and flood water segmentation). Results indicate that across all tested scenarios a U-Net model with Mobilenet-V3 backbone pre-trained on ImageNet performs best. While using R-G-B image bands performs well, adding the near infrared band (if available) slightly improves prediction results. Similarly, adding slope information from an independent digital elevation model increases accuracies. Train-time augmentation and contrast enhancement could improve transferability across sensors and in particular between satellite and aerial images. Moreover, adding noisy training data from freely available online resources could further improve performance with minimal annotation effort.

Segment-then-Segment: Context-Preserving Crop-Based Segmentation for Large Biomedical Images.

Medical images are often of huge size, which presents a challenge in terms of memory requirements when training machine learning models. Commonly, the images are downsampled to overcome this challenge, but this leads to a loss of information. We present a general approach for training semantic segmentation neural networks on much smaller input sizes called Segment-then-Segment. To reduce the input size, we use image crops instead of downscaling. One neural network performs the initial segmentation on a downscaled image. This segmentation is then used to take the most salient crops of the full-resolution image with the surrounding context. Each crop is segmented using a second specially trained neural network. The segmentation masks of each crop are joined to form the final output image. We evaluate our approach on multiple medical image modalities (microscopy, colonoscopy, and CT) and show that this approach greatly improves segmentation performance with small network input sizes when compared to baseline models trained on downscaled images, especially in terms of pixel-wise recall.

Semantic segmentation of superficial layer in intracoronary optical coherence tomography based on cropping-merging and deep learning

Manual labelling and qualitative analysis of lesion tissues in vessel optical coherence tomography (OCT) images are time-consuming and laborious for cardiovascular specialists. To beat these issues, a semantic segmentation method was presented based on deep learning on the OCT image analysis of the human vessel to reduce the diagnosis pressure of cardiovascular doctors. An outer border of ROI was segmented based on the level-set method to obtain the visible superficial layer containing useful information. And then, cropping square patches from the ROI with its patch center pixel inside the ROI and using patches as the input data. To reuse the preceding layers' feature maps, we used the dense block to replace the normal convolution layer, and simultaneously, employed a skip-connection from m the down-sampling path to the up-sampling path to keep the spatial information. With the advantage of SegNet on semantic segmentation, a dense-block-SegNet (DBSegNet) is constructed to complete the pixel-level segmentation. Training and testing were executed on 7 datasets (22, 210 images) to assess the model. Tenfold cross-validation method was implemented to measure the classification outcomes and the semantic segmentation capacity of our model. In deep learning experiment, sensitivities for calcified, fibrous and lipid plaques were 91.81 ± 3.60 %, 92.81 ± 2.72 % and 91.78 ± 1.62 %, respectively. A 3-D volume was created to insert each prediction slice along the depth axis to compute the maximum type number of each pixel in order to identify the final type of each pixel. Post-processing was used to refine the classification findings in order to eliminate classification errors. Semantic segmentation neural network based on the cropped input data and feature reusing was a feasible approach for the vessel tissue pixel-classification of IVOCT image. The proposed method has the potential to become a useful tool for specialists in analysing lesion kinds and determining clinical lesion characteristics (e.g., distribution position, angle and depth).

A Texture Integrated Deep Neural Network for Semantic Segmentation of Urban Meshes

3-D geo-information is essential for many urban related applications. Point cloud and mesh are two common representations of the 3-D urban surface. Compared to point cloud data, mesh possesses indispensable advantages, such as high-resolution image texture and sharp geometry representation. Semantic segmentation, as an important way to obtain 3-D geo-information, however, is mainly performed on the point cloud data. Due to the complex geometry representation and lack of efficient utilizing of image texture information, the semantic segmentation of the mesh is still a challenging task for urban 3-D geo-information acquisition. In this article, we propose a texture and geometry integrated deep learning method for the mesh segmentation task. A novel texture convolution module is introduced to capture image texture features. The texture features are concatenate with nontexture features on a point cloud that represents by the center of gravity (COG) of the mesh triangles. A hierarchical deep network is employed to segment the COG point cloud. Our experimental results show that the proposed network significantly improves the accuracy with the introduced texture convolution module (1.9% for overall accuracy and 4.0% for average F1 score). It also compares with other state-of-the-art methods on the public SUM-Helsinki dataset and achieves considerable results.

DWin-HRFormer: A High-Resolution Transformer Model With Directional Windows for Semantic Segmentation of Urban Construction Land

In this paper, a deep neural network for semantic segmentation of high-resolution remote sensing images is proposed for urban construction land classification. The network follows a high-resolution network architecture. Specifically, a directional self-attention on the paths of different resolutions is proposed, aiming to correct the directional bias caused by the attention of strip windows during the model learning, while also reducing the computational complexity, and allowing the model to improve both the accuracy and speed. At the end of the network, a distributed alignment module with spatial information is constructed to train additional learnable parameters, to adjust the biased decision boundaries through a two-stage learning strategy, and alleviate the problem of accuracy degradation due to the unbalanced training data. We tested the proposed method and compared it with the current state-of-the-art semantic segmentation methods on the Luojia-FGLC and WHDLD datasets, and the proposed one obtained the best performance. We also verified the effectiveness of each component of the network through ablation experiments. The code and model will be available at https://github.com/Zhzhyd/DWin-HRFormer.

Aerial Fluvial Image Dataset for Deep Semantic Segmentation Neural Networks and Its Benchmarks

Aerial Fluvial Image Dataset for Deep Semantic Segmentation Neural Networks and Its Benchmarks

Malaria detection using custom Semantic Segmentation Neural Network Architecture

Malaria is a significant disease that affects both animals and humans. The four main Plasmodium species that cause human malaria are Plasmodium falciparum, Plasmodium vivax, Plasmodium malariae, and Plasmodium ovale. Plasmodium knowlesi, a parasite typically infecting forest macaque monkeys, was recently revealed to be able to be transmitted by anophelines and provoke malaria in humans. This provides an increasing risk of spreading the disease to areas previously unaffected with it and infecting people during the increasingly popular travels abroad. Microscopic examination remains one of the most often used methods for its laboratory confirmation. These tests, however, should be performed immediately after receiving samples from a firstcontact doctor to allow immediate therapy. This research presents a novel, semantic segmentation neural network architecture designed to quickly create a classification mask, giving the doctor information about the position, shape, and possible affiliation of detected elements. The evaluation method is based on a light microscope imagery and was created to overcome problems resulting from the human diagnosis specifics. There are 3 abstract classes containing healthy cells, cells with malaria and background. The outputted mask can be later mapped to a more readable form with the inclusion of contrasting colors, next to an original image for quick validation. Such an approach allows for semi-automatic recognition of possible disease, nevertheless still giving the final verdict to the specialist. The developed solution has achieved a high recognition accuracy of 96.65%, while the computer power requirements are kept at a minimum. The proposed solution can help reduce misclassification rates by providing additional data for the doctor and speed up the entire process with the early diagnosis made by a deep learning model.

Comparative Study of Real-Time Semantic Segmentation Networks in Aerial Images During Flooding Events

Real-time semantic segmentation of aerial imagery is essential for unmanned ariel vehicle applications, including military surveillance, land characterization, and disaster damage assessments. Recent real-time semantic segmentation neural networks promise low computation and inference time, appropriate for resource-limited platforms, such as edge devices. However, these methods are mainly trained on human-centric view datasets, such as Cityscapes and CamVid, unsuitable for aerial applications. Furthermore, we do not know the feasibility of these models under adversarial settings, such as flooding events. To solve these problems, we train the most recent real-time semantic segmentation architectures on the FloodNet dataset containing annotated aerial images captured after hurricane Harvey. This article comprehensively studies several lightweight architectures, including encoder–decoder and two-pathway architectures, evaluating their performance on aerial imagery datasets. Moreover, we benchmark the efficiency and accuracy of different models on the FloodNet dataset to examine the practicability of these models during emergency response for aerial image segmentation. Some lightweight models attain more than 60% test mIoU on the FloodNet dataset and yield qualitative results on images. This article highlights the strengths and weaknesses of current segmentation models for aerial imagery, requiring low computation and inference time. Our experiment has direct applications during catastrophic events, such as flooding events.

LightFGCNet: A Lightweight and Focusing on Global Context Information Semantic Segmentation Network for Remote Sensing Imagery

Convolutional neural networks have attracted much attention for their use in the semantic segmentation of remote sensing imagery. The effectiveness of semantic segmentation of remote sensing images is significantly influenced by contextual information extraction. The traditional convolutional neural network is constrained by the size of the convolution kernel and mainly concentrates on local contextual information. We suggest a new lightweight global context semantic segmentation network, LightFGCNet, to fully utilize the global context data and to further reduce the model parameters. It uses an encoder–decoder architecture and gradually combines feature information from adjacent encoder blocks during the decoding upsampling stage, allowing the network to better extract global context information. Considering that the frequent merging of feature information produces a significant quantity of redundant noise, we build a unique and lightweight parallel channel spatial attention module (PCSAM) for a few critical contextual features. Additionally, we design a multi-scale fusion module (MSFM) to acquire multi-scale feature target information. We conduct comprehensive experiments on the two well-known datasets ISPRS Vaihingen and WHU Building. The findings demonstrate that our suggested strategy can efficiently decrease the number of parameters. Separately, the number of parameters and FLOPs are 3.12 M and 23.5 G, respectively, and the mIoU and IoU of our model on the two datasets are 70.45% and 89.87%, respectively, which is significantly better than what the conventional convolutional neural networks for semantic segmentation can deliver.

PMOD-Net: point-cloud-map-based metric scale obstacle detection by using a monocular camera

Metric scale obstacle detection, which detects obstacles and measures the distances to them with a metric scale, is a key function in autonomous driving. A monocular camera is inexpensive and effective for detecting objects in images. However, it cannot measure the distances to objects with a metric scale because it can estimate only relative distance. 3D point cloud maps can determine the distances of fixed objects in the 3D map; however, they cannot detect non-fixed obstacles that are not in the 3D map. Therefore, we developed a new method for detecting non-fixed obstacles using a monocular camera and 3D point cloud maps. We used a semantic segmentation neural network (NN) for detecting obstacles and an image-guided depth completion NN for densifying a sparse depth map with a metric scale. We proposed a multitask NN that three-dimensionally reconstructed non-fixed obstacles using the shape information obtained by the semantic segmentation NN. The detection accuracy of the proposed multitask NN was 1.3 times higher than that of a single-task method. Moreover, our robot avoided obstacles using the proposed NN.

Beyond classification: directly training spiking neural networks for semantic segmentation

Spiking neural networks (SNNs) have recently emerged as the low-power alternative to artificial neural networks (ANNs) because of their sparse, asynchronous, and binary event-driven processing. Due to their energy efficiency, SNNs have a high possibility of being deployed for real-world, resource-constrained systems such as autonomous vehicles and drones. However, owing to their non-differentiable and complex neuronal dynamics, most previous SNN optimization methods have been limited to image recognition. In this paper, we explore the SNN applications beyond classification and present semantic segmentation networks configured with spiking neurons. Specifically, we first investigate two representative SNN optimization techniques for recognition tasks (i.e., ANN-SNN conversion and surrogate gradient learning) on semantic segmentation datasets. We observe that, when converted from ANNs, SNNs suffer from high latency and low performance due to the spatial variance of features. Therefore, we directly train networks with surrogate gradient learning, resulting in lower latency and higher performance than ANN-SNN conversion. Moreover, we redesign two fundamental ANN segmentation architectures (i.e., Fully Convolutional Networks and DeepLab) for the SNN domain. We conduct experiments on three semantic segmentation benchmarks including PASCAL VOC2012 dataset, DDD17 event-based dataset, and synthetic segmentation dataset combined CIFAR10 and MNIST datasets. In addition to showing the feasibility of SNNs for semantic segmentation, we show that SNNs can be more robust and energy-efficient compared to their ANN counterparts in this domain.

Research on Green View Index of Urban Roads Based on Street View Image Recognition: A Case Study of Changsha Downtown Areas

In this paper, we took the urban roads in the Changsha downtown areas as an example to identify the green view index (GVI) of urban roads based on street view images (SVIs). First, the road network information was obtained through OpenStreetMap, and the coordinate information of sampling points was processed using ArcGIS. Secondly, the SVIs were downloaded from Baidu Map according to the latitude and longitude coordinates of the sampling points. Moreover, semantic segmentation neural network software was used to semantically segment the SVIs for recognizing the objects in each part of the images. Finally, the objects related to green vegetation were statistically analyzed to obtain the GVI of the sampling points. The GVI was mapped to the map in ArcGIS software for data visualization and analysis. The results showed the average GVI of the study area was 12.56%. An amount of 27% have very poor green perception, 40% have poor green perception, 19% have general green perception, 10% have strong green perception, and 4% have very strong green perception. In the administrative districts, the highest GVI is Yuhua District with 14.15%, while the lowest is Kaifu District with 8.75%. The average GVI of the new urban area is higher than that of the old urban area, as the old urban area has higher building density and a lower greenery level. This paper systematically evaluated the levels of GVI and greening status of urban streets within the Changsha downtown areas through SVIs data analysis, and provided guidance and suggestions for the greening development of Changsha City.

Exploring uncertainty measures in convolutional neural network for semantic segmentation of oral cancer images.

Oral cancer is one of the most prevalent cancers, especially in middle- and low-income countries such as India. Automatic segmentation of oral cancer images can improve the diagnostic workflow, which is a significant task in oral cancer image analysis. Despite the remarkable success of deep-learning networks in medical segmentation, they rarely provide uncertainty quantification for their output. We aim to estimate uncertainty in a deep-learning approach to semantic segmentation of oral cancer images and to improve the accuracy and reliability of predictions. This work introduced a UNet-based Bayesian deep-learning (BDL) model to segment potentially malignant and malignant lesion areas in the oral cavity. The model can quantify uncertainty in predictions. We also developed an efficient model that increased the inference speed, which is almost six times smaller and two times faster (inference speed) than the original UNet. The dataset in this study was collected using our customized screening platform and was annotated by oral oncology specialists. The proposed approach achieved good segmentation performance as well as good uncertainty estimation performance. In the experiments, we observed an improvement in pixel accuracy and mean intersection over union by removing uncertain pixels. This result reflects that the model provided less accurate predictions in uncertain areas that may need more attention and further inspection. The experiments also showed that with some performance compromises, the efficient model reduced computation time and model size, which expands the potential for implementation on portable devices used in resource-limited settings. Our study demonstrates the UNet-based BDL model not only can perform potentially malignant and malignant oral lesion segmentation, but also can provide informative pixel-level uncertainty estimation. With this extra uncertainty information, the accuracy and reliability of the model’s prediction can be improved.

Quantized Semantic Segmentation Deep Architecture for Deployment on an Edge Computing Device for Image Segmentation

In the field of computer vision technology, deep learning of image processing has become an emerging research area. The semantic segmentation of an image is among the utmost essential and significant tasks in image-processing research, offering a wide range of application fields such as autonomous driving systems, medical diagnosis, surveillance security, etc. Thus far, many studies have suggested and developed neural network modules in deep learning. To the best of our knowledge, all existing neural networks for semantic segmentation have large parameter sizes and it is therefore unfeasible to implement those architectures in low-power and memory-limited embedded platforms such as FPGAs. Building an embedded platform with that architecture is possible after reducing the parameter size without affecting the module’s architecture. The quantization technique lowers the precision of the neural network parameters while mostly keeping the accuracy. In this paper, we propose a quantization algorithm for a semantic segmentation deep learning architecture, which reduces the parameter size by four to eight times with a negligible accuracy abatement. As long as the parameter size is reduced, the deep learning architecture is improved in terms of required storage, computational speed, and power efficiency.

Multi-species weed density assessment based on semantic segmentation neural network

Multi-species weed density assessment based on semantic segmentation neural network

Fast Detection of Tomato Sucker Using Semantic Segmentation Neural Networks Based on RGB-D Images.

Tomato sucker or axillary shoots should be removed to increase the yield and reduce the disease on tomato plants. It is an essential step in the tomato plant care process. It is usually performed manually by farmers. An automated approach can save a lot of time and labor. In the literature review, we see that semantic segmentation is a process of recognizing or classifying each pixel in an image, and it can help machines recognize and localize tomato suckers. This paper proposes a semantic segmentation neural network that can detect tomato suckers quickly by the tomato plant images. We choose RGB-D images which capture not only the visual of objects but also the distance information from objects to the camera. We make a tomato RGB-D image dataset for training and evaluating the proposed neural network. The proposed semantic segmentation neural network can run in real-time at 138.2 frames per second. Its number of parameters is 680, 760, much smaller than other semantic segmentation neural networks. It can correctly detect suckers at 80.2%. It requires low system resources and is suitable for the tomato dataset. We compare it to other popular non-real-time and real-time networks on the accuracy, time of execution, and sucker detection to prove its better performance.

Detection and localization of fugitive emissions in industrial plants using surveillance cameras

Industrial plants commonly generate gas emissions that are not caused intentionally. These emissions are known as fugitive emissions. Early detection of fugitive emissions helps to find points of failure in the different processes and avoid sources of pollution, helping to reduce danger to the environment and to respect legislation. Despite the importance of the problem, there are no published solutions in the specialized literature about the location and automated detection of fugitive emissions in industrial plants. Therefore, this article proposes an effective approach based on convolutional neural networks for semantic segmentation. The proposed solution takes advantage of existing surveillance cameras to apply state-of-the-art image segmentation methods, in particular, the semantic segmentation network DeeplabV3 + . This work explores aspects such as the ability to differentiate gases like water vapor and clouds from fugitive emissions, the possibility of reusing models in different industrial plants, the differences between multi-class and binary classification, the importance of proportions in the number of images in each class, the use of weights to balance classes, the comparison of a standard size test versus a real use case test, and the feasibility of an area-based alarm system to warn of emissions. This paper describes a methodology to configure the proposed solution for a specific industrial facility.