Deep Learning Semantic Segmentation Research Articles

Quantized Semantic Segmentation Deep Architecture for Deployment on an Edge Computing Device for Image Segmentation

In the field of computer vision technology, deep learning of image processing has become an emerging research area. The semantic segmentation of an image is among the utmost essential and significant tasks in image-processing research, offering a wide range of application fields such as autonomous driving systems, medical diagnosis, surveillance security, etc. Thus far, many studies have suggested and developed neural network modules in deep learning. To the best of our knowledge, all existing neural networks for semantic segmentation have large parameter sizes and it is therefore unfeasible to implement those architectures in low-power and memory-limited embedded platforms such as FPGAs. Building an embedded platform with that architecture is possible after reducing the parameter size without affecting the module’s architecture. The quantization technique lowers the precision of the neural network parameters while mostly keeping the accuracy. In this paper, we propose a quantization algorithm for a semantic segmentation deep learning architecture, which reduces the parameter size by four to eight times with a negligible accuracy abatement. As long as the parameter size is reduced, the deep learning architecture is improved in terms of required storage, computational speed, and power efficiency.

A Method for Extracting Building Information from Remote Sensing Images Based on Deep Learning.

Semantic segmentation of remote sensing images is an important issue in remote sensing tasks. Existing algorithms can extract information more accurately, but it is difficult to capture the contours of objects and further reveal the interaction information between different objects in the image. Therefore, a deep learning-based method for extracting building information from remote sensing images is proposed. First, the deep learning semantic segmentation model DeepLabv3+ and Mixconv2d are combined, and convolution kernels of different sizes are used for feature recognition. Then, the regularization method based on Rdrop Loss improves the accuracy and efficiency of contour capture for objects of different resolutions, and at the same time improves the consistency of dataset fitting. Finally, the proposed remote sensing image information extraction method is verified based on the self-built dataset. The experimental results show that the proposed algorithm can effectively improve the algorithm efficiency and result accuracy, and has good segmentation performance.

CT-derived body composition measurements as predictors for neoadjuvant treatment tolerance and survival in gastroesophageal adenocarcinoma.

Treatment for gastroesophageal adenocarcinomas can result in significant morbidity and mortality. The purpose of this study is to supplement methods for choosing treatment strategy by assessing the relationship between CT-derived body composition, patient, and tumor features, and clinical outcomes in this population. Patients with neoadjuvant treatment, biopsy-proven gastroesophageal adenocarcinoma, and initial staging CTs were retrospectively identified from institutional clinic encounters between 2000 and 2019. Details about patient, disease, treatment, and outcomes (including therapy tolerance and survival) were extracted from electronic medical records. A deep learning semantic segmentation algorithm was utilized to measure cross-sectional areas of skeletal muscle (SM), visceral fat (VF), and subcutaneous fat (SF) at the L3 vertebra level on staging CTs. Univariate and multivariate analyses were performed to assess the relationships between predictors and outcomes. 142 patients were evaluated. Median survival was 52months. Univariate and multivariate analysis showed significant associations between treatment tolerance and SM and VF area, SM to fat and VF to SF ratios, and skeletal muscle index (SMI) (p = 0.004-0.04). Increased survival was associated with increased body mass index (BMI) (p = 0.01) and increased SMI (p = 0.004). A multivariate Cox model consisting of BMI, SMI, age, gender, and stage demonstrated that patients in the high-risk group had significantly lower survival (HR = 1.77, 95% CI = 1.13-2.78, p = 0.008). CT-based measures of body composition in patients with gastroesophageal adenocarcinoma may be independent predictors of treatment complications and survival and can supplement methods for assessing functional status during treatment planning.

Bayesian deep learning for semantic segmentation of food images

Deep learning has provided promising results in various applications; however, algorithms tend to be overconfident in their predictions, even though they may be entirely wrong. Particularly for critical applications, the model should provide answers only when it is very sure of them. This article presents a Bayesian version of two different state-of-the-art semantic segmentation methods to perform multi-class segmentation of foods and estimate the uncertainty about the given predictions. The proposed methods were evaluated on three public pixel-annotated food datasets. As a result, we can conclude that Bayesian methods improve the performance achieved by the baseline architectures and, in addition, provide information to improve decision-making. Furthermore, based on the extracted uncertainty map, we proposed three measures to rank the images according to the degree of noisy annotations they contained. Note that the top 135 images ranked by one of these measures include more than half of the worst-labeled food images.

Strip Attention Networks for Road Extraction

In recent years, deep learning methods have been widely used for road extraction in remote sensing images. However, the existing deep learning semantic segmentation networks generally show poor continuity in road segmentation due to the high-class similarity between roads and buildings surrounding roads in remote sensing images, and the existence of shadows and occlusion. To deal with this problem, this paper proposes strip attention networks (SANet) for extracting roads in remote sensing images. Firstly, a strip attention module (SAM) is designed to extract the contextual information and spatial position information of the roads. Secondly, a channel attention fusion module (CAF) is designed to fuse low-level features and high-level features. The network is trained and tested using the CITY-OSM dataset, DeepGlobe road extraction dataset, and CHN6-CUG dataset. The test results indicate that SANet exhibits excellent road segmentation performance and can better solve the problem of poor road segmentation continuity compared with other networks.

Formation and propagation of cracks in RRP Nb3Sn wires studied by deep learning applied to x-ray tomography

This paper reports a novel non-destructive and non-invasive method to investigate crack formation and propagation in high-performance Nb3Sn wires by combining x-ray tomography and deep learning networks. The next generation of high field magnet applications relies on the development of new Nb3Sn wires capable to withstand the large stresses generated by Lorentz forces during magnets operation. These stresses can cause a permanent reduction of the transport properties generated by residual deformation of the Nb3Sn crystal lattice as well as the formation of cracks in the brittle Nb3Sn filaments. Studies for the development of the high luminosity LHC (HL-LHC) upgrade showed that nominal transverse compressive stresses above 150 MPa may be sufficient to generate cracks in the wires. In the case of fusion magnets, wires experience periodic bending due to the electro-magnetic cycles of the reactor which over time may induce wire deformation and filament cracks. Therefore, it has become essential to develop a quantitative method for the characterization of crack formation and propagation under compressive loads. The x-ray tomographic data of a series of restacked-rod-process (RRP) Nb3Sn wires was acquired at the micro-tomography beamline ID19 of the European Synchrotron Radiation Facility (ESRF), after intentionally inducing a broad spectrum of cracks in the Nb3Sn sub-elements. The samples were submitted to transvers compressive stresses, with and without epoxy impregnation, at different pressures, up to 238 MPa. The resulting tomographic images were analysed by means of deep learning semantic segmentation networks, using U-net, a convolutional neural network (CNN), to identify and segment cracks inside the wires. The trained CNN was able to analyse large volumes of tomographic data, thus enabling a systematic approach for investigating the mechanical damages in Nb3Sn wires. We will show the complete three-dimensional reconstruction of various cracks and discuss their impact on the electro-mechanical performance of the analysed wires.

A Hierarchical Information Extraction Method for Large-Scale Centralized Photovoltaic Power Plants Based on Multi-Source Remote Sensing Images

In the context of global sustainable development, solar energy is very widely used. The installed capacity of photovoltaic panels in countries around the world, especially in China, is increasing steadily and rapidly. In order to obtain accurate information about photovoltaic panels and provide data support for the macro-control of the photovoltaic industry, this paper proposed a hierarchical information extraction method, including positioning information and shape information, and carried out photovoltaic panel distribution mapping. This method is suitable for large-scale centralized photovoltaic power plants based on multi-source satellite remote sensing images. This experiment takes the three northwest provinces of China as the research area. First, a deep learning scene classification model, the EfficientNet-B5 model, is used to locate the photovoltaic power plants on 16-m spatial resolution images. This step obtains the area that contains or may contain photovoltaic panels, greatly reducing the study area with an accuracy of 99.97%. Second, a deep learning semantic segmentation model, the U2-Net model, is used to precisely locate photovoltaic panels on 2-m spatial resolution images. This step achieves the exact extraction results of the photovoltaic panels from the area obtained in the previous step, with an accuracy of 97.686%. This paper verifies the superiority of a hierarchical information extraction method in terms of accuracy and efficiency through comparative experiments with DeepLabV3+, U-Net, SegNet, and FCN8s. This meaningful work identified 180 centralized photovoltaic power plants in the study area. Additionally, this method makes full use of the characteristics of different remote sensing data sources. This method can be applied to the rapid extraction of global photovoltaic panels.

Segmentation and weight prediction of grape ear based on SFNet-ResNet18

ABSTRACT In this paper, the segment and weight prediction problems are investigated for ear of grape based on deep learning technologies. The image datum is collected from ZaoHeiBao grape in a greenhouse by camera. The grape ear target segmentation model is constructed by cross combining three backbone networks (ResNet18, ResNet50, and ResNet101) and four deep learning semantic segmentation networks (SFNet, GCNet, EMANet, and Deeplabv3). The experimental results show that for the SFNet-ResNet18 model, whose structural size is 52.68MB, the mean Intersection over Union (mIoU) is , the mean Pixel Accuracy (mPA) is , and the average segmentation speed of the image () is 0.217s. Therefore, the performance of the SFNet-ResNet18 model outperforms other combined network models and is selected to segment grape ears. Furthermore, on the basis of the segmentation results of grape ears by using the SFNet-ResNet18 model, the grape ear weight is predicted by adopting fractional regression model. The value of is 0.8903, which means that the selected model could accurately predict the weight of grape ears. The proposed method can not only segment the grape ears and accurately predict the weight of the grape ears, but also provide theoretical and technical support for grape yield prediction.

New deep learning method for efficient extraction of small water from remote sensing images.

Extracting water bodies from remote sensing images is important in many fields, such as in water resources information acquisition and analysis. Conventional methods of water body extraction enhance the differences between water bodies and other interfering water bodies to improve the accuracy of water body boundary extraction. Multiple methods must be used alternately to extract water body boundaries more accurately. Water body extraction methods combined with neural networks struggle to improve the extraction accuracy of fine water bodies while ensuring an overall extraction effect. In this study, false color processing and a generative adversarial network (GAN) were added to reconstruct remote sensing images and enhance the features of tiny water bodies. In addition, a multi-scale input strategy was designed to reduce the training cost. We input the processed data into a new water body extraction method based on strip pooling for remote sensing images, which is an improvement of DeepLabv3+. Strip pooling was introduced in the DeepLabv3+ network to better extract water bodies with a discrete distribution at long distances using different strip kernels. The experiments and tests show that the proposed method can improve the accuracy of water body extraction and is effective in fine water body extraction. Compared with seven other traditional remote sensing water body extraction methods and deep learning semantic segmentation methods, the prediction accuracy of the proposed method reaches 94.72%. In summary, the proposed method performs water body extraction better than existing methods.

Deriving Urban Boundaries of Henan Province, China, Based on Sentinel-2 and Deep Learning Methods

Accurate urban boundary data can directly reflect the expansion of urban space, help us accurately grasp the scale and form of urban space, and play a vital role in urban land development and policy-making. However, the lack of reliable multiscale and high-precision urban boundary data products and relevant training datasets has become one of the major factors hindering their application. The purpose of this study is to combine Sentinel-2 remote-sensing images and supplementary geographic data to generate a reliable high-precision urban boundary dataset for Henan Province (called HNUB2018). First, this study puts forward a clear definition of “urban boundary”. Using this concept as its basis, it proposes a set of operable urban boundary delimitation rules and technical processes. Then, based on Sentinel-2 remote-sensing images and supplementary geographic data, the urban boundaries of Henan Province are delimited by a visual interpretation method. Finally, the applicability of the dataset is verified by using a classical semantic segmentation deep learning model. The results show that (1) HNUB2018 has clear and rich detailed features as well as a detailed spatial structure of urban boundaries. The overall accuracy of HNUB2018 is 92.82% and the kappa coefficient reaches 0.8553, which is better than GUB (Henan) in overall accuracy. (2) HNUB2018 is well suited for deep learning, with excellent reliability and scientific validity. The research results of this paper can provide data support for studies of urban sprawl monitoring and territorial spatial planning, and will support the development of reliable datasets for fields such as intelligent mapping of urban boundaries, showing prospects and possibilities for wide application in urban research.

Nodular and Micronodular Basal Cell Carcinoma Subtypes Are Different Tumors Based on Their Morphological Architecture and Their Interaction with the Surrounding Stroma.

Basal cell carcinoma (BCC) is the most frequent cancer of the skin and comprises low-risk and high-risk subtypes. We selected a low-risk subtype, namely, nodular (N), and a high-risk subtype, namely, micronodular (MN), with the aim to identify differences between them using a classical morphometric approach through a gray-level co-occurrence matrix and histogram analysis, as well as an approach based on deep learning semantic segmentation. From whole-slide images, pathologists selected 216 N and 201 MN BCC images. The two groups were then manually segmented and compared based on four morphological areas: center of the BCC islands (tumor, T), peripheral palisading of the BCC islands (touching tumor, TT), peritumoral cleft (PC) and surrounding stroma (S). We found that the TT pattern varied the least, while the PC pattern varied the most between the two subtypes. The combination of two distinct analysis approaches yielded fresh insights into the characterization of BCC, and thus, we were able to describe two different morphological patterns for the T component of the two subtypes.

Semi-supervised deep learning and low-cost cameras for the semantic segmentation of natural images in viticulture

Automatic yield monitoring and in-field robotic harvesting by low-cost cameras require object detection and segmentation solutions to tackle the poor quality of natural images and the lack of exactly-labeled datasets of consistent sizes. This work proposed the application of deep learning for semantic segmentation of natural images acquired by a low-cost RGB-D camera in a commercial vineyard. Several deep architectures were trained and compared on 85 labeled images. Three semi-supervised learning methods (PseudoLabeling, Distillation and Model Distillation) were proposed to take advantage of 320 non-annotated images. In these experiments, the DeepLabV3+ architecture with a ResNext50 backbone, trained with the set of labeled images, achieved the best overall accuracy of 84.78%. In contrast, the Manet architecture combined with the EfficientnetB3 backbone reached the highest accuracy for the bunch class (85.69%). The application of semi-supervised learning methods boosted the segmentation accuracy between 5.62 and 6.01%, on average. Further discussions are presented to show the effects of a fine-grained manual image annotation on the accuracy of the proposed methods and to compare time requirements.

Tradeoffs between UAS Spatial Resolution and Accuracy for Deep Learning Semantic Segmentation Applied to Wetland Vegetation Species Mapping

Recent advances in image classification of fine spatial resolution imagery from unoccupied aircraft systems (UASs) have allowed for mapping vegetation based on both multispectral reflectance and fine textural details. Convolutional neural network (CNN)-based models can take advantage of the spatial detail present in UAS imagery by implicitly learning shapes and textures associated with classes to produce highly accurate maps. However, the spatial resolution of UAS data is infrequently examined in CNN classification, and there are important tradeoffs between spatial resolution and classification accuracy. To improve the understanding of the relationship between spatial resolution and classification accuracy for a CNN-based model, we captured 7.6 cm imagery with a UAS in a wetland environment containing graminoid (grass-like) plant species and simulated a range of spatial resolutions up to 76.0 cm. We evaluated two methods for the simulation of coarser spatial resolution imagery, averaging before and after orthomosaic stitching, and then trained and applied a U-Net CNN model for each resolution and method. We found untuned overall accuracies exceeding 70% at the finest spatial resolutions, but classification accuracy decreased as spatial resolution coarsened, particularly beyond a 22.8 cm resolution. Coarsening the spatial resolution from 7.6 cm to 22.8 cm could permit a ninefold increase in survey area, with only a moderate reduction in classification accuracy. This study provides insight into the impact of the spatial resolution on deep learning semantic segmentation performance and information that can potentially be useful for optimizing precise UAS-based mapping projects.

Performance and limitations of deep learning semantic segmentation of multiple defects in transmission electron micrographs

Transmission electron microscopy (TEM) is a popular method for characterizing and quantifying defects in materials. Analyzing digitized TEM images is typically done manually, which is a time-consuming and potentially error-prone task that is not scalable to large dataset sizes, motivating development of automated methods for quantifying and analyzing defects in TEM images. In this work, we perform semantic segmentation of multiple defect types in electron microscopy images of irradiated FeCrAl alloys using a deep learning mask regional convolutional neural network (Mask R-CNN) model. We evaluate the performance of the model based on distributions of defect shapes, sizes, and areal densities relevant to informing physical modeling and understanding irradiated Fe-based materials properties. To better understand the performance and present limitations of the model, we provide examples of useful evaluation tests, which include a suite of random splits and dataset-size-dependent and domain-targeted cross-validation tests, exposing potential weak points in the model applicability domain. Our model predicts the expected irradiation-induced material hardening to within 10–20 MPa (about 10% of total hardening), on par with experimental error. Finally, we discuss the first phase of an effort to provide an easy-to-use, open-source object detection tool to the broader community for identifying defects in new images.

Application of Deep Learning in Land Use Classification for Soil Erosion Using Remote Sensing

China is suffering from serious soil and water loss due to improper land use, leading to flood and drought disasters, and industrial and agricultural reduction. The commonly used monitoring methods of soil and water loss rely on the visual interpretation of remote sensing images, which is time-consuming and laborious. In this study, deep learning semantic segmentation technology was applied to monitoring land use for soil and water conservation. Manual visual interpretation mark samples and land use products collected in recent years were combined to construct training samples for deep learning. Deep convolution neural network model based on ResNet152 and DeeplabV3+ was selected for performing land use classification. Then the soil and water loss were quantified using the Chinese Soil Loss Equation (CSLE) based on the hydrological, geographic, vegetation cover and land use information. The experimental results show that the deep learning model can quickly extract robust edge features from remote sensing images and has high pixel accuracy (83.4%). Although the model accuracy was affected by the land cover types and image quality in different study area, it can still achieve an accuracy higher than 70% in other counties through our experiment. It can further improve the information service level of soil and water conservation product, and provide a useful guideline for automatic land use interpretation and identification of soil and water conservation status based on high resolution remote sensing images.

Automatic segmentation of cattle rib-eye area in ultrasound images using the UNet++ deep neural network

• Deep Neural Networks perform well the semantic segmentation task in ultrasound images. • UNet ++ leads out better results inside the group of the Deep Neural Networks tested. • Cattle Rib-Eye Area can be mapped using the UNet++ neural network with high accuracy. Ultrasound imaging is commonly used to estimate the size of various cuts of meat or quality traits in live animals. Unfortunately, ultrasound images are known for having large amount of visual noise, which can make it difficult to define the exact boundaries or shapes of the regions of the interest in these images. Therefore, new strategies related to the digital image processing field are required to improve the process of obtaining information from these groups of images. In this context, artificial intelligence, through deep learning methods particularly, has proved to be an optimized and efficient strategy, but that has not yet been investigated in the cattle rib-eye area. This paper aims to investigate the feasibility of applying the Unet++ deep neural network to automatic segmentation of cattle rib-eye area in ultrasound images. Additionally, several well established deep learning semantic segmentation models are compared with Unet++ performance. These architectures are FCN, U-Net, SegNet, and Deeplab v3+. The models were tested on a dataset composed of gray scale images of cattle ultrasound. All models showed excellent results in both location and boundaries. Best results showed 97.37 % in IoU, 1.14 cm 2 in MAE and coefficient of determination ( R 2 ) of 0.999 . The labeled rib-eye area dataset used in this study is available for future research.

MangoGAN: a general adversarial network-based deep learning architecture for mango tree crown detection

We present MangoGAN, a general adversarial network (GAN)-based deep learning semantic segmentation model for the detection of mango tree crowns in remotely sensed aerial images. The aerial images are acquired by low-altitude remote sensing carried out using a quadrotor unmanned aerial vehicle in a mango orchard. Aerial images are acquired with a vision spectrum optical sensor, also popularly known as RGB images as the payload. MangoGAN is trained on 1430 images patches of size 240 × 240 pixels. The testing was carried out on 160 images. Results are analyzed using the precision, recall, F1 parameters derived from contingency matrix and by visualization using Gradcam method. The performance of the MangoGAN is compared with peer architectures trained on the same data. MangoGAN outperforms its peer architectures

Semantic SLAM Based on Deep Learning in Endocavity Environment

Traditional endoscopic treatment methods restrict the surgeon’s field of view. New approaches to laparoscopic visualization have emerged due to the advent of robot-assisted surgical techniques. Lumen simultaneous localization and mapping (SLAM) technology can use the image sequence taken by the endoscope to estimate the pose of the endoscope and reconstruct the lumen scene in minimally invasive surgery. This technology gives the surgeon better visual perception and is the basis for the development of surgical navigation systems as well as medical augmented reality. However, the movement of surgical instruments in the internal cavity can interfere with the SLAM algorithm, and the feature points extracted from the surgical instruments may cause errors. Therefore, we propose a modified endocavity SLAM method combined with deep learning semantic segmentation that introduces a convolution neural network based on U-Net architecture with a symmetric encoder–decoder structure in the visual odometry with the goals of solving the binary segmentation problem between surgical instruments and the lumen background and distinguishing dynamic feature points. Its segmentation performance is improved by using pretrained encoders on the network model to obtain more accurate pixel-level instrument segmentation. In this setting, the semantic segmentation is used to reject the feature points on the surgical instruments and reduce the impact caused by dynamic surgical instruments. This can provide more stable and accurate mapping results compared to ordinary SLAM systems.

Semi-Supervised Deep Learning Semantic Segmentation for 3D Volumetric Computed Tomographic Scoring of Chronic Rhinosinusitis: Clinical Correlations and Comparison with Lund-Mackay Scoring.

Background: The traditional Lund-Mackay score (TLMs) is unable to subgrade the volume of inflammatory disease. We aimed to propose an effective modification and calculated the volume-based modified LM score (VMLMs), which should correlate more strongly with clinical symptoms than the TLMs. Methods: Semi-supervised learning with pseudo-labels used for self-training was adopted to train our convolutional neural networks, with the algorithm including a combination of MobileNet, SENet, and ResNet. A total of 175 CT sets, with 50 participants that would undergo sinus surgery, were recruited. The Sinonasal Outcomes Test-22 (SNOT-22) was used to assess disease-specific symptoms before and after surgery. A 3D-projected view was created and VMLMs were calculated for further comparison. Results: Our methods showed a significant improvement both in sinus classification and segmentation as compared to state-of-the-art networks, with an average Dice coefficient of 91.57%, an MioU of 89.43%, and a pixel accuracy of 99.75%. The sinus volume exhibited sex dimorphism. There was a significant positive correlation between volume and height, but a trend toward a negative correlation between maxillary sinus and age. Subjects who underwent surgery had significantly greater TLMs (14.9 vs. 7.38) and VMLMs (11.65 vs. 4.34) than those who did not. ROC-AUC analyses showed that the VMLMs had excellent discrimination at classifying a high probability of postoperative improvement with SNOT-22 reduction. Conclusions: Our method is suitable for obtaining detailed information, excellent sinus boundary prediction, and differentiating the target from its surrounding structure. These findings demonstrate the promise of CT-based volumetric analysis of sinus mucosal inflammation.

Performance boosting of conventional deep learning-based semantic segmentation leveraging unsupervised clustering

In Scan-to-BIM, semantically understanding 3D point clouds is an essential process that must precede 3D BIM elements generation. The main idea of this research stemmed from an assumption where recognizing a group of points would be simpler than assigning label per point from the machine's perspective when using deep learning classifiers. To validate our assumption, conventional point-wise classification problem was formulated as segment-wise classification problem leveraging unsupervised clustering. A single parameter executable hierarchical density-based algorithm and PointNet-based algorithms are chosen for segmentation and classification purposes. Using same baseline architecture, our segment-wise classification framework showed performance boosts of 1.12% and 7.66% for S3DIS and 3DFacilities compared to point-wise classification approach. Solving the nature of class imbalance problem and dataset augmentation ability are determined to be the contributing factors for showing superior results. Given semantic segmentation deep learning algorithm, our proposed framework provides an opportunity to improve the performance leveraging unsupervised method.