Deep Semantic Segmentation Network Research Articles

Computer vision-based statistical crack quantification for concrete structures

This paper newly proposed a computer vision-based crack quantification algorithm using a statistical approach. Recently, high-resolution digital images have been effectively utilized for automated crack width and length evaluation on concrete structures. However, cracks are often difficult to accurately measure by randomly distributed and complex shapes. To overcome the technical limitation, a novel statistical crack quantification algorithm is proposed and experimentally validated through concrete structures in this paper. First, cracks on digital images are automatically detected using a deep semantic segmentation network. Then, multi-branched cracks are separated into single cracks through crack map generation. Each separated crack length and width is calculated by the Euclidean distance transform algorithm. Finally, crack width is presented as a representative value with a statistical confidence interval. The quantitative crack evaluation results for width and length were successfully compared with the actual field measurement values by average differences of 18.07% and − 26.28%, respectively.

Semi-Supervised Land Cover Classification of Remote Sensing Imagery Using CycleGAN and EfficientNet

Image classification of very high resolution (VHR) images is a fundamental task in the remote sensing domain for various applications, such as land cover mapping, vegetation mapping, and urban planning. Recently, deep learning-based semantic segmentation networks demonstrated the promising performance for pixel-level image classification. However, deep learning-based approaches are generally limited by the requirement of a sufficient amount of labeled data to obtain stable accuracy, and acquiring reference labels of remotely-sensed VHR images is very labor-extensive and expensive. Hence, this paper applied a semi-supervised learning-based CycleGAN and EfficientNet for VHR remote sensing image classification to overcome this problem. The proposed method achieved the highest accuracy than the other benchmarks. The largest increase in accuracy was observed in a test site containing complex objects due to the regularization effect of the semi-supervised method using unlabeled data. Moreover, results indicated that a relatively sufficient amount of unlabeled data compared with labeled data are required to increase the classification accuracy by controlling the amount of labeled and unlabeled data. Finally, we verified that the semi-supervised method returned significantly improved results irrespective of the three classification network structures, displaying the applicability of the method for semi-supervised image classification on remotely-sensed VHR images.

Forest Segmentation with Spatial Pyramid Pooling Modules: A Surveillance System Based on Satellite Images

The global deforestation rate continues to worsen each year, and will eventually lead to various negative consequences for humans and the environment. It is essential to develop an effective forest monitoring system to detect any changes in forest areas, in particular, by monitoring the progress of forest conservation efforts. In general, changes in forest status are difficult to annotate manually, whereby the boundaries can be small in size or hard to discern, especially in areas that are bordering residential areas. The previously implemented forest monitoring systems were ineffective due to their use of low-resolution satellite images and the inefficiency of drone-based data that offer a limited field of view. Most government agencies also still rely on manual annotation, which makes the monitoring process time-consuming, tedious, and expensive. Therefore, the goal of this study is to overcome these issues by developing a forest monitoring system that relies on a robust deep semantic segmentation network that is capable of discerning forest boundaries automatically, so that any changes over the years can be tracked. The backbone of this system is based on satellite imaging supplied to a modified U-Net deep architecture to incorporate multi-scale modules to deliver the semantic segmentation output. A dataset of 6048 Landsat-8 satellite sub-images that were taken from eight land parcels of forest areas was collected and annotated, and then further divided into training and testing datasets. The novelty of this system is the optimal integration of the spatial pyramid pooling (SPP) mechanism into the base model, which allows the model to effectively segment forest areas regardless of their varying sizes, patterns, and colors. To investigate the impact of SPP on the forest segmentation system, a set of experiments was conducted by integrating several variants of SPP ranging from two to four parallel paths with different combinations of pooling kernel size, placed at the bottleneck layer of the U-Net model. The results demonstrated the effectiveness of the SPP module in improving the performance of the forest segmentation system by 2.57%, 6.74%, and 7.75% in accuracy (acc), intersection over union (IoU), and F1-score (F1score), respectively. The best SPP variant consists of four parallel paths with a combination of pooling kernel sizes of 2×2, 4×4, 6×6, and 8×8 pixels that produced the highest acc, IoU, and F1score of 86.71%, 75.59%, and 82.88%, respectively. As a result, the multi-scale module improved the proposed forest segmentation system, making it a highly useful system for government and private agencies in tracking any changes in forest areas.

A deep learning semantic segmentation network with attention mechanism for concrete crack detection

In this research, an attention-based feature fusion network (AFFNet), with a backbone residual network (ResNet101) enhanced with two attention mechanism modules, is proposed for automatic pixel-level detection of concrete crack. In particular, the inclusion of attention mechanism modules, for example, the vertical and horizontal compression attention module (VH-CAM) and the efficient channel attention upsample module (ECAUM), is to enable selective concentration on the crack feature. The VH-CAM generates a feature map integrating pixel-level information in vertical and horizontal directions. The ECAUM applied on each decoder layer combines efficient channel attention (ECA) and feature fusion, which can provide rich contextual information as guidance to help low-level features recover crack localization. The proposed model is evaluated on the test dataset and the results reach 84.49% for mean intersection over union (MIoU). Comparison with other state-of-the-art models proves high efficiency and accuracy of the proposed method.

Deep Learning and Laser-Based 3-D Pixel-Level Rail Surface Defect Detection Method

Rail surface defect inspection is of particular importance in modern railways. Accurate and efficient surface defect detection approaches support optimized maintenance. This enables safe operation of the railway network. However, the scale and harsh working environments of the railway still pose challenges to existing manual and vision-based inspection methods. Inspired by recent advances in laser measurement and deep learning in computer vision, this paper proposes a laser-based 3D pixel-level rail surface defect detection method that combines high-precision laser measurement data with the concept of deep semantic segmentation. In the proposed method, the rail surface is first measured in 3D using a low-cost 2D laser triangulation sensor. Then, a new deep semantic segmentation network is introduced. The network is composed of a fully convolutional segmentation module and two symmetric mapping modules, which can take 3D laser measurement data as input and output 3D pixel-level defect detection results in an end-to-end manner. The modular design of the network allows the use of various segmentation modules for different applications or scenarios. Experiments on a 3D rail dataset demonstrate the feasibility of the proposed method with a pixel-level detection accuracy measured by mean Intersection over Union (mIoU) of up to 87.9%. The 3D output provides not only location and boundary information but also the 3D characterization of defects, giving an essential reference for further defect management and repair tasks.

Optimizing Spatial Relationships in GCN to Improve the Classification Accuracy of Remote Sensing Images

Semantic segmentation of remote sensing images is one of the core tasks of remote sensing image interpretation. With the continuous development of artificial intelligence technology, the use of deep learning methods for interpreting remote-sensing images has matured. Existing neural networks disregard the spatial relationship between two targets in remote sensing images. Semantic segmentation models that combine convolutional neural networks (CNNs) and graph convolutional neural networks (GCNs) cause a lack of feature boundaries, which leads to the unsatisfactory segmentation of various target feature boundaries. In this paper, we propose a new semantic segmentation model for remote sensing images (called DGCN hereinafter), which combines deep semantic segmentation networks (DSSN) and GCNs. In the GCN module, a loss function for boundary information is employed to optimize the learning of spatial relationship features between the target features and their relationships. A hierarchical fusion method is utilized for feature fusion and classification to optimize the spatial relationship information in the original feature information. Extensive experiments on ISPRS 2D and DeepGlobe semantic segmentation datasets show that compared with the existing semantic segmentation models of remote sensing images, the DGCN significantly optimizes the segmentation effect of feature boundaries, effectively reduces the noise in the segmentation results and improves the segmentation accuracy, which demonstrates the advancements of our model.

Geographic knowledge graph-guided deep semantic segmentation network for remote sensing imagery

Geographic knowledge graph-guided deep semantic segmentation network for remote sensing imagery

Semantic Segmentation and 3D Reconstruction of Concrete Cracks

Damage assessment of concrete structures is necessary to prevent disasters and ensure the safety of infrastructure such as buildings, sidewalks, dams, and bridges. Cracks are among the most prominent damage types in such structures. In this paper, a solution is proposed for identifying and modeling cracks in concrete structures using a stereo camera. First, crack pixels are identified using deep learning-based semantic segmentation networks trained on a custom dataset. Various techniques for improving the accuracy of these networks are implemented and evaluated. Second, modifications are applied to the stereo camera’s calibration model to ensure accurate estimation of the systematic errors and the orientations of the cameras. Finally, two 3D reconstruction methods are proposed, one of which is based on detecting the dominant structural plane surrounding the crack, while the second method focuses on stereo inference. The experiments performed on close-range images of complex and challenging scenes show that structural cracks can be identified with a precision of 96% and recall of 85%. In addition, an accurate 3D replica of cracks can be produced with an accuracy higher than 1 mm, from which the cracks’ size and other geometric features can be deduced.

Semantic Cameras for 360-Degree Environment Perception in Automated Urban Driving

The European UP-Drive project addresses transportation-related challenges by providing key contributions that enable fully automated vehicle navigation and parking in complex urban areas, which results in a safer, inclusive, affordable and environmentally friendly transportation system. For this purpose, the project consortium developed a prototype electrical vehicle equipped with cameras and LiDARs sensors that is capable to autonomously drive around the city and find available parking spots. In UP-Drive, we created an accurate, robust and redundant multi-modal environment perception system that provides 360° coverage around the vehicle. This paper summarizes the work of the project related to the surround view semantic perception using fisheye and narrow field-of-view semantic virtual cameras. Deep learning-based semantic, instance and panoptic segmentation networks, which satisfy requirements in accuracy and efficiency have been developed and integrated into the final prototype. The UP-Drive automated vehicle has been successfully demonstrated in urban areas after extensive experiments and numerous field tests.

Strip Attention Networks for Road Extraction

In recent years, deep learning methods have been widely used for road extraction in remote sensing images. However, the existing deep learning semantic segmentation networks generally show poor continuity in road segmentation due to the high-class similarity between roads and buildings surrounding roads in remote sensing images, and the existence of shadows and occlusion. To deal with this problem, this paper proposes strip attention networks (SANet) for extracting roads in remote sensing images. Firstly, a strip attention module (SAM) is designed to extract the contextual information and spatial position information of the roads. Secondly, a channel attention fusion module (CAF) is designed to fuse low-level features and high-level features. The network is trained and tested using the CITY-OSM dataset, DeepGlobe road extraction dataset, and CHN6-CUG dataset. The test results indicate that SANet exhibits excellent road segmentation performance and can better solve the problem of poor road segmentation continuity compared with other networks.

Formation and propagation of cracks in RRP Nb3Sn wires studied by deep learning applied to x-ray tomography

This paper reports a novel non-destructive and non-invasive method to investigate crack formation and propagation in high-performance Nb3Sn wires by combining x-ray tomography and deep learning networks. The next generation of high field magnet applications relies on the development of new Nb3Sn wires capable to withstand the large stresses generated by Lorentz forces during magnets operation. These stresses can cause a permanent reduction of the transport properties generated by residual deformation of the Nb3Sn crystal lattice as well as the formation of cracks in the brittle Nb3Sn filaments. Studies for the development of the high luminosity LHC (HL-LHC) upgrade showed that nominal transverse compressive stresses above 150 MPa may be sufficient to generate cracks in the wires. In the case of fusion magnets, wires experience periodic bending due to the electro-magnetic cycles of the reactor which over time may induce wire deformation and filament cracks. Therefore, it has become essential to develop a quantitative method for the characterization of crack formation and propagation under compressive loads. The x-ray tomographic data of a series of restacked-rod-process (RRP) Nb3Sn wires was acquired at the micro-tomography beamline ID19 of the European Synchrotron Radiation Facility (ESRF), after intentionally inducing a broad spectrum of cracks in the Nb3Sn sub-elements. The samples were submitted to transvers compressive stresses, with and without epoxy impregnation, at different pressures, up to 238 MPa. The resulting tomographic images were analysed by means of deep learning semantic segmentation networks, using U-net, a convolutional neural network (CNN), to identify and segment cracks inside the wires. The trained CNN was able to analyse large volumes of tomographic data, thus enabling a systematic approach for investigating the mechanical damages in Nb3Sn wires. We will show the complete three-dimensional reconstruction of various cracks and discuss their impact on the electro-mechanical performance of the analysed wires.

Deep edge enhancement-based semantic segmentation network for farmland segmentation with satellite imagery

The participation of insured smallholders and involvement of related requirements have been emerging and increasing with the popularization and promotion of agricultural insurance. Compared with traditional field surveys, remote sensing and deep learning technologies, with their rapid development in recent years, have provided an automatic and effective means for underwriting control in the agricultural insurance industry. Moreover, “smallholder-wise underwriting” is intensively expected by the local government and insurance companies to ensure precise claim settlements. In this process, farmland segmentation is a crucial and fundamental step for smallholder farmland claim settlements. However, segmentation of smallholder farmland parcels from satellite images remains a challenge, as the ridges and roads between farmland parcels are usually narrow and display confusing characteristics with their neighboring farmlands. This challenge causes undersegmentation on the farmland boundaries, leading to inaccurate and incomplete farmland parcel recognition. In this study, we aim to solve this problem by proposing a novel Deep Edge Enhancement Semantic Segmentation Network to refine parcels’ boundary segmentation and improve the closure level of the farmland segmentation. We designed a framework for farmland edge enhancement through pseudo road label generation and model fusion. The proposed network aggregates feature extraction for ridge and road segmentation to improve its performance on farmland parcel recognition. Furthermore, we found that the commonly used image segmentation evaluation metric, such as pixel-wise–based mean Intersection over Union, cannot objectively reflect the effectiveness of the smallholder farmland segmentation. Therefore, we proposed two block-wise farmland evaluation metrics that are consistent with the practical evaluation rules and requirements of farmland segmentation at the parcel level. We implemented experiments in the study area of Jiaxiang County in northern China using the GaoFen-1 4-band multispectral images (red, green, blue, and near-infrared) at a spatial resolution of 2 m. Experimental results demonstrated the effectiveness of our method, which outperformed the baseline DeepLabv3+ network.

Segmentation and weight prediction of grape ear based on SFNet-ResNet18

ABSTRACT In this paper, the segment and weight prediction problems are investigated for ear of grape based on deep learning technologies. The image datum is collected from ZaoHeiBao grape in a greenhouse by camera. The grape ear target segmentation model is constructed by cross combining three backbone networks (ResNet18, ResNet50, and ResNet101) and four deep learning semantic segmentation networks (SFNet, GCNet, EMANet, and Deeplabv3). The experimental results show that for the SFNet-ResNet18 model, whose structural size is 52.68MB, the mean Intersection over Union (mIoU) is , the mean Pixel Accuracy (mPA) is , and the average segmentation speed of the image () is 0.217s. Therefore, the performance of the SFNet-ResNet18 model outperforms other combined network models and is selected to segment grape ears. Furthermore, on the basis of the segmentation results of grape ears by using the SFNet-ResNet18 model, the grape ear weight is predicted by adopting fractional regression model. The value of is 0.8903, which means that the selected model could accurately predict the weight of grape ears. The proposed method can not only segment the grape ears and accurately predict the weight of the grape ears, but also provide theoretical and technical support for grape yield prediction.

Rice mapping based on Sentinel-1 images using the coupling of prior knowledge and deep semantic segmentation network: A case study in Northeast China from 2019 to 2021

Usually, continuous effective optical observation data collected from a single-sensor within one year is not available, most studies conducted rice mapping in Northeast China using optical remote sensing images collected from multi-years and multi-sensors. Meanwhile, the accuracies of mapping results are limited by the employed traditional machine learning algorithms, which cannot fully exploit the deep abstract features contained in the multi-temporal images. Consequently, this paper developed a deep learning framework for large-scale rice mapping based on multi-temporal Sentinel-1 images in practice. It was achieved via semantic segmentation based on coupling of U-Net and prior knowledge (i.e., the coupled U-Net). Classification experiments were conducted in regions covered by complete and incomplete multi-temporal Sentinel-1 images to validate advantages of the coupled U-Net. Besides, feature visualization experiments were conducted, and the results showed that the coupled U-Net could further robustly learn deep abstract feature that was more suitable to express land covers. Finally, rice maps of Northeast China in different years were produced by the coupled U-Net and Sentinel-1 images. For the results of 2021, both rice producer’s and user’s accuracies exceeded 85%, while its overall accuracy and F1-score were higher than 0.9. For the results of 2019 and 2020, the rice areas of Northeast China extracted from Sentinel-1 images were 4.0% and 4.9% less than those of the subnational statistics data. This study provided a viable option toward practical large-scale rice mapping based on multi-temporal Sentinel-1 images through the coupling of prior knowledge and deep learning technology.

SAR ship target detection method based on CNN structure with wavelet and attention mechanism.

Ship target detection in synthetic aperture radar (SAR) images is an important application field. Due to the existence of sea clutter, especially the SAR imaging in huge wave area, SAR images contain a lot of complex noise, which brings great challenges to the effective detection of ship targets in SAR images. Although the deep semantic segmentation network has been widely used in the detection of ship targets in recent years, the global information of the image cannot be fully utilized. To solve this problem, a new convolutional neural network (CNN) method based on wavelet and attention mechanism was proposed in this paper, called the WA-CNN algorithm. The new method uses the U-Net structure to construct the network, which not only effectively reduces the depth of the network structure, but also significantly improves the complexity of the network. The basic network of WA-CNN algorithm consists of encoder and decoder. Dual tree complex wavelet transform (DTCWT) is introduced into the pooling layer of the encoder to smooth the speckle noise in SAR images, which is beneficial to preserve the contour structure and detail information of the target in the feature image. The attention mechanism theory is added into the decoder to obtain the global information of the ship target. Two public SAR image datasets were used to verify the proposed method, and good experimental results were obtained. This shows that the method proposed in this article is effective and feasible.

Automated Apple Recognition System Using Semantic Segmentation Networks with Group and Shuffle Operators

Apples are one of the most consumed fruits, and they require efficient harvesting procedures to remains in optimal states for a longer period, especially during transportation. Therefore, automation has been adopted by many orchard operators to help in the harvesting process, which includes apple localization on the trees. The de facto sensor that is currently used for this task is the standard camera, which can capture wide view information of various apple trees from a reasonable distance. Therefore, this paper aims to produce the output mask of the apple locations on the tree automatically by using a deep semantic segmentation network. The network must be robust enough to overcome all challenges of shadow, surrounding illumination, size variations, and occlusion to produce accurate pixel-wise localization of the apples. A high-resolution deep architecture is embedded with an optimized design of group and shuffle operators (GSO) to produce the best apple segmentation network. GSO allows the network to reduce the dependency on a few sets of dominant convolutional filters by forcing each smaller group to contribute effectively to the task of extracting optimal apple features. The experimental results show that the proposed network, GSHR-Net, with two sets of group convolution applied to all layers produced the best mean intersection over union of 0.8045. The performance has been benchmarked with 11 other state-of-the-art deep semantic segmentation networks. For future work, the network performance can be increased by integrating synthetic augmented data to further optimize the training phase. Moreover, spatial and channel-based attention mechanisms can also be explored by emphasizing some strategic locations of the apples, which makes the recognition more accurate.

DRs-UNet: A Deep Semantic Segmentation Network for the Recognition of Active Landslides from InSAR Imagery in the Three Rivers Region of the Qinghai–Tibet Plateau

At present, Synthetic Aperture Radar Interferometry (InSAR) has been an important technique for active landslides recognition in the geological survey field. However, the traditional interpretation method through human–computer interaction highly relies on expert experience, which is time-consuming and subjective. To solve the problem, this study designed an end-to-end semantic segmentation network, called deep residual shrinkage U-Net (DRs-UNet), to automatically extract potential active landslides in InSAR imagery. The proposed model was inspired by the structure of U-Net and adopted a residual shrinkage building unit (RSBU) as the feature extraction block in its encoder part. The method of this study has three main advantages: (1) The RSBU in the encoder part incorporated with soft thresholding can reduce the influence of noise from InSAR images. (2) The residual connection of the RSBU makes the training of the network easier and accelerates the convergency process. (3) The feature fusion of the corresponding layers between the encoder and decoder effectively improves the classification accuracy. Two widely used networks, U-Net and SegNet, were trained under the same experiment environment to compare with the proposed method. The experiment results in the test set show that our method achieved the best performance; specifically, the F1 score is 1.48% and 4.1% higher than U-Net and SegNet, which indicates a better balance between precision and recall. Additionally, our method has the best IoU score of over 90%. Furthermore, we applied our network to a test area located in Zhongxinrong County along Jinsha River where landslides are highly evolved. The quantitative evaluation results prove that our method is effective for the automatic recognition of potential active landslide hazards from InSAR imagery.

Combining deep learning and ontology reasoning for remote sensing image semantic segmentation

Because of its wide potential applications, remote sensing (RS) image semantic segmentation has attracted increasing research interest in recent years. Until now, deep semantic segmentation network (DSSN) has achieved a certain degree of success on semantic segmentation of RS imagery and can obviously outperform the traditional methods based on hand-crafted features. As a classic data-driven technique, DSSN can be trained by an end-to-end mechanism and is competent for employing low-level and mid-level cues (i.e., the discriminative image structure) to understand RS images. However, its interpretability and reliability are poor due to the nature weakness of the data-driven deep learning methods. By contrast, human beings have an excellent inference capacity and can reliably interpret RS imagery with the basic RS domain knowledge. Ontological reasoning is an ideal way to imitate and employ the domain knowledge of human beings. However, it is still rarely explored and adopted in the RS domain. As a solution of the aforementioned critical limitation of DSSN, this study proposes a collaboratively boosting framework (CBF) to combine the data-driven deep learning module and knowledge-guided ontology reasoning module in an iterative manner. The deep learning module adopts the DSSN architecture and takes the integration of the original image and inferred channels as the input of the DSSN. In addition, the ontology reasoning module is composed of intra- and extra-taxonomy reasoning. More specifically, the intra-taxonomy reasoning directly corrects misclassifications of the deep learning module based on the domain knowledge, which is the key to improve the classification performance. The extra-taxonomy reasoning aims to generate the inferred channels beyond the current taxonomy to improve the discriminative performance of DSSN in the original RS image space. On the one hand, benefiting from the referred channels from the ontology reasoning module, the deep learning module using the integration of the original image and referred channels can achieve better classification performance than only using the original image. On the other hand, better classification results from the deep learning module further improve the performance of the ontology reasoning module. As a whole, the deep learning and ontology reasoning modules are mutually boosted in the iterations. Extensive experiments on two publicly open RS datasets such as UCM and ISPRS Potsdam show that our proposed CBF can outperform the competitive baselines with a large margin.

SRSe-Net: Super-Resolution-Based Semantic Segmentation Network for Green Tide Extraction

Due to the phenomenon of mixed pixels in low-resolution remote sensing images, the green tide spectral features with low Enteromorpha coverage are not obvious. Super-resolution technology based on deep learning can supplement more detailed information for subsequent semantic segmentation tasks. In this paper, a novel green tide extraction method for MODIS images based on super-resolution and a deep semantic segmentation network was proposed. Inspired by the idea of transfer learning, a super-resolution model (i.e., WDSR) is first pre-trained with high spatial resolution GF1-WFV images, and then the representations learned in the GF1-WFV image domain are transferred to the MODIS image domain. The improvement of remote sensing image resolution enables us to better distinguish the green tide patches from the surrounding seawater. As a result, a deep semantic segmentation network (SRSe-Net) suitable for large-scale green tide information extraction is proposed. The SRSe-Net introduced the dense connection mechanism on the basis of U-Net and replaces the convolution operations with dense blocks, which effectively obtained the detailed green tide boundary information by strengthening the propagation and reusing features. In addition, the SRSe-Net reducs the pooling layer and adds a bridge module in the final stage of the encoder. The experimental results show that a SRSe-Net can obtain more accurate segmentation results with fewer network parameters.

Unsupervised Semantic Segmentation of Aerial Images With Application to UAV Localization

Recent advances and applications of aerial image semantic segmentation have yielded an increase in their use in day-to-day tasks. However, state-of-art algorithms, composed mostly of deep semantic segmentation networks (DSSNs), may not be suitable for application domains in which labels (targeted output masks) are scarce. This work proposes a fully unsupervised semantic segmentation method able to find the appropriate number of semantics labels (yielded from clustering) without the need of a previously annotated dataset. Once the semantic labels are identified, a classifier trained from these labels can be used to assign semantics to new input images. In contrast to other clustering-based segmentation methods, our method does not require any input parameter to find the clustering partition that best accommodates data. Empirical results on two real datasets showed that the use of semantics in a template-matching-based unmanned aerial vehicle (UAV) localization pipeline reduced the overall position estimation error.