Deep Semantic Segmentation Network Research Articles

Combining Deep Semantic Segmentation Network and Graph Convolutional Neural Network for Semantic Segmentation of Remote Sensing Imagery

Although the deep semantic segmentation network (DSSN) has been widely used in remote sensing (RS) image semantic segmentation, it still does not fully mind the spatial relationship cues between objects when extracting deep visual features through convolutional filters and pooling layers. In fact, the spatial distribution between objects from different classes has a strong correlation characteristic. For example, buildings tend to be close to roads. In view of the strong appearance extraction ability of DSSN and the powerful topological relationship modeling capability of the graph convolutional neural network (GCN), a DSSN-GCN framework, which combines the advantages of DSSN and GCN, is proposed in this paper for RS image semantic segmentation. To lift the appearance extraction ability, this paper proposes a new DSSN called the attention residual U-shaped network (AttResUNet), which leverages residual blocks to encode feature maps and the attention module to refine the features. As far as GCN, the graph is built, where graph nodes are denoted by the superpixels and the graph weight is calculated by considering the spectral information and spatial information of the nodes. The AttResUNet is trained to extract the high-level features to initialize the graph nodes. Then the GCN combines features and spatial relationships between nodes to conduct classification. It is worth noting that the usage of spatial relationship knowledge boosts the performance and robustness of the classification module. In addition, benefiting from modeling GCN on the superpixel level, the boundaries of objects are restored to a certain extent and there are less pixel-level noises in the final classification result. Extensive experiments on two publicly open datasets show that DSSN-GCN model outperforms the competitive baseline (i.e., the DSSN model) and the DSSN-GCN when adopting AttResUNet achieves the best performance, which demonstrates the advance of our method.

Automatic Detection and Location of Weld Beads With Deep Convolutional Neural Networks

Welding quality detection is a critical link in modern manufacturing, and the weld bead location is a prerequisite for the high-precision assessment of welding quality. It is generally necessary for weld bead detection to be accomplished in the context of complex industrial environments. However, conventional detection and location methods based on specific detection conditions or prior knowledge lack accuracy and adaptability. To precisely detect and locate the weld beads in real industrial environments, a novel weld bead detection and location algorithm is proposed based on deep convolutional neural networks. Because there is no open data set of weld beads and the samples in real industrial applications are insufficient for effective model training of the deep convolutional neural network, a novel data augmentation method based on a deep semantic segmentation network is proposed to increase the sample diversity and enlarge the data set. Then, a dynamic sample updating strategy is put forward to cover more welding situations. Finally, faced with the weak-feature and weak-texture characteristics of weld beads, a simplified YOLOV3 model is proposed to realize end-to-end weld bead location. Experiments demonstrate that the proposed method could effectively satisfy the robustness and precision requirements for weld bead detection and location combined with a deep semantic segmentation network and simplified YOLOV3 model.

SCAU-Net: Spatial-Channel Attention U-Net for Gland Segmentation.

With the development of medical technology, image semantic segmentation is of great significance for morphological analysis, quantification, and diagnosis of human tissues. However, manual detection and segmentation is a time-consuming task. Especially for biomedical image, only experts are able to identify tissues and mark their contours. In recent years, the development of deep learning has greatly improved the accuracy of computer automatic segmentation. This paper proposes a deep learning image semantic segmentation network named Spatial-Channel Attention U-Net (SCAU-Net) based on current research status of medical image. SCAU-Net has an encoder-decoder-style symmetrical structure integrated with spatial and channel attention as plug-and-play modules. The main idea is to enhance local related features and restrain irrelevant features at the spatial and channel levels. Experiments on the gland dataset GlaS and CRAG show that the proposed SCAU-Net model is superior to the classic U-Net model in image segmentation task, with 1% improvement on Dice score and 1.5% improvement on Jaccard score.

AVNet: A retinal artery/vein classification network with category-attention weighted fusion

Background and Objective: Automatic artery/vein (A/V) classification in retinal images is of great importance in detecting vascular abnormalities, which may provide biomarkers for early diagnosis of many systemic diseases. It is intuitive to apply popular deep semantic segmentation network for A/V classification. However, the model is required to provide powerful representation ability since vessel is much more complex than general objects. Moreover, deep network may lead to inconsistent classification results for the same vessel due to the lack of structured optimization objective.Methods: In this paper, we propose a novel segmentation network named AVNet, which effectively enhances the classification ability of the model by integrating category-attention weighted fusion (CWF) module, significantly improving the pixel-level A/V classification results. Then, a graph based vascular structure reconstruction (VSR) algorithm is employed to reduce the segment-wise inconsistency, verifying the effect of the graph model on noisy vessel segmentation results.Results: The proposed method has been verified on three datasets, i.e. DRIVE, LES-AV and WIDE. AVNet achieves pixel-level accuracies of 90.62%, 90.34%, and 93.16%, respectively, and VSR further improves the performance by 0.19%, 1.85% and 0.64%, achieving the state-of-the-art results on these three datasets.Conclusion: The proposed method achieves competitive performance in A/V classification task.

Deep Learning-Based Detection of Pigment Signs for Analysis and Diagnosis of Retinitis Pigmentosa.

Ophthalmological analysis plays a vital role in the diagnosis of various eye diseases, such as glaucoma, retinitis pigmentosa (RP), and diabetic and hypertensive retinopathy. RP is a genetic retinal disorder that leads to progressive vision degeneration and initially causes night blindness. Currently, the most commonly applied method for diagnosing retinal diseases is optical coherence tomography (OCT)-based disease analysis. In contrast, fundus imaging-based disease diagnosis is considered a low-cost diagnostic solution for retinal diseases. This study focuses on the detection of RP from the fundus image, which is a crucial task because of the low quality of fundus images and non-cooperative image acquisition conditions. Automatic detection of pigment signs in fundus images can help ophthalmologists and medical practitioners in diagnosing and analyzing RP disorders. To accurately segment pigment signs for diagnostic purposes, we present an automatic RP segmentation network (RPS-Net), which is a specifically designed deep learning-based semantic segmentation network to accurately detect and segment the pigment signs with fewer trainable parameters. Compared with the conventional deep learning methods, the proposed method applies a feature enhancement policy through multiple dense connections between the convolutional layers, which enables the network to discriminate between normal and diseased eyes, and accurately segment the diseased area from the background. Because pigment spots can be very small and consist of very few pixels, the RPS-Net provides fine segmentation, even in the case of degraded images, by importing high-frequency information from the preceding layers through concatenation inside and outside the encoder-decoder. To evaluate the proposed RPS-Net, experiments were performed based on 4-fold cross-validation using the publicly available Retinal Images for Pigment Signs (RIPS) dataset for detection and segmentation of retinal pigments. Experimental results show that RPS-Net achieved superior segmentation performance for RP diagnosis, compared with the state-of-the-art methods.

Two-pathway anti-interference neural network based on the retinal perception mechanism for classification of remote sensing images from unmanned aerial vehicles

The ultrahigh resolution of unmanned aerial vehicle (UAV) remote sensing images and tilting photography with multiple perspectives provide complete and detailed ground observation data for various engineering applications. However, noise and interference information make learning the typical features of ground objects difficult for current deep learning semantic segmentation networks. The hierarchical cognitive structure of human vision and the information transmission modes of retinal cone and rod cells were used to design a two-pathway anti-interference network for retinal perception mechanism simulation (RPMS). In the first pathway, the hierarchical cognition of cone cells was simulated by a one-to-one connected multiscale dilated convolution structure. In the second pathway, the hierarchical cognition of rod cells was simulated by a multiscale pyramid structure with many-to-one connections. With the one-to-one connection, the ability of RPMS to recognize detailed edges was strengthened. Furthermore, the many-to-one connection helped RPMS resist the disturbance from noise and interference. By combining the feature maps of the two paths, RPMS exhibited stronger noise resistance, better texture recognition, and better detail recognition compared with other semantic segmentation networks in the classification experiments. Thus this technique is suitable for UAV remote sensing image classification and has a broad application potential.

Cascaded deep learning-based efficient approach for license plate detection and recognition

Automatic license plate (ALP) detection and recognition is an important task for both traffic surveillance and parking management systems, as well as being crucial to maintaining the flow of modern civic life. Various ALP detection and recognition methods have been proposed to date. These methods generally use various image processing and machine learning techniques. In this paper, a cascaded deep learning approach is proposed in order to construct an efficient ALP detection and recognition system for the vehicles of northern Iraq. The license plates in northern Iraq contain three regions, namely a plate number, a city region, and a country region. The proposed method initially employs several preprocessing techniques such as Gaussian filtering and adaptive image contrast enhancement to make the input images more suited to further processing. Then, a deep semantic segmentation network is used in order to determine the three license plate regions of the input image. Segmentation is then carried out via deep encoder-decoder network architecture. The determined license plate regions are fed into two separate convolutional neural network (CNN) models for both Arabic number recognition and the city determination. For Arabic number recognition, an end-to-end CNN model was constructed and trained, whilst for the city recognition, a pretrained CNN model was further fine-tuned. A new license plate dataset was also constructed and used in the experimental works of the study. The performance of the proposed method was evaluated both in terms of detection and recognition. For detection, recall, precision and F-measure scores were used, and for recognition, classification accuracy was used. The obtained results showed the proposed method to be efficient in both license plate detection and recognition. The calculated recall, precision and F-measure scores were 92.10%, 94.43%, and 91.01%, respectively. Moreover, the classification accuracies for Arabic numbers and city labels were shown to be 99.37% and 92.26%, respectively.

Lightweight Prediction and Boundary Attention-Based Semantic Segmentation for Road Scene Understanding

Semantic segmentation is one of the most commonly used techniques for road scene understanding. Recently developed deep learning-based semantic segmentation networks are typically based on the encoder-decoder structure and have made great progress in road scene understanding. However, these conventional networks still encounter difficulties in recovering spatial details. To overcome this problem, we introduce a lightweight prediction and boundary-aware refinement module that can hierarchically refine the segmentation results with spatial details. The proposed refinement module has two attention units called the upper-level prediction attention unit and the upper-level boundary attention unit. The upper-level prediction attention unit emphasizes the features in the regions that need to be refined by using predicted class probability from the upper-level, whereas the upper-level boundary attention unit focuses on the features near the semantic boundary of the upper-level segmentation result. By using the proposed prediction and boundary-aware refinement module in the decoder network, the segmentation result can gradually be improved in a top-down manner to a finer and more complete one. Experimental results on the Cityscapes and CamVid datasets demonstrate that the proposed prediction and boundary attention-based refinement module can achieve considerable performance improvement in segmentation accuracy with a marginal increase in computational complexity.

Multistage GAN for Fabric Defect Detection.

Fabric defect detection is an intriguing but challenging topic. Many methods have been proposed for fabric defect detection, but these methods are still suboptimal due to the complex diversity of both fabric textures and defects. In this paper, we propose a generative adversarial network (GAN)-based framework for fabric defect detection. Considering existing challenges in real-world applications, the proposed fabric defect detection system is capable of learning existing fabric defect samples and automatically adapting to different fabric textures during different application periods. Specifically, we customize a deep semantic segmentation network for fabric defect detection that can detect different defect types. Furthermore, we attempted to train a multistage GAN to synthesize reasonable defects in new defect-free samples. First, a texture-conditioned GAN is trained to explore the conditional distribution of defects given different texture backgrounds. Given a novel fabric, we aim to generate reasonable defective patches. Then, a GAN-based fusion network fuses the generated defects to specific locations. Finally, the well-trained multistage GAN continuously updates the existing fabric defect datasets and contributes to the fine-tuning of the semantic segmentation network to better detect defects under different conditions. Comprehensive experiments on various representative fabric samples are conducted to verify the detection performance of our proposed method.

Concrete crack detection using context‐aware deep semantic segmentation network

AbstractComputer‐vision and deep‐learning techniques are being increasingly applied to inspect, monitor, and assess infrastructure conditions including detection of cracks. Traditional vision‐based methods to detect cracks lack accuracy and generalization to work on complicated infrastructural conditions. This paper presents a novel context‐aware deep convolutional semantic segmentation network to effectively detect cracks in structural infrastructure under various conditions. The proposed method applies a pixel‐wise deep semantic segmentation network to segment the cracks on images with arbitrary sizes without retraining the prediction network. Meanwhile, a context‐aware fusion algorithm that leverages local cross‐state and cross‐space constraints is proposed to fuse the predictions of image patches. This method is evaluated on three datasets: CrackForest Dataset (CFD) and Tomorrows Road Infrastructure Monitoring, Management Dataset (TRIMMD) and a Customized Field Test Dataset (CFTD) and achieves Boundary F1 (BF) score of 0.8234, 0.8252, and 0.7937 under 2‐pixel error tolerance margin in CFD, TRIMMD, and CFTD, respectively. The proposed method advances the state‐of‐the‐art performance of BF score by approximately 2.71% in CFD, 1.47% in TRIMMD, and 4.14% in CFTD. Moreover, the averaged processing time of the proposed system is 0.7 s on a typical desktop with Intel® Quad‐Core™ i7‐7700 CPU@3.6 GHz Processor, 16GB RAM and NVIDIA GeForce GTX 1060 6GB GPU for an image of size 256 × 256 pixels.

Smallholder Crop Area Mapped with a Semantic Segmentation Deep Learning Method

The growing population in China has led to an increasing importance of crop area (CA) protection. A powerful tool for acquiring accurate and up-to-date CA maps is automatic mapping using information extracted from high spatial resolution remote sensing (RS) images. RS image information extraction includes feature classification, which is a long-standing research issue in the RS community. Emerging deep learning techniques, such as the deep semantic segmentation network technique, are effective methods to automatically discover relevant contextual features and get better image classification results. In this study, we exploited deep semantic segmentation networks to classify and extract CA from high-resolution RS images. WorldView-2 (WV-2) images with only Red-Green-Blue (RGB) bands were used to confirm the effectiveness of the proposed semantic classification framework for information extraction and the CA mapping task. Specifically, we used the deep learning framework TensorFlow to construct a platform for sampling, training, testing, and classifying to extract and map CA on the basis of DeepLabv3+. By leveraging per-pixel and random sample point accuracy evaluation methods, we conclude that the proposed approach can efficiently obtain acceptable accuracy (Overall Accuracy = 95%, Kappa = 0.90) of CA classification in the study area, and the approach performs better than other deep semantic segmentation networks (U-Net/PspNet/SegNet/DeepLabv2) and traditional machine learning methods, such as Maximum Likelihood (ML), Support Vector Machine (SVM), and RF (Random Forest). Furthermore, the proposed approach is highly scalable for the variety of crop types in a crop area. Overall, the proposed approach can train a precise and effective model that is capable of adequately describing the small, irregular fields of smallholder agriculture and handling the great level of details in RGB high spatial resolution images.