Simultaneous Precise Localization And Classification of metal rust defects for robotic-driven maintenance and prefabrication using residual attention U-Net

Abstract

We present a Simultaneous Precise Localization And Classification architecture employing a U-Net model (called SPLAC U-Net) for an automated simultaneous detection/localization of corrosion and recognition of rust grade from RGB images on metal structures. The proposed architecture is compatible with the innovative idea of automated maintenance and prefabrication since (i) rust detection is carried out at pixel level accuracy in a highly precise manner and (ii) localization/classification is performed simultaneously. Prior works on corrosion identification tend to focus either on localizing defected regions with bounding boxes, with low therefore localization precision, or on only visual inspection or classification and thus they are incompatible with the prefabrication concept. The proposed scheme consists of three layers. In the first layer, we utilize a lightweight deep U-Net model for semantically localizing corroded regions in images. In the second layer, we adopt a novel data projection scheme for refining the contours of the detected objects (i.e., defected regions) of the first layer, enhancing, therefore, localization precision. We find that the network after the boundary refinement process exceeds the typical deep learning methods performance on the accuracy, F1-score, and Intersection over Union (IoU) metrics. In the third and final layer of SPLAC U-Net, we extend the U-Net model by adding residual attention modules to efficiently classify multiple rust grades based on the ISO 8501-1 taxonomy. Experimental results using objective evaluation criteria, such as accuracy, precision, recall, F1-score, IoU, and Receiver Operating Characteristic Curve (ROC) show the outperformance of the proposed SLPAC U-Net in simultaneous detecting/localizing and classifying rust regions compared to other state-of-the-art methods exploiting either color distribution properties or traditional deep learning models. Lastly, a comparison of the computational performance shows that the total required time of the proposed SPLAC U-Net model (segmentation + projection refinement) is less than solely segmenting an image by using traditional deep learning models.