Attention-Based Multi-Scale Convolutional Neural Network (A+MCNN) for Multi-Class Classification in Road Images.

Elham Eslami,Hae-Bum Yun

doi:10.3390/s21155137

Abstract

Automated pavement distress recognition is a key step in smart infrastructure assessment. Advances in deep learning and computer vision have improved the automated recognition of pavement distresses in road surface images. This task remains challenging due to the high variation of defects in shapes and sizes, demanding a better incorporation of contextual information into deep networks. In this paper, we show that an attention-based multi-scale convolutional neural network (A+MCNN) improves the automated classification of common distress and non-distress objects in pavement images by (i) encoding contextual information through multi-scale input tiles and (ii) employing a mid-fusion approach with an attention module for heterogeneous image contexts from different input scales. A+MCNN is trained and tested with four distress classes (crack, crack seal, patch, pothole), five non-distress classes (joint, marker, manhole cover, curbing, shoulder), and two pavement classes (asphalt, concrete). A+MCNN is compared with four deep classifiers that are widely used in transportation applications and a generic CNN classifier (as the control model). The results show that A+MCNN consistently outperforms the baselines by 1∼26% on average in terms of the F-score. A comprehensive discussion is also presented regarding how these classifiers perform differently on different road objects, which has been rarely addressed in the existing literature.

Highlights

According to the 2021 America’s Infrastructure Report Card by the American Society of Civil Engineers (ASCE) [1], road infrastructures in the U.S.A. are graded D on average, showing poor pavement conditions
We presented a novel attention-based multi-scale convolutional neural network (A+multi-scale CNN (MCNN)) to improve the multi-class classification of asphalt and concrete images in pavement application
The A+MCNN was evaluated with a comprehensive pixel-level annotated dataset (UCF-PAVE 2017) collected by four different line-scanning cameras

Summary

Introduction

According to the 2021 America’s Infrastructure Report Card by the American Society of Civil Engineers (ASCE) [1], road infrastructures in the U.S.A. are graded D on average, showing poor pavement conditions. State and municipal departments of transportation (DOTs) conduct regular surveys to measure road conditions in terms of (i) cracking and patching, (ii) ride quality, and (iii) rutting. To measure cracking and patching, an image-based survey method is often adopted using high-speed line-scanning cameras mounted on a vehicle. Many technical challenges still exist regarding the accurate, reliable, and rapid detection, classification, and quantification of various distress and non-distress objects from images collected from large road networks. The challenges are mainly due to (i) variations in image collection conditions, such as camera calibrations, lighting conditions, and image qualities; (ii) variations in the appearance of road distress and non-distress objects in terms of shapes, sizes, orientations, textures, colors, etc.; (iii) the existence of grooving, oil or water stains, dirt or sand, skid marks, leaves, etc.; and (iv) the huge number of images to process for large road networks

Objectives

Methods

Results

Discussion

Conclusion