Pavement damage detection is essential for subsequent road maintenance decisions. However, recent detection networks have low accuracy and fail to detect most diseases on the road, which means that testing is very inefficient. Therefore, this study uses the unmanned aerial vehicle (UAV) road damage database and describes a multi-level attention mechanism called Multi-level Attention Block (MLAB) to strengthen the utilization of essential features by the You Only Look Once version 3 (YOLO v3). Adding MLAB between the backbone and feature fusion parts effectively increases the mAP value of the proposed network to 68.75%, while the accuracy of the original network is only 61.09%. The network is able to detect longitudinal cracks, transverse cracks, repairs, and potholes with high accuracy, and significantly improves the accuracy of alligator cracks and oblique cracks. The findings of this study will accelerate the application of non-destructive automatic road damage detection.