Building inspection and maintenance are becoming increasingly essential means by which to consider the deterioration problems of old, reinforced concrete (RC) buildings. While such inspection work can be conducted with the aid of computer vision-based technology, this technology remains challenged, since real-world structural defects and environmental conditions are varied and complex. In recent years, object detection algorithms have improved to achieve greater speed and accuracy with the help of deep learning. In this paper, an advanced object detector, YOLOv5s, was successfully applied to the recognition of common structural defects including cracks, delamination, exposed reinforcement, rust stains, spalling, tile cracks, tile delamination, and tile loss. Compared with the other advanced object detectors of YOLO (i.e., YOLOv5m, YOLOv5l, YOLOv5x, YOLOv4, and YOLOv3) based on the built data set, the YOLOv5s algorithm shows an obvious advantage for defect detection, achieving 64.5% and 67.0% mean average precision (mAP) for training and testing, respectively. It also takes less than 0.1 seconds to detect a defect on an image. The lightweight and high detection performance of the YOLOv5s algorithm shows great promise for potential deployment on an onboard inspection device, such as an unmanned aerial vehicle (UAV) or a robot, to achieve real-time structural inspection.