A drone detection system can achieve real-time inspection of concrete bridge damage. However, it is difficult to deploy existing detection algorithms on a portable system due to their high computational cost. Therefore, a portable damage detection algorithm called CD-YOLOv8 is designed. CD-YOLOv8 is operated on Jetson Xavier NX to detect drone-captured bridge images in real-time. The multi-scale feature adaptive fusion module is used to reduce model parameters and improve detection speed. The spatial context pyramid and deformable convolutional structure are introduced to improve the detection accuracy of tiny concrete bridge damage. Compared with the YOLOv8n, the mAP50 of CD-YOLOv8 is increased by 3.6 %, while the model parameters and FLOPs are decreased by 0.05 M and 0.7 G, respectively. CD-YOLOv8 is also tested on the Jetson Xavier NX with a detection speed of 44 FPS, which is 15.79 % faster than the YOLOv8n. Therefore, CD-YOLOv8 has high accuracy and excellent real-time performance.