It is difficult to detect defects such as shallow dents and rust on rolling surfaces by using the traditional vision-based surface inspection method (line light source and line array camera mode) which has a low sensitivity to these defects. This paper presents a method that introduces the fringe projection technique for traditional visual inspection devices to overcome the limitations of the traditional methods and uses deep-learning techniques for detecting defects such as cuts, abrasions, dents, and rust on the rolling surfaces of drum-shaped rollers. A new artificial-intelligence-based labeling method, namely, the Padua Incremental Mask Labeling Method, has been introduced for accelerating the calibration process used for defect detection, and based on a one-stage architecture, the You-Only-Look-Once-OurNet (YOLO-OurNet) deep-learning network has been designed for detecting the defects on the rolling surfaces of drum-shaped rollers. From the results of the experimental tests, the time required for detecting a defect has been found to be 0.024s, an accuracy rate of up to 99.2%, and the value of object detection evaluation index F1 of up to 0.988. Our method outperforms the related method on the domain of rolling surface defect detection.