Potential safety hazards (PSHs) around tracks need to be accurately and timely detected to ensure high- speed railroad operating safety. Unmanned aerial vehicle (UAV)-based railroad inspection has potential to complement visual inspections by providing better overhead views and mitigating safety concerns. This paper proposes a hybrid learning architecture, named YOLORS (you only look once railroad scene), to parse driveways and tracks in UAV images, and evaluate PSHs along tracks. First, a stripe-pool-based parsing branch is designed and assembled in YOLORS to precisely accomplish the pixel-level parsing tasks. Second, a new data augmentation method called object-guided mosaic is developed and incorporated into the architecture for improving small object detection performance. Finally, extensive experiments indicate YOLORS demonstrates exceptional performance in both pixel-level parsing and object detection tasks, which outperforms the current advanced models. This proposed approach can quickly convert UAV imagery containing complicated railroad scenes into useful information with a high detection rate.