Achieving safe navigation of unmanned surface vehicle (USV) in low light conditions remains a focus of research. The key to achieving the intelligent navigation of USV is the precise segmentation of navigable areas and fast obstacle detection. There are several challenges in processing infrared thermal imaging shoreline images using only semantic segmentation networks, including difficulties in extracting shorelines due to blurred details at the edges of infrared images, segmenting small targets, and complex obstacles due to small temperature differences in complex backgrounds. To address these issues, this paper introduces a novel UYF-Net network. The proposed method combines the improved U-Net with the YOLOv5m using a decision-level fusion strategy to achieve a better speed-accuracy tradeoff. Experimental results demonstrate that our method outperforms all comparative approaches discussed in this paper, further confirming the enhanced precision in segmentation, increased detection speed, and more stable model training process of the proposed network.