Several deep learning algorithms are currently focused on object detection in adverse weather scenarios for autonomous driving systems. However, these algorithms face challenges in real-time scenarios, leading to a reduction in detection accuracy. To tackle these issues, this paper introduces a lightweight object detection model named Self Driving Cars You Only Look Once Adverse Weather Detection Network (SD-YOLO-AWDNet), derived from enhancements to the YOLOv5 algorithm. The model incorporates four progressive improvement levels within the YOLOv5 framework. This includes integrating C3Ghost and GhostConv modules in the backbone to enhance detection speed by reducing computational overhead during feature extraction. To address potential accuracy issues arising from these modules, Depthwise-Seprable Dilated Convolutions (DSDC) are introduced, striking a balance between accuracy and parameter reduction. The model further incorporates a Coordinate Attention (CA) module in the GhostBottleneck to enhance feature extraction and eliminate unnecessary features, improving precision in object detection. Additionally, a novel “Focal Distribution Loss” replaces CIoU Loss, accelerating bounding box regression and loss reduction. Test dataset experiments demonstrate that SD-YOLO-AWDNet outperforms YOLOv5 with a 54% decrease in FLOPs, a 52.53% decrease in model parameters, a 2.24% increase in mAP, and a threefold improvement in detection speed.