To address the issues of low automation, reliance on manual screening by professionals, and long detection cycles in current urban drainage pipeline defect detection, this study proposes an improved object detection algorithm called EFE-SSD (enhanced feature extraction SSD), based on the SSD (Single Shot MultiBox Detector) network. Firstly, the RFB_s module is added to the SSD backbone network to enhance its feature extraction capabilities. Additionally, multiple scale features are fused to improve the detection performance of small target defects to some extent. Then, the improved ECA attention mechanism is used to adjust the channel weights of the output layer, suppressing irrelevant features. Finally, the Focal Loss is employed to replace the cross-entropy loss in the SSD network, effectively addressing the issue of imbalanced positive and negative samples during training. This increases the weight of difficult-to-classify samples during network training, further improving the detection accuracy of the network. Experimental results show that EFE-SSD achieves a detection mAP of 92.2% for four types of pipeline defects: Settled deposits, Displaced joints, Deformations, and Roots. Compared to the SSD algorithm, the model’s mAP was increased by 2.26 percentage points—ensuring the accuracy of pipeline defect detection.