To address the problems of current mainstream semantic segmentation network such as rough edge segmentation of navel oranges defects, poor accuracy of small target defect segmentation and insufficient deep-level semantic extraction of defects, feature information will be lost, a multi-class segmentation model based on improved DeepLabv3+ is proposed to detect the surface defects of navel oranges. The Coordinate Attention Mechanism is embedded into the DeepLabv3+ network for better semantic segmentation performance, while the dilated convolution of Atrous Spatial Pyramid Pooling structure is replaced with deformable empty convolution to improve the fitting ability of the network to target shape changes and irregular defects. In addition, a BiFPN-based feature fusion branch is introduced at the DeepLabv3+ encoder side to realize multi-scale feature fusion and enrich feature space and semantic information. The experimental results show that the average intersection ratio and average pixel intersection ratio accuracies of the improved DeepLabv3+ model on the navel orange surface defect dataset are 77.32% and 86.38%, which are 3.81% and 5.29% higher than the original DeepLabv3+ network, respectively, improving the extraction capability of navel orange defect features and having better segmentation performance.