Federated learning can effectively ensure data security and improve the problem of data islanding. However, the performance of federated learning-based schemes could be better due to the imbalance of image data. Therefore, this paper proposes a federated learning approach based on priori knowledge and a bilateral segmentation network for image edge extraction. First, federated learning can distribute training images for some special complex images due to the small sample and unshared data. Then, the image with similar edge information to the original image is learned to obtain prior knowledge, and the local uniform sparsity method is used to strengthen the detail features and weaken the background features. Based on the bilateral segmentation network, we introduce a dilated pyramid pooling layer and multi-scale feature fusion module to fuse the shallow detailed features in the context path with the deep abstract features obtained through the dilated pyramid pooling. The final result is obtained by fusing the result with prior knowledge and the result with the context path. Finally, we conduct experiments on some public datasets, and the results show that the proposed method greatly improves extraction accuracy compared with the traditional and the most advanced methods.