The rapid development of deep learning has accelerated the progress of related technologies in the computer vision field and it has broad application prospects. Due to flower inter-class similarity and intra-class differences, flower image classification has essential research value. To achieve flower image classification, this paper proposes a deep learning method using the current powerful object detection algorithm YOLOv5 to achieve fine-grained image classification of flowers. Overlap and occluded objects often appear in the images of the flowers, so the DIoU_NMS algorithm is used to select the target box to enhance the detection of the blocked objects. The experimental dataset comes from the Kaggle platform, and experimental results show that the proposed model in this paper can effectively identify five types of flowers contained in the dataset, Precision reaching 0.942, Recall reaching 0.933, and mAP reaching 0.959. Compared with YOLOv3 and Faster-RCNN, this model has high recognition accuracy, real-time performance, and good robustness. The mAP of this model is 0.051 higher than the mAP of YOLOv3 and 0.102 higher than the mAP of Raster-RCNN.