This paper presents an integrated Unmanned Aerial Vehicle (UAV) imagery system and artificial intelligence-based smart farming for saffron cultivation. This work is about monitoring the growth of saffron flowers through UAV imagery every week or month during the pre-harvesting time while the percentage of the covered area by leaf and saffron flowers is calculated. This is crucial for predicting saffron growing areas over cultivated regions. From the vertical viewing of UAV imagery from above, it is problematic from the bird’s eye perspective to identify the particular saffron-growing areas of things, so the multi-class classification of flowers (subjects) recognition system is proposed. So, the implementation of this work is divided into four phases. The first two phases (Phase 1 and Phase 2) are about the algorithms for Saffron growing region detections and predictions using the modified You Only Look Once (YOLO) higher version model followed by statistical analysis based region corner detections. The outcomes of the first two phases are used to form a database of saffron flowers with different scales, orientations, and deformations. This is further used to build a two-class classification model in the third phase (Phase 3) that discriminates saffron vs. non-saffron flowers. In the fourth phase, the two-class classification model is extended to a multi-class flower recognition model for recognizing 900 flower images database available in the largest social media website dedicated exclusively to gardening in the Phase 4. Extensive experiments are conducted, and the performances are compared with some existing state-of-the-art methods that show the superiority of the proposed system.