Stripe rust caused by Puccinia striiformis f. sp. tritici and leaf rust caused by Puccinia triticina, are two devastating diseases on wheat, which seriously affect the production safety of wheat. Timely detection and identification of the two diseases are essential for taking effective disease management measures to reduce wheat yield losses. To realize the accurate identification of wheat stripe rust and wheat leaf rust during the different growth stages, in this study, the image-based identification of wheat stripe rust and wheat leaf rust during different growth stages was investigated based on deep learning using image processing technology. Based on the YOLOv5s model, we built identification models of wheat stripe rust and wheat leaf rust during the seedling stage, stem elongation stage, booting stage, inflorescence emergence stage, anthesis stage, milk development stage, and all the growth stages. The models were tested on the different testing sets in the different individual growth stages and in all the growth stages. The results showed that the models performed differently in disease image identification. The model based on the disease images acquired during an individual growth stage was not suitable for the identification of the disease images acquired during the other individual growth stages, except for the model based on the disease images acquired during the milk development stage, which had acceptable identification performance on the testing sets in the anthesis stage and the milk development stage. In addition, the results demonstrated that wheat growth stages had a great influence on the image identification of the two diseases. The model built based on the disease images acquired in all the growth stages produced acceptable identification results. Mean F1 Score values between 64.06% and 79.98% and mean average precision (mAP) values between 66.55% and 82.80% were achieved on each testing set composed of the disease images acquired during an individual growth stage and on the testing set composed of the disease images acquired during all the growth stages. This study provides a basis for the image-based identification of wheat stripe rust and wheat leaf rust during the different growth stages, and it provides a reference for the accurate identification of other plant diseases.