An agricultural product plays a major role in the economical growth of developing countries. Agricultural products like pepper and corn are the essential crops with respect to human health food security. But, these two crops are prone to different diseases such as gray leaf spot, common rust and fruit rot which affects the productivity of crops. Further, the severity identification is also a challenging one. To address these limitations, this work presents different approaches for identifying the crop lesions and predicting the severity and thereby increasing the productivity of crops. The development of the proposed model includes steps such as dataset collection, noise removal, segmentation, feature extraction, classification and severity prediction. Initially, the crop images are pre-processed by the median filter and the pre-processed images are processed are segmented, extracted and classified by the optimized U-Net model. Moreover, hybrid optimizer which is the integration of GJA (Golden jackal algorithm) and RDA (Red deer algorithm) are utilized for precise segmentation and classification. Finally, the severity prediction is computed for the diseased leaves by the measuring the size of image pixels. The experimentation is carried out on the benchmark PlantVillage dataset; the accuracy and precision values achieved are 99.2% and 99.1%. Thus, the experimental outcomes show the effectiveness and stability of the model.