Global food security can be influenced by the diseases in crop plants as several diseases straightforwardly influence the quality of the grains, vegetables, fruits, etc., which also results in affecting of agricultural productivity. Like other plants, the mango tree is also affected by several diseases, and also the identification of multi-disease classification with a single leaf is more complex, and also it is impossible to detect diseases with bare eyes. Based on the other plants, the mango tree is also affected by various diseases, which is more difficult to detect the disorders with bare eyes. It is error-prone, inconsistent, and unreliable. Here, the mango trees are affected during the production, and also affect the plant health regarding multi-diseases. When the plants are affected by the diseases, it may cause fewer amounts of productivity, as a result, impacting the economy. However, it is more critical to detect plant diseases with the large varieties of trees and plants. Various research tasks on deep learning approaches focus on identifying the diseases in plants including leaves and fruits. Thus, the main objective of this paper is to implement an effective and appropriate technique for diagnosing mango tree diseases and their symptoms through fruit and leaf images, and thus, there is a need for an appropriate system for cost-effective and early solutions to this problem. Hence, the main intention of this work is to implement an efficient and suitable technique for diagnosing mango tree diseases and also identify the symptoms through fruit and leaf images. Intending to overcome the existing challenges, there is a need for an appropriate system for achieving cost-effectiveness and also creating an early solution to resolve this problem. This paper intends to present novel deep learning models for mango tree multi-disease classification. Initially, the data collection is done for gathering the diseased parts of the mango tree in terms of leaf and fruit images. Then, the contrast enhancement of the images is performed by the “Contrast-Limited Adaptive Histogram Equalization (CLAHE)”. For the deep feature extraction of leaf images, and fruit images, Convolutional Neural Network (CNN) is employed, and the features from both inputs are concatenated for further processing. Further, the weighted feature selection is adopted for selecting the most significant features by the Adaptive Squirrel-Grey Wolf Search Optimization (AS-GWSO). Enhanced “Long Short Term Memory (LSTM)” is applied in the classification part with parameter optimization using the same AS-GWSO for enhancing classification accuracy. At last, the results of the designed system on various mango tree diseases verify that the designed approach has yielded the highest accuracy by evaluating conventional approaches. Therefore, it would also alleviate and treat the affected mango leaf diseases accurately.