With the continuing changes in the structure of plant and cultivation patterns, new diseases are constantly appearing on the leaves of plant, exacerbating the threat to food security and agricultural production in many areas of the world. Thus, a rapid and accurate recognition of various diseases in plant will not only significantly reduce unnecessary planting costs, but also alleviate the economic losses and environmental pollution caused by incorrect disease diagnosis. Recent advances in deep learning have improved the performance in recognizing plant leaf diseases. In this paper, we present a general framework for recognizing plant diseases. Firstly, we propose a deep feature descriptor based on transfer learning to obtain a high-level latent feature representation. Then, we integrate the deep features with traditional handcrafted features by feature fusion to capture the local texture information in plant leaf images. In addition, centre loss is incorporated to further enhance the discriminative ability of the fused feature. The centre loss simultaneously minimizes intra-class distance and maximizes inter-class distance to learn both compact and separate features. Extensive experiments have been conducted on three publicly available datasets (two Apple Leaf datasets and one Coffee Leaf dataset) to validate the effectiveness of proposed method. The propose method achieves 99.79%, 92.59% and 97.12% classification accuracies on the three datasets, respectively. The experiment results demonstrate that the proposed method effectively captures the discriminative feature representation for distinguishing plant leaf diseases.