Crop disease identification and interpretation method based on multimodal deep learning

Abstract

Identification methods of crop diseases based on image modality alone have achieved relative success under limited and restricted conditions. As a data-driven technology, its performance depends on a large amount of image labeling data. Many of the existing methods neglected the role and value of other modal data except images and only relies on low-level image features for disease recognition without utilizing high-level domain knowledge, leading to poor credibility and interpretability of identification results. This paper targets tomato and cucumber common invasive diseases recognition as the research object. First, for the problem of insufficient utilization of multimodal data in existing models, the semantic embedding methods for disease images and disease description texts were examined, and the correlation and complementarity between the two types of modal data was utilized to realize the joint representation learning of disease features. Second, in response to the requirements of reliable identification and interpretability of diseases, the knowledge representation and knowledge embedding mechanism in the field of disease identification was studied, and the high-level domain knowledge graph was used as the external guidance for image feature learning and disease identification. Lastly, a disease identification model based on “image-text” multimodal collaborative representation and knowledge assistance (ITK-Net) was constructed. The proposed model achieved an identification accuracy, precision, sensitivity and specificity of 99.63%, 99%, 99.07% and 99.78% respectively on a dataset composed of “image-text” pairs. Meanwhile, semantic interpretation was performed on the model inference process. The achievement of this paper can offer a new method for disease identification based on multimodal data and domain knowledge, which might help improve the intelligence level of crop disease identification.