Combined Processing of Hyperspectral and Thermal Images of Plants in Soil for the Early Diagnosis of Drought

Abstract

The possibilities of explainable artificial intelligence (XAI) in the early diagnosis of drought in plants based on hyperspectral images (HSI) are investigated. To provide the explainability and high accuracy to the result, we used the markup of HSI by superimposed Thermal IR (TIR) images of the last day of the experiment. Traditional HSI-based NDVI (Normalized Difference Vegetation Index) images were also constructed. The markup of HSIs based on their clustering by the k-means method into 5 classes was also objectified: wet plants; plants in a state of drought; wet soil; dry soil; background. For HSI, on the day of the experiment started, the number of clusters was set to 2 less to reflect the absence of drought circumstances. For use in training and testing, all HSIs channels are marked up with the results of clustering. The HIS-TIR-combination made it possible to determine the temperature for each plant pixel in HSI, and as the result to determine the number of days without watering. A fully connected Double Layer Perceptron (DLP) neural network was used to solve classification and regression problems. The trained DLP-regressor showed the average accuracy of predicting the temperature of plants on the control days of the experiment RMSE = 0.52 degrees, providing an error in predicting the day of the beginning of the drought for near 2 days. The DLP-classifier was able to classify the drought of the plant in the early stages (the fifth day) with an accuracy of 97.3%. Software tools: pytorch, scikit-learn, pysptools.