Visual Analysis of Agricultural Workers using Explainable Artificial Intelligence (XAI) on Class Activation Map (CAM) with Characteristic Point Data Output from OpenCV-based Analysis

Abstract

In this study, we use explainable artificial intelligence (XAI) based on class activation map (CAM) techniques. Specifically, we use Grad-CAM, Grad-CAM++, and ScoreCAM to analyze outdoor physical agricultural (agri-) worker image datasets. In previous studies, we developed body-sensing systems to analyze human dynamics with the aim of enhancing agri-techniques, training methodologies, and worker development. These include distant, visual data-based sensing systems that capture image and movie datasets related to agri-worker motion and posture. For this study, we first obtained the aforementioned image datasets for researcher review. Then, we developed and executed Python programs with Open-Source Computer Vision (OpenCV) libraries and PyTorch to run XAI-oriented systems based on CAM techniques and obtained heat map-pictures of the visual explanations. Besides, we implement optical flow-based image analyses using our Visual C++ programs with OpenCV libraries, automatically set and chase the characteristic points related to the video datasets. Next, we analyze the dataset features and compare experienced and inexperienced subject groups. We investigate the output’s features, accuracies, and robustness to be able to make recommendations for real agri-workers, managers, product-developers, and researchers. Our findings indicate that the visualized output datasets are especially useful and may support further development of applied methods for these groups.