MindReaD: Enhancing Pedestrian-Vehicle Interaction with Micro-Level Reasoning Data Annotation

Abstract

Listen

Translate article

Rich training datasets are essential for machine learning models to work well. Safety-critical applications, such as autonomous driving, that heavily rely on supervised learning algorithms, require accurate prediction of human behavioral intentions. Training such prediction algorithms requires not only micro-level action labels but also the estimation of human cognitive states and corresponding reasoning explanations at the same granularity level. However, traditional data labeling tools have limited support for these annotation tasks and deficient data structure. This article proposes a novel annotation tool, MindReaD, that facilitates labeling micro-level interactions with free-form reasoning text and grounding text with their visual representations. Focusing on pedestrian–vehicle interactions as a case study, we defined a data structure to label interaction events, collected data from 83 participants using multiple UI designs, and used the data to train ML models for predicting pedestrian crossing intent. The results show that the unique UI design features of Point-and-Explain and multiple prompts can significantly increase the quality of reasoning data annotations. The model trained with data using the proposed data annotation design outperforms the baseline data, with approximately 7% improvement in overall accuracy. User experience study also confirms positive user attitudes towards the developed data annotation system. The data annotation tool will be released to the public.