Evaluating the Influence of Incorrect Reassurances on Trust in Imperfect Automated Decision Aid

Abstract

Prior research on trust in automation has primarily focused on instances where decision aids form predictions based on the same raw input information as humans. This study extends the paradigms into the human-processed input framework, where machine predictions are based on human-processed data, instead of the raw input. The human-processed input framework introduces a unique error pattern, “incorrect reassurance,” where faulty automation prediction erroneously validates users’ initial errors. This study examined the impact of “incorrect reassurance” on human trust in automation in the human-processed input framework. Thirty-five participants completed a Mental Rotation Task (MRT), using an imperfect automated decision aid. Results show that incorrect reassurance patterns led to larger trust decrement, lower final performance, and quicker final reaction time compared to other performance patterns. The findings emphasize the importance of understanding trust in automation in the context of the human-processed input framework.