Evaluation of information quality derived from AI-related information systems used for risk applications

Abstract

The use of artificial intelligence, such as text and image generators, is increasingly prevalent and accessible. These technologies have been both criticized and also promoted for their potential to automate tasks and increase efficiencies. From a risk perspective, we can interpret these technologies as a new addition to an already-abundant toolkit that can aid in risk applications. However, these new technologies need to be vetted for use in risk applications, regardless of whether the technology was designed for risk analysts, risk management, or risk communication purposes. As these AI-related technologies continue to grow in use, there is a need to gauge the quality of information that emerges from these technologies as that information is used for risk applications. This paper leverages current risk science quality indicators to develop 14 criteria that can be used to gauge the quality of information derived from AI-related information systems to be used for risk applications. Those 14 criteria are then demonstrated on a widely used AI-based information system. The insights from this paper can help understand both the benefits of these technologies and also additional implications for risk science methods, policies, and research.