A Survey Protocol to Assess Meaningfulness and Usefulness of Automated Topic Finding in the NASA Aviation Safety Reporting System

Abstract

Context: The NASA Aviation Safety Reporting System (ASRS) is a voluntary confidential aviation safety reporting system. The ASRS receives reports from pilots, air traffic controllers, flight attendants and other involved in aviation operations. The reports are de-identified and coded by ASRS expert safety analysts and a short descriptive synopsis is written to describe the safety issue. The de-identified reports are then disseminated to the aviation community in a number of ways including entry into an online database, Safety Alert Bulletins and For Your Information Notices, and the CALLBACK newsletter. An opportunity of providing additional identification of safety concerns is with the use of topic modeling. Topic modeling can improve the dissemination of safety concerns by grouping and summarizing large collections of reports simultaneously. However, the generated summaries must be both meaningful and useful. Aim: We propose a methodology to evaluate whether automated topic finding using topic modeling provides meaningful and useful topics. Method: We extend the total error survey methodology to evaluate user topic comprehension of machine learning outputs. To accomplish this we performed a literature review to identify existing methods and define a construct for topic comprehension, utilizing existing ASRS synopsis writing practices to more precisely define meaningfulness and usefulness. Results: Nine responses were obtained providing interpretations of computer-generated topics for evaluation. Participants interpretation of computer-generated topics of report sets, match the title and description of ASRS report sets written separately by analysts. Conclusion: We conclude computer-generated topics, when grouping adjusted rand index is above 90%, are both meaningful and useful. However, the surveying of user understanding in machine learning outputs presents challenges due to the explosion of parameters to control for and the lack of systematic approach presented in the literature. More reproducible work and survey protocols are needed in the literature and our work is one step towards that direction.