Data Collection and Analysis of Inter-Area Communication During a Disaster Exercise at a Large Hospital

Abstract

Hospitals, especially regional disaster base hospitals, play a critical role in saving lives during a disaster. Therefore, it is important for the hospitals to conduct disaster response exercises and thoroughly evaluate the results to understand the current level of response capability and to identify potential problems in disaster response and hospital business continuity. However, because data collection and analysis of disaster exercise requires a lot of manpower and time, such evaluation has not been well conducted so far. Aiming to develop evaluation indices of exercise performance, we collected data on patient and document flow, inter-departmental communication, and exercise participant behavior using video cameras, voice recorders, and NFC tags. Focusing on inter-departmental communication, this paper describes the data collection using voice recorders attached to PHS and reports on an attempt to use a large-scale language model to automatically classify the verbal data into several performative verbs.Methods: We collected data during disaster response exercises conducted at a major hospital in Kanagawa Prefecture, Japan, in 2022 and 2023. This hospital is designated as a regional disaster base hospital, which is expected to play a central role in regional disaster medicine. These two exercises were both designed to accommodate mass casualties caused by a major earthquake. We used a voice recorder that can record a conversation via PHS/smartphone; we can record the voice from both sides with a single voice recorder. We attached this voice recorder to several exercise players in different departments to collect communication data on information sharing and command and control. The conversation data was transcribed and used for the further analysis. We analyzed the conversation content from the viewpoint of performative verbs to calculate the anticipation ratio of inter-departmental communication. Usually, this kind of analysis is done manually, which requires a lot of man-hours. On the other hand, in this analysis, we applied the GPT-4.0 language model to automatically classify the conversations into nine performative verbs: greet, inform, acknowledge, request, query, accept, declare, confirm, and suggest. Results and Discussions: We compare the results obtained by GPT with those of human analysts to evaluate the reliability of the classification. We confirmed that the kappa value is 0.73, which indicates that there is substantial agreement between the GPT and manual classification. Then, we calculated the anticipation ratio, which is the ratio of push to pull information, and is often used as a rough indicator of efficient information sharing. By comparing the ratio between 2022 and 2023, we found that the ratio was higher for the command post in 2023, indicating that the command post in 2023 proactively provided information to other departments in advance before they were asked. Conclusion: Through this study, we confirmed that inter-departmental conversations in the exercise can be clearly recorded with the voice recorder attached to the PHS. We also confirmed that a large language model can be used for the classification by performative verbs, thus saving man-hours in calculating the anticipation ratio.