Building a Natural Language Processing Artificial Intelligence to Predict Suicide-Related Events Based on Patient Portal Message Data

Abstract

ObjectiveTo develop a natural language processing artificial intelligence model trained on text from patient portal messages to predict 30-day suicide-related events (SRE). Patients and MethodsPatient portal messages sent by patients between January 1, 2013, and October 31, 2017 were screened for an associated SRE within 30 days. For both patient portal messages associated with a 30-day SRE and a randomized control set, we automatically extracted several features: (1) frequencies of keywords; (2) message metadata; and (3) message sentiment. ResultsA total of 840 patient portal messages were included in our final analysis, including 420 messages with and without an associated 30-day SRE. Patient messages with an associated 30-day SRE had a mean sentiment score that was less than those without an SRE (P<.001). Messages with an associated 30-day SRE had greater word counts (P=.002) and more use of ellipses (P=.02), but less use of exclamation marks (P=.04) and question marks (P=.007) compared with messages without a 30-day SRE. The neural network machine learning model had the highest area under the receiver operating curve at 0.710, with a sensitivity of 56.0% and a specificity of 69.0%. ConclusionA natural language processing artificial intelligence model trained on a subset of patient portal message data was able to predict 30-day SRE at a level comparable to commonly used suicide assessment tools. Predictors that conveyed the overall tone of a patient message, such as the sentiment score, were more highly weighted by machine learning models in predicting 30-day SRE than the frequencies of individual words.