Patient experience, an emerging marker of healthcare quality to policy makers and payers, is an area where Pain Medicine has historically underperformed. Real-time monitoring of patient feedback may be a critical tool to optimize the patient experience while eliminating the need for manual intervention. Here we leverage the Collaborative Health Outcomes Information Registry (CHOIR) platform and the Stanford Patient Experience Questionnaire (SPEQ) and develop a machine-mediated automatic classification of patient free-text feedback in terms of the sentiment expressed. SPEQ measures patient experience with 20 items in 7 touch points plus one free-text item. Over a three-week period at an academic multidisciplinary pain center, 628 patients were given SPEQ post-visit, of whom 123 (20%) completed the questionnaire, of whom 63 (51%) provided free-text feedback. The length of the feedback ranged from 2 to 513 words, with an average of 81.5 and SD of 105.8 words. We performed manual coding in terms of 1) aspects of patient experience referenced, 2) sentiment expressed for each aspect, and 3) overall sentiment. Next we performed natural language processing (NLP) using recursive neural network (Socher et al 2013), which rates each sentence as having Positive, Neutral, or Negative sentiment. In order to derive an overall rating of passage-level sentiment, we performed a receiver-operating curve analysis. The optimal cut-off for classifying the feedback as Negative is a threshold of 68.5% or more of the sentences being Negative. There is moderate agreement between machine-mediated rating with single-rater coding, with Kappa of 0.57 and AUC of 0.79. In summary, we have developed an algorithm that enables automatic classification of patient feedback. Large-volume and real-time processing of patient feedback enables organization-wide deployment of patient experience monitoring with low cost and resource burden. Future work will focus on optimizing this algorithm using training data from additional patient feedback. Patient experience, an emerging marker of healthcare quality to policy makers and payers, is an area where Pain Medicine has historically underperformed. Real-time monitoring of patient feedback may be a critical tool to optimize the patient experience while eliminating the need for manual intervention. Here we leverage the Collaborative Health Outcomes Information Registry (CHOIR) platform and the Stanford Patient Experience Questionnaire (SPEQ) and develop a machine-mediated automatic classification of patient free-text feedback in terms of the sentiment expressed. SPEQ measures patient experience with 20 items in 7 touch points plus one free-text item. Over a three-week period at an academic multidisciplinary pain center, 628 patients were given SPEQ post-visit, of whom 123 (20%) completed the questionnaire, of whom 63 (51%) provided free-text feedback. The length of the feedback ranged from 2 to 513 words, with an average of 81.5 and SD of 105.8 words. We performed manual coding in terms of 1) aspects of patient experience referenced, 2) sentiment expressed for each aspect, and 3) overall sentiment. Next we performed natural language processing (NLP) using recursive neural network (Socher et al 2013), which rates each sentence as having Positive, Neutral, or Negative sentiment. In order to derive an overall rating of passage-level sentiment, we performed a receiver-operating curve analysis. The optimal cut-off for classifying the feedback as Negative is a threshold of 68.5% or more of the sentences being Negative. There is moderate agreement between machine-mediated rating with single-rater coding, with Kappa of 0.57 and AUC of 0.79. In summary, we have developed an algorithm that enables automatic classification of patient feedback. Large-volume and real-time processing of patient feedback enables organization-wide deployment of patient experience monitoring with low cost and resource burden. Future work will focus on optimizing this algorithm using training data from additional patient feedback.
Read full abstract