Machine learning and sentiment analysis of unstructured free-text information about patient experience online

Abstract

BackgroundTraditional measures of patient experience have included surveys and, more recently, structured patient-reported outcome measures. There are also large amounts of unstructured, free-text information about the quality of health care available on the internet from blogs, social networks, and health-care rating websites that we are not scrutinising. In other industries, real-time natural language processing, such as sentiment analysis, of large datasets has provided a useful analytical approach to find patterns and understand data. If these techniques can be applied to health care, it opens up a novel approach to analyse large volumes of textual information about patient experience. The large number of free-text comments on the UK NHS Choices website allows an opportunity to examine these data through sentiment analysis. These comments are matched with the users' own quantitative ratings of the service, presenting an opportunity to measure the accuracy of natural language processing methods against the patient's own assessment. Simultaneously, the NHS has a developed programme of patient experience measurement via a national survey of hospital inpatients. Using these data sources, we have a natural opportunity to compare our sentiment analysis of comments to traditional patient surveys at an organisational level. MethodsWe tried to predict whether a patient would recommend a hospital, whether the hospital was clean, and whether they were treated with dignity from their free-text descriptions. We applied machine learning and natural language processing techniques to all (6400) online comments about hospitals on the NHS Choices website in 2010. We used open-source Weka data mining software. We used comments from NHS Choices data from 2008, 2009, and 2011 to train the software. Data from 2010 were used to test the predicting accuracy of the approach. We included our own a priori classification of the 1000 most common words and phrases in the analysis. Having calculated the accuracy of our prediction algorithm, we compared the results obtained with the national inpatient survey results for the same year (2010) at the hospital trust level with Spearman's test for rank correlation. FindingsWe were able to predict patients' rating of their care from their free-text comments with an accuracy of 81% for hospital cleanliness, 83% for treatment with dignity, and 89% for overall recommendation. We observed mild to moderate associations between our machine learning predictions and patient survey quantitative responses for the three categories examined: cleanliness (Spearman ρ=0·37, p<0·0001), dignity (ρ=0·50, p<0·0001), and overall recommendation (ρ=0·46, p<0·0001). InterpretationThe prediction accuracy that we have achieved using this machine learning process suggests that we are able to predict, from free text, a reasonably accurate assessment of patients' opinion about different performance aspects of a hospital. We also find that these machine learning predictions are associated to an extent with results of more conventional surveys. This work is ongoing and an iterative process, but suggests that it might be possible to monitor the so-called online cloud of patient experience in real-time and by doing so harness the value of patient opinion. FundingImperial College London is grateful for support from the National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care scheme, the National Institute for Health Research Biomedical Research Centre Funding scheme, and the Imperial Centre for Patient Safety and Service Quality.