Abstract
BackgroundOnline reviews may act as a rich source of data to assess the quality of dental practices. Assessing the content and sentiment of reviews on a large scale is time consuming and expensive. Automation of the process of assigning sentiment to big data samples of reviews may allow for reviews to be used as Patient Reported Experience Measures for primary care dentistry.AimTo assess the reliability of three different online sentiment analysis tools (Amazon Comprehend DetectSentiment API (ACDAPI), Google and Monkeylearn) at assessing the sentiment of reviews of dental practices working on National Health Service contracts in the United Kingdom.MethodsA Python 3 script was used to mine 15800 reviews from 4803 unique dental practices on the NHS.uk websites between April 2018 –March 2019. A random sample of 270 reviews were rated by the three sentiment analysis tools. These reviews were rated by 3 blinded independent human reviewers and a pooled sentiment score was assigned. Kappa statistics and polychoric evalutaiton were used to assess the level of agreement. Disagreements between the automated and human reviewers were qualitatively assessed.ResultsThere was good agreement between the sentiment assigned to reviews by the human reviews and ACDAPI (k = 0.660). The Google (k = 0.706) and Monkeylearn (k = 0.728) showed slightly better agreement at the expense of usability on a massive dataset. There were 33 disagreements in rating between ACDAPI and human reviewers, of which n = 16 were due to syntax errors, n = 10 were due to misappropriation of the strength of conflicting emotions and n = 7 were due to a lack of overtly emotive language in the text.ConclusionsThere is good agreement between the sentiment of an online review assigned by a group of humans and by cloud-based sentiment analysis. This may allow the use of automated sentiment analysis for quality assessment of dental service provision in the NHS.
Highlights
Online reviews are increasingly being used in order to allow patients to express their views about the quality of their health care [1]
There was good agreement between the sentiment assigned to reviews by the human reviews and Amazon Comprehend DetectSentiment Application Protocol Interface (ACDAPI) (k = 0.660)
Assessing the reliability of automatic sentiment analysis tools on rating the sentiment in dental reviews were due to syntax errors, n = 10 were due to misappropriation of the strength of conflicting emotions and n = 7 were due to a lack of overtly emotive language in the text
Summary
Online reviews are increasingly being used in order to allow patients to express their views about the quality of their health care [1]. A study by Holliday et al [3] demonstrates that patients value the use of online ratings and feel that experience ratings should be publicly available; physicians were less open to this data being used to assess the quality of care they provide. NHS.uk (formerly NHS Choices) is a website that collates data for National Health Service (NHS) healthcare services in the UK This website allows patients to rate their healthcare provider using a 5-star system and free text comments. There are large number of reviews that are left by patients on their dental practices means that this may be a useful source of information for the assessment of patients’ experiences in and the quality of primary dental care. Automation of the process of assigning sentiment to big data samples of reviews may allow for reviews to be used as Patient Reported Experience Measures for primary care dentistry
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
