Perfect ratings with negative comments: Learning from contradictory patient survey responses

Abstract

This research explores why patients give perfect domain scores yet provide negative comments on surveys. In order to explore this phenomenon, vendor-supplied in-patient survey data from eleven different hospitals of a major U.S. health care system were utilized. The dataset included survey scores and comments from 56,900 patients, collected from January 2015 through October 2016. Of the total number of responses, 30,485 (54%) contained at least one comment. For our analysis, we use a two-step approach: a quantitative analysis on the domain scores augmented by a qualitative text analysis of patients’ comments. To focus the research, we start by building a hospital recommendation model using logistic regression that predicts a patient’s likelihood to recommend the hospital; we use this to further evaluate the top four most predictive domains. In these domains (personal issues, nurses, hospital room, and physicians), a significant percentage of patients who rated their experience with a perfect domain score left a comment categorized as not positive, thus giving rise to stark contrasts between survey scores and comments provided by patients. Within each domain, natural language analysis of patient comments shows that, despite providing perfect survey scores, patients have much to say to health care organizations about their experiences in the hospital. A summary of comments also shows that respondents provide negative comments on issues that are outside the survey domains. Results confirm that harvesting and analyzing comments from these patients is important, because much can be learned from their narratives. Implications for health care professionals and organizations are discussed.