Deciphering Public Voices in the Digital Era

Abstract

Problem, research strategy, and findings Planners are increasingly using online public engagement approaches to broaden their reach in communities. This results in substantial volumes of digital, text-based public feedback data, making it difficult to analyze efficiently and derive meaningful insights. We explored the use of the novel large language model (LLM), ChatGPT, in analyzing a public feedback data set collected via online submissions in Hamilton City (New Zealand) in response to a proposed local plan change. Specifically, we initially employed zero-shot prompts with ChatGPT for tasks like summarizing, topic identification, and sentiment analysis and compared the results with those obtained by human planners and two standard natural language processing (NLP) techniques: latent Dirichlet allocation (LDA) topic modeling and lexicon-based sentiment analysis. The findings show that zero-shot prompting effectively identified political stances (accuracy: 81.7%), reasons (87.3%), decisions sought (85.8%), and associated sentiments (94.1%). Although subject to several limitations, ChatGPT demonstrates promise in automating the analysis of public feedback, offering substantial time and cost savings. In addition, few-shot prompting enhanced performance in more complex tasks, such as topic identification involving planning jargon. We also provide insights for urban planners to better harness the power of ChatGPT to analyze citizen feedback. Takeaway for practice ChatGPT presents a transformative opportunity for planners, particularly those dealing with growing volumes of public feedback data. However, it cannot be entirely relied upon. Planners must be mindful of ChatGPT’s limitations, including its sensitivity to prompt phrasing, inherent biases from training data, tendency to overgeneralize, and occasional omission of nuanced details. To enhance accuracy, planners should prescreen data for consistency, provide clear and iteratively tested prompts, use few-shot prompts for complex analysis, and explore various combinations of prompting strategies to develop an effective local approach. It is also crucial to ensure human review of the results.