Hybridizing qualitative coding with natural language processing and deep learning to assess public comments: A case study of the clean power plan

Abstract

Public comments are a rich source of data on attitudes toward public policy, but the scale poses major challenges for qualitative analyses. Supervised deep learning and natural language processing potentially enable classification of these data but require high quality labeled inputs. This study investigates whether hybridizing rigorous qualitative coding methods with machine learning approaches can classify large amounts of policy-oriented text data (public comments) while keeping manual effort tractable. Using a convolutional neural network, we evaluate spatiotemporal variation in themes and expressed public attitudes toward a specific US climate policy, the Clean Power Plan (CPP), which was proposed in 2014 and repealed in 2019 without ever taking effect. Public comments were solicited for both the proposal and proposed repeal across eight cities, representing a large and highly targeted dataset on dynamic public attitudes toward the CPP. Rigorous manual coding and data augmentation techniques enabled good model performance (F1 scores of 0.71 and 0.81, respectively, for sentiment and topic classification), even with a very small training set. We find that most speakers supported the CPP despite its eventual repeal and uncover notable rhetorical trends like competing narratives of justice for those affected by climate change versus fossil fuel host communities.