Sentiment Analysis of Social Survey Data for Local City Councils

Abstract

Big data analytics can be used by smart cities to improve their citizens’ liveability, health, and wellbeing. Social surveys and also social media can be employed to engage with their communities, and these can require sophisticated analysis techniques. This research was focused on carrying out a sentiment analysis from social surveys. Data analysis techniques using RStudio and Python were applied to several open-source datasets, which included the 2018 Social Indicators Survey dataset published by the City of Melbourne (CoM) and the Casey Next short survey 2016 dataset published by the City of Casey (CoC). The qualitative nature of the CoC dataset responses could produce rich insights using sentiment analysis, unlike the quantitative CoM dataset. RStudio analysis created word cloud visualizations and bar charts for sentiment values. These were then used to inform social media analysis via the Twitter application programming interface. The R codes were all integrated within a Shiny application to create a set of user-friendly interactive web apps that generate sentiment analysis both from the historic survey data and more immediately from the Twitter feeds. The web apps were embedded within a website that provides a customisable solution to estimate sentiment for key issues. Global sentiment was also compared between the social media approach and the 2016 survey dataset analysis and showed some correlation, although there are caveats on the use of social media for sentiment analysis. Further refinement of the methodology is required to improve the social media app and to calibrate it against analysis of recent survey data.