Social Media Intelligence and Learning Environment: an Open Source Framework for Social Media Data Collection, Analysis and Curation

Abstract

Social Media Intelligence and Learning Environment (SMILE) is an open source framework bringing cutting-edge computational models on social media data to social science researchers and students with any level of programming and computation expertise. Many existing social media analysis tools require programming knowledge, a fee, or are closed source, making it challenging for social science researchers to apply existing and new methods to social media data. SMILE provides a user-friendly web interface, through which researchers can perform a wide spectrum of research tasks, ranging from social media data collection, natural language processing, text classification, social network analysis, and generating human readable outputs and visualizations. SMILE has adopted several technologies to support its needs. The data service of SMILE leverages the GraphQL language to provide an efficient and succinct API for client to communicate with a heterogeneous collection of social media APIs, including Twitter and Reddit. SMILE implements a microservices design and utilizes Amazon AWS services, such as Lambda and Batch for computation, S3 for data storage, and Elasticsearch for a Twitter streaming database, which makes it more portable, economic, and resilient. Analysis outputs can be shared with the larger community using Clowder, an open source data management system to support data curation of long tail data and metadata. SMILE is one of the main applications deployed as a standalone tool within the Social Media Macroscope (SMM), a science gateway based on the HUBzero platform. Over 200 users have used SMILE since its first release in 2018.