Open source social media intelligence for enabling government applications

Abstract

Open-source social media intelligence (OSSMInt) is a field comprising of techniques and applications to analyze and mine open-source social media data for extracting actionable information and useful insights. The focus of the work presented in this paper is on novel applications and techniques of OSSMInt in the government sector. I propose and develop several novel use-cases and applications around OSSMInt for government and broadly divide them into three categories: identification, prediction, and response applications. In particular, I present solutions, tools and techniques for analyzing data from micro-blogging website to analyze citizen complaints and grievances in the public sector [response]. The research presented in this paper also describes my work on analyzing data from Twitter micro-blogging website to early forecast a civil unrest and protest [prediction]. Furthermore, I build various applications around identification and detection that are useful for the government and security analysts. I demonstrate the application of OSSMInt for identifying religious conflicts within society by mining public opinions on Tumblr website and fill the gaps of offline surveys. The study presented in this paper propose solutions for enabling law enforcement agencies to detect, prevent and combat online radicalization and extremism (content, users, and communities) by mining data from Tumblr, Twitter and YouTube websites [identification]. Furthermore, I also propose and build an application for detecting secret message exchanged in an adversarial communication and capture the obfuscated terms in messages. The central component of my proposed solution approach is the application of information retrieval and machine learning based techniques and algorithms. The study consists of experimenting with a diverse range of machine learning algorithms such as unsupervised, semi-supervised and supervised learning (k-NN, SVM, naive Bayes, random forest and decision tree) based algorithms. Based on the experimental results, I observe that there are several untapped opportunities as well as technical challenges in exploiting open-source social media data by the government for intelligence gathering. Furthermore, social media is a rich source of information for building security informatics based applications by mining the content, users, links and relationships on social media platforms.