Twitter Streaming API Research Articles

Real-time Sentiment Analysis from Data Streaming

Public sentiment is a potent indicator of how people perceive and receive a topic. It has the power to make or break companies and people. Twitter is one of the best platforms in today’s generation to gauge public sentiment. [10] Utilizing the power and influence Twitter has we decided to create a service that would enable us to know how a trending topic is being viewed by the masses in real-time. The user gives the topic as input to the front-end graphical user interface that topic is then taken and fed to the Twitter streaming API. Tweets containing the hashtag of the topic mentioned by the user are returned and the sentiment of those tweets is predicted and sent to the front end where analysis prediction of the sentiment of the tweets is done dynamically as the tweets come in. By using our service for a few minutes the user will get to know what the overall outlook of a topic is and use that information as a guiding beacon for any future decisions regarding that topic.

Designing a platform for Tweet Collection, Analytics and Storage (TweetCASP)

In this era of big data, the streaming, storage, and analysis of large amounts of data present a variety of challenges. Several challenges must be addressed by designers of data-intensive systems in order to retrieve useful information. Collecting, storing, and analyzing data requires a collection and analytics platform comprised of an appropriate choice of data processing and analytics technologies in order to acquire meaningful insight. In this paper, we report on TweetCASP (Tweet Collection, Analytics and Storage Platfrom), which gathers tweets based on user-entered keywords using Twitter's Streaming API, providing an environment for real-time analytics on streaming data and permanently storing data in an Apache Cassandra NoSQL datastore to fulfill future batch-oriented data processing requirements. Moreover, The TweetCASP presents an example of a data-intensive system used by software developers, designers, and researchers for data collecting and analytics.

Invasion@Ukraine: Providing and Describing a Twitter Streaming Dataset That Captures the Outbreak of War between Russia and Ukraine in 2022

Social media can be a mirror of human interaction, society, and historic disruptions. Their reach enables the global dissemination of information in the shortest possible time and, thus, the individual participation of people worldwide in global events in almost real-time. However, these platforms can be equally efficiently used in information warfare to manipulate human perception and opinion formation. Within this paper, we describe a dataset of raw tweets collected via the Twitter Streaming API in the context of the onset of the war, which Russia started in Ukraine on February 24, 2022. A distinctive feature of the dataset is that it covers the period from one week before to one week after Russia invasion of Ukraine. This paper details the acquisition process and provides first insights into the content of the data stream. In addition, the data has been annotated with availability tags, resulting from rehydration attempts at two points in time: directly after data acquisition and shortly before manuscript submission. This may provide information on Twitter moderation policies. Further, we provide a detailed list of other published dataset covering the same topic. On the content level, we can show that our dataset comprises several distinct topics related to the conflict and conspiracy narratives -- topics that deserve more profound investigation. Therefore, the presented dataset is also made available to the community in an extended version with pseudonymized tweet content upon request.

Public Perceptions of Flavored Waterpipe Smoking on Twitter.

Waterpipe tobacco smoking has become increasingly popular in recent years, especially among youth. We aimed to understand longitudinal trends in the prevalence and user perception of waterpipes and their flavors on Twitter. We extracted waterpipe-related tweets from March 2021 to May 2022 using the Twitter Streaming API and classified them into promotional tweets and non-promotional tweets. We examined the longitudinal trends regarding the waterpipe flavors mentioned on Twitter and conducted sentiment analysis on each waterpipe flavor-related non-promotional tweet. Among over 1.3 million waterpipe-related tweets, 1,158,884 tweets were classified as non-promotional and 235,132 were classified as promotional. The most frequently mentioned waterpipe flavor groups were fruit (34%), sweets (17%), and beverages (15%) among all flavor-containing non-promotional tweets (17,746 tweets). The least mentioned flavor groups were tobacco (unflavored, 4%) and spices (2%). Sentiment analysis showed that among non-promotional waterpipe-related tweets, 39% were neutral, 36% were positive, and 23% were negative. The most preferred waterpipe flavors were fruit, mixed, and alcohol flavors. The least preferred flavor groups were tobacco and spice flavors. Our study provided valuable information on the prevalence of waterpipe flavors that can be used to support the future regulation of flavored waterpipe tobacco products given the nature of the current regulations on other flavored tobacco products.

Real-time Twitter Sentiment Analysis for Moroccan Universities using Machine Learning and Big Data Technologies

In recent years, sentiment analysis (SA) has raised the interest of researchers in several domains, including higher education. It can be applied to measure the quality of the services supplied by the higher education institution and construct a university ranking mechanism from social media like Twitter. Hence, this study presents a novel system for Twitter sentiment prediction on Moroccan public universities in real-time. It consists of two phases: offline sentiment analysis phase and real-time prediction phase. In the offline phase, the collected French tweets about twelve Moroccan universities were classified according to their sentiment into ‘positive’, ‘negative’, or ‘neutral’ using six machine learning algorithms (random forest, multinomial Naive Bayes classifier, logistic regression, decision tree, linear support vector classifier, and extreme gradient boosting) with the term frequency-inverse document frequency (TF-IDF) and count vectorizer feature extraction techniques. The results reveal that random forest classifier coupled with TF-IDF has obtained the best test accuracy of 90%. This model was then applied on real-time tweets. The real-time prediction pipeline comprises Twitter streaming API for data collection, Apache Kafka for data ingestion, Apache Spark for real-time sentiment analysis, Elasticsearch for real-time data exploration, and Kibana for data visualization. The obtained results can be used by the Ministry of higher education, scientific research and innovation of Morocco for decision-making process.

Potential Impact of FDA Flavor Enforcement Policy on Vaping Behavior on Twitter.

In January 2020, the FDA announced an electronic cigarette (e-cigarette) flavor enforcement policy to restrict the sale of all unauthorized cartridge-based flavored e-cigarettes except tobacco and menthol flavors, which was implemented on 6 February 2020. This study aimed to understand the potential influence of this policy on one vaping behavior change—quitting vaping—using Twitter data. Twitter posts (tweets) related to e-cigarettes were collected between June 2019 and October 2020 through a Twitter streaming API. Based on the geolocation and keywords related to quitting vaping, tweets mentioning quitting vaping from the US were filtered. The demographics (age and gender) of Twitter users who mentioned quitting vaping were further inferred using a deep learning algorithm (deepFace). The proportion of tweets and Twitter users mentioning quitting vaping were compared between before and after the announcement and implementation of the flavor policy. Compared to before the FDA flavor policy, the proportion of tweets (from 0.11% to 0.20% and 0.24%) and Twitter users (from 0.15% to 0.70% and 0.86%) mentioning quitting vaping were significantly higher after the announcement and implementation of the policy (p-value < 0.001). In addition, there was an increasing trend in the proportion of female and young adults (18–35 years old) mentioning quitting vaping on Twitter after the announcement and implementation of the policy compared to that before the policy. Our results showed that the FDA flavor enforcement policy did have a positive impact on quitting vaping on Twitter. Our study provides an initial evaluation of the potential influence of the FDA flavor enforcement policy on user vaping behavior.

An Association of Twitter Derived Measures of Discrimination and Race-Motivated Hate Crimes

In 2020, there was a sharp rise in Anti-Asian American hate crimes. While the causes are still being explored, one of the main reasons this occurred was due to the rise in Anti-Asian American rhetoric connected to the COVID-19 pandemic. This exemplifies how prejudices often can turn into violence, more specifically hate crimes. And hateful speech, oftentimes spread through the internet, can help spread and deepen that prejudice to a wider audience. In previous research, prejuidice and discrimination has largely been studied through self-reported measures. While those studies are necessary, they are susceptible to many pitfalls and should be supplemented with statistics-based measures. This study employs one of these emerging measures. This study used the Twitter Streaming API to collect tweets containing a racial slur, an extension of hateful speech, geo-located from users in the United States over a period of two months. With these, this study compared statewide Twitter racial slur usage and statewide average sentiment polarity of the tweets with race-motivated hate crimes from the state. Though no relationship was found between statewide sentiment expressed and race-motivated hate crimes, this study did find a significant positive relationship between statewide Twitter slur usage and hate crimes. These findings support previous studies that have connected Internet-based measures of discrimination with aspects of general well-being. and even specifically hate crimes. This study shows the potential of using data derived from social media to create cost effective and expansive indicators of an area’s social climate around a topic, even race.

Supply chain risk identification: a real-time data-mining approach

PurposeThe global pandemic COVID-19 unveils transforming the supply chain (SC) to be more resilient against unprecedented events. Identifying and assessing these risk factors is the most significant phase in supply chain risk management (SCRM). The earlier risk quantification methods make timely decision-making more complex due to their inability to provide early warning. The paper aims to propose a model for analyzing the social media data to understand the potential SC risk factors in real-time.Design/methodology/approachIn this paper, the potential of text-mining, one of the most popular Artificial Intelligence (AI)-based data analytics approaches for extracting information from social media is exploited. The model retrieves the information using Twitter streaming API from online SC forums.FindingsThe potential risk factors that disrupt SC performance are obtained from the recent data by text-mining analyses. The outcomes carry valuable insights about some contemporary SC issues due to the pandemic during the year 2021. The most frequent risk factors using rule mining techniques are also analyzed.Originality/valueThis study presents the significant role of Twitter in real-time risk identification from online SC platforms like “Supply Chain Dive”, “Supply Chain Brain” and “Supply Chain Digest”. The results indicate the significant role of data analytics in achieving accurate decision-making. Future research will extend to represent a digital twin for identifying potential risks through social media analytics, assessing risk propagation and obtaining mitigation strategies.

Performance of Sentiment Classification on Tweets of Clothing Brands

Social media such as Facebook, Instagram, LinkedIn, and Twitter ease the sharing of ideas, thoughts, videos, and photos and information through the building of virtual networks and communities. This has allowed companies and products to reach a wider audience in terms of marketing and advertising, and to gauge feedback from the public. This research investigates clothing brand mentions on Twitter to perform sentiment analysis on users’ thoughts on three clothing brands, namely Asos, Uniqlo and Topshop. The data is collected by applying python libraries, Tweepy to access data from the Twitter streaming API. Following that, data pre-processing such as tokenization, filtering, stemming, and case normalization are performed to remove outliers. Then, the TextBlob algorithm is applied to label the tweet data into three classes; Positive, Negative and Neutral based on the polarity of the tweets. Word embeddings are also created using Word2Vec with TF-IDF. The word embeddings are fed into classification models namely Support Vector Machine (SVM), Naïve Bayes (NB), Random Forest (RF), Logistic Regression (LR) and Multilayer Perceptron (MLP) by comparing their accuracy performances. The models went through training and testing process on a curated tweet dataset comprising 24000 records with three clothing brands (Asos, Uniqlo, Topshop). The classification process was carried out by SVM, NB, RF, LR and MLP with a ratio of 50-50 and 70-30 train-test splits. Hyperparameter tuning was implemented by GridSearchCV to find the best parameters of classification models in order to optimize the best results. The evaluation of performance was measured with accuracy, precision, recall and F1-Score. In the 50-50 train-test splits, LR achieved the highest accuracy by scoring 82%, 87% and 87% on Asos, Uniqlo and Topshop respectively. In the 70-30 train-test splits, LR also achieved highest accuracy by scoring 85%, 90% and 90% for the three clothing brands respectively.

A chronological and geographical analysis of personal reports of COVID-19 on Twitter from the UK.

ObjectiveGiven the uncertainty about the trends and extent of the rapidly evolving COVID-19 outbreak, and the lack of extensive testing in the United Kingdom, our understanding of COVID-19 transmission is limited. We proposed to use Twitter to identify personal reports of COVID-19 to assess whether this data can help inform as a source of data to help us understand and model the transmission and trajectory of COVID-19.MethodsWe used natural language processing and machine learning framework. We collected tweets (excluding retweets) from the Twitter Streaming API that indicate that the user or a member of the user's household had been exposed to COVID-19. The tweets were required to be geo-tagged or have profile location metadata in the UK.ResultsWe identified a high level of agreement between personal reports from Twitter and lab-confirmed cases by geographical region in the UK. Temporal analysis indicated that personal reports from Twitter appear up to 2 weeks before UK government lab-confirmed cases are recorded.ConclusionsAnalysis of tweets may indicate trends in COVID-19 in the UK and provide signals of geographical locations where resources may need to be targeted or where regional policies may need to be put in place to further limit the spread of COVID-19. It may also help inform policy makers of the restrictions in lockdown that are most effective or ineffective.

Deep spatio-temporal emotion analysis of geo-tagged tweets for predicting location based communal emotion during COVID-19 Lock-down

Due to the COVID-19 pandemic, countries across the globe has enforced lockdown restrictions that influence the people’s socio-economic lifecycle. The objective of this paper is to predict the communal emotion of people from different locations during the COVID-19 lockdown. The proposed work aims in developing a deep spatio-temporal analysis framework of geo-tagged tweets to predict the emotions of different topics based on location. An optimized Latent Dirichlet Allocation (LDA) approach is presented for finding the optimal hyper-parameters using grid search. A multi-class emotion classification model is then built via a Recurrent Neural Network (RNN) to predict emotions for each topic based on locations. The proposed work is experimented with the twitter streaming API dataset. The experimental results prove that the presented LDA model-using grid search along with the RNN model for emotion classification outperforms the other state of art methods with an improved accuracy of 94.6%.

Is the Sample Good Enough? Comparing Data from Twitter's Streaming API with Twitter's Firehose

Twitter is a social media giant famous for the exchange of short, 140-character messages called "tweets". In the scientific community, the microblogging site is known for openness in sharing its data. It provides a glance into its millions of users and billions of tweets through a "Streaming API" which provides a sample of all tweets matching some parameters preset by the API user. The API service has been used by many researchers, companies, and governmental institutions that want to extract knowledge in accordance with a diverse array of questions pertaining to social media. The essential drawback of the Twitter API is the lack of documentation concerning what and how much data users get. This leads researchers to question whether the sampled data is a valid representation of the overall activity on Twitter. In this work we embark on answering this question by comparing data collected using Twitter's sampled API service with data collected using the full, albeit costly, Firehose stream that includes every single published tweet. We compare both datasets using common statistical metrics as well as metrics that allow us to compare topics, networks, and locations of tweets. The results of our work will help researchers and practitioners understand the implications of using the Streaming API.

Breast Cancer Identification from Patients’ Tweet Streaming Using Machine Learning Solution on Spark

Twitter integrates with streaming data technologies and machine learning to add new value to healthcare. This paper presented a real-time system to predict breast cancer based on streaming patient’s health data from Twitter. The proposed system consists of two major components: developing an offline building model and an online prediction pipeline. For the first component, we made a correlation between the features to determine the correlation between features and reduce the number of features from the Breast Cancer Wisconsin Diagnostic dataset. Two feature selection algorithms are recursive feature elimination and univariate feature selection algorithms which are applied to features after correlation to select the essential features. Four decision trees, logistic regression, support vector machine, and random forest classifier have been used on features after correlation and feature selection. Also, hyperparameter tuning and cross-validation have been applied with machine learning to optimize models and enhance accuracy. Apache Spark, Apache Kafka, and Twitter Streaming API are used to develop the second component. The best model with the highest accuracy obtained from the first component predicts breast cancer in real time from tweets’ streaming. The results showed that the best model is the random forest classifier which achieved the best accuracy.

In search of bluer skies: Would people move to places of better air qualities?

This study examines the statistical relationship between people's movements and difference in air quality between the origin and the destination, using computational social science approach. We put forward an important environmentally driven human movement question: Would people move to places of better air qualities? Utilizing Twitter Streaming API, 39,928,132 randomized geo-coded tweets posted by 8,551,557 individual users during the period of 01/01/2016 – 31/12/2017 were extracted to examine the movements of Twitter users. Coupling Twitter users’ international movements with air quality data, our results have shown that Twitter users tend to move to places of better air qualities in general. We find that in most countries, the net move-out rate tends to be positively correlated with the monthly PM2.5 concentration in the origin country, and the number of international movements to the destination country is negatively correlated with the decrease in PM2.5 concentration in the origin country. For domestic movements, our results have shown that Twitter users in Indian cities, including Mumbai, one of the largest and highly polluted cities, tend to move to other Indian cities of better air qualities. Our results confirm that people across different parts of the world tend to move to places of better air qualities.

Predicting Coronavirus Pandemic in Real-Time Using Machine Learning and Big Data Streaming System

Twitter is a virtual social network where people share their posts and opinions about the current situation, such as the coronavirus pandemic. It is considered the most significant streaming data source for machine learning research in terms of analysis, prediction, knowledge extraction, and opinions. Sentiment analysis is a text analysis method that has gained further significance due to social networks’ emergence. Therefore, this paper introduces a real-time system for sentiment prediction on Twitter streaming data for tweets about the coronavirus pandemic. The proposed system aims to find the optimal machine learning model that obtains the best performance for coronavirus sentiment analysis prediction and then uses it in real-time. The proposed system has been developed into two components: developing an offline sentiment analysis and modeling an online prediction pipeline. The system has two components: the offline and the online components. For the offline component of the system, the historical tweets’ dataset was collected in duration 23/01/2020 and 01/06/2020 and filtered by #COVID-19 and #Coronavirus hashtags. Two feature extraction methods of textual data analysis were used, n-gram and TF-ID, to extract the dataset’s essential features, collected using coronavirus hashtags. Then, five regular machine learning algorithms were performed and compared: decision tree, logistic regression, k-nearest neighbors, random forest, and support vector machine to select the best model for the online prediction component. The online prediction pipeline was developed using Twitter Streaming API, Apache Kafka, and Apache Spark. The experimental results indicate that the RF model using the unigram feature extraction method has achieved the best performance, and it is used for sentiment prediction on Twitter streaming data for coronavirus.

Twitter monitoring evidence of Covid-19 infodemic in Italy

BackgroundRisk communication is essential for risk management, especially during alarming events, in order to create a balanced risk perception. The tweets follow up can be useful to timely evidence “media storms” or “infodemics” thus suggesting corrective interventions. The spread of Covid-19 has been the occasion to observe the evolution of tweets about this topic, with the aim of evidence peaks of fearMethodsOn 30th January 2020 we started a large collection of Italian and English tweets containing the following keywords: “coronavirus”, “2019-nCov”, “CoronavirusOutbreak”, “CoronavirusWho”, “WuhanCoronavirus”, “coronaviruschina”, “coronavirusitalia”, “virus cina”, “virus cinese”, “virus mascherine”, “virus paura”, “virus polmonite” and “virus vaccine”. “Covid-19” and “SARS-CoV-2” were added later. For the data collection Twitter Streaming API was used.ResultsThe total number of tweets collected till 26th February was approximately 20M, 2M in Italian and 18M in English. The trend of the volume of tweets gathered shows a first small peak between Jan 30th and Feb 1st, but a second, higher, peak is visible after the 21st February, following the report of the Covid-19 clusters in Italy. Italian tweets account for 6% of the first peak and for 22% of the second. The Italian word “paura” firstly appeared on Feb 5th, with a peak after Feb 21th.Conclusions”infodemics” or “media storms”, are situations of uncontrolled diffusion of news and opinion, frequently leading to irrational and counterproductive behaviors. The Italian evolution of tweets indicate since the first day of cluster report, a clear escalation of fear, that suggests timely corrective actions to facilitate the disease control.Key messages With the twitter monitoring we demonstrate that since 21th February an infodemic is spreading in Italy.The twitter follow up can be used to timely evidence a public alarm, motivate corrective interventions and. evaluate the efficacy of them.

The dynamics of negative stereotypes as revealed by tweeting behavior in the aftermath of the Charlie Hebdo terrorist attack

We describe the evolution of a stereotype as it emerged in tweets about the Charlie Hebdo terrorist attack in Paris in early 2015. Our focus is on terms associated with the Muslim community and the Islamic world. The data (400k tweets) were collected via Twitter streaming API and consisted of tweets that contained at least one of 16 hashtags associated with the Charlie Hebdo attack (e.g., #JeSuisCharlie, #IAmCharlie, #ParisAttacks), collected between January 14th and February 9th. From these data, we generated pairwise co-occurrence frequencies between key words such as “Islam”, “Muslim(s)”, “Arab(s)”, and “The Prophet” and possible associates such as: “terrorism”, “terror”, “terrorist(s)”, “kill(ed)”, “free”, “freedom” and “love”. We use changes in frequency of co-occurring words to define ways in which acute negative and positive stereotypes towards Muslims and Islam arise and evolve in three phases during the period of interest. We identify a positively-valenced backlash in a subset of tweets associated with the “origins of Islam”. Results depict the emergence and transformation of implicit online stereotypes related to Islam from naturally occurring social media data and how pro-as well as anti-Islam online small-world networks evolve in response to a terrorist attack.

Twitter mining using semi-supervised classification for relevance filtering in syndromic surveillance.

We investigate the use of Twitter data to deliver signals for syndromic surveillance in order to assess its ability to augment existing syndromic surveillance efforts and give a better understanding of symptomatic people who do not seek healthcare advice directly. We focus on a specific syndrome—asthma/difficulty breathing. We outline data collection using the Twitter streaming API as well as analysis and pre-processing of the collected data. Even with keyword-based data collection, many of the tweets collected are not be relevant because they represent chatter, or talk of awareness instead of an individual suffering a particular condition. In light of this, we set out to identify relevant tweets to collect a strong and reliable signal. For this, we investigate text classification techniques, and in particular we focus on semi-supervised classification techniques since they enable us to use more of the Twitter data collected while only doing very minimal labelling. In this paper, we propose a semi-supervised approach to symptomatic tweet classification and relevance filtering. We also propose alternative techniques to popular deep learning approaches. Additionally, we highlight the use of emojis and other special features capturing the tweet’s tone to improve the classification performance. Our results show that negative emojis and those that denote laughter provide the best classification performance in conjunction with a simple word-level n-gram approach. We obtain good performance in classifying symptomatic tweets with both supervised and semi-supervised algorithms and found that the proposed semi-supervised algorithms preserve more of the relevant tweets and may be advantageous in the context of a weak signal. Finally, we found some correlation (r = 0.414, p = 0.0004) between the Twitter signal generated with the semi-supervised system and data from consultations for related health conditions.

Generating a Highlight Moments Summary Video of Apolitical Event using Ontological Analysis on Social Media Speech Sentiment

Numerous viewers choose to watch political or presidential debates highlights via TV or internet, rather than seeing the whole debate nowadays, which requires a lot of time. However, the task of making a debate summary, which can be considered neutral and does not give out a negative nor a positive image of the speaker, has never been an easy one, due to personal or political beliefs bias of the video maker. This study came up with a solution that generates highlights of a political event, based on twitter social network flow. Twitter streaming API is used to detect an event's tweets stream using specific hashtags, and detect on a timescale the extreme changes of volume of tweets, which will determine the highlight moments of our video summary at first, then a process is set up based on a group of ontologies that analyze each tweet of these moments to calculate the percentage of each sentiment’s positivity, then classify those moments by category (positive, negative or neutral).

Identification of Trending Topics Using Periodically Collected Twitter Data

Social media is an interactive personal tool to articulate an individual's cognizance. This project involves one such micro blogging platform, Twitter. Trends can simply be defined as the frequently mentioned topics throughout the stream of user activities. Mining twitter data for identifying trending topics provides an overview of the topics and issues that are currently popular within the online community. Therefore, the most effective and suitable methodology should be implemented to identify the short term high intensity discussion topic. The trigrams or higher order n-grams are used to determine the trending topic. Twitter Streaming API is used to collect data from the Twitter accounts using API keys and the formatted tweets are stored in a non SQL database. Subsequent steps include data cleansing followed by stemming. The processed data is subjected to trend prediction algorithms like DB Scan, Frequent Pattern Mining, Trees(fuzzy/inductive/decision), Soft frequent pattern mining and empirical statistics such as Frequency metric, TF-IDF, Normalized term frequency and Entropy based on the key parameters to identify the most trending event within a period of time. Thus, the trending topics can be detected with a reasonably close approximation to the expected outcome. This can be used in detecting and predicting events for an early warning system (or) prediction tools and also artificially intelligent services like web search system or recognition systems.