Short Text Streams Research Articles

A topic-enhanced dirichlet model for short text stream clustering

A topic-enhanced dirichlet model for short text stream clustering

A context-enhanced Dirichlet model for online clustering in short text streams

Online clustering of short text streams has become significant due to the popularity of news and social media platforms. The objective of online clustering is to maintain active topics (clusters) by automatically detecting new topics and forgetting outdated ones. Most existing approaches exploit static and high dimensional semantic term representation of the text to enhance the clustering quality. While these approaches use inference procedures that depend on a fixed batch size to reduce the number of clusters related to a given topic and bring it closer to the actual number of topics. This paper proposes a non-parametric Dirichlet model with episodic inference (EINDM) to cluster the evolving short text stream by introducing a window-based low-dimensional semantic term representation which captures the contextual relationships between words. In addition, an episodic inference procedure is introduced to reduce the cluster sparsity in the model. Furthermore, a novel “word specificity” measure is proposed based on neighborhood terms for evolving contexts for individual terms. Extensive empirical evaluation demonstrates that EINDM yields the best performance, in terms of NMI, homogeneity, and cluster purity, compared to recent state-of-the-art clustering models.

Recent methods on short text stream clustering: A survey study

AbstractThe volume and the velocity of data in social media are increasing and the social media has become a very useful environment to detect and track the real‐world events. However, to fulfill this, it is crucial to group‐related texts according to their topics and clustering takes an essential role at this point since we have no prior knowledge about the topics and their evolution in social media. In this survey, we review the current approaches and techniques proposed for short text stream clustering in recent years. The reviewed techniques are grouped according to their methodology and discussed in detail. Also, the datasets utilized to evaluate the performance of the proposed methods and the results are summarized together with the clustering quality measures used for these evaluations. Furthermore, current challenges about short‐text stream clustering are discussed.This article is categorized under: Data: Types and Structure > Streaming Data

A Drift-Sensitive Distributed LSTM Method for Short Text Stream Classification

Real-world applications especially in the fields of social media have produced massive short text streams. Unlike traditional normal texts, these data present the characteristics of short length, high-volume, high-velocity and variable data distribution etc, which lead to the issues of data sparsity and concept drift. It is hence very challenging for existing short text classification algorithms. Therefore, we propose a flexible Long Short-Term Memory (LSTM) ensemble network based short text stream classification approach, which is implemented in a distributed mode while maintaining the high-accuracy advantage of deep learning models. More specifically, external resource based short text embedding using a pretrained embedding model and CNN is first proposed for the solution to the data sparsity of short texts. Second, to adapt to the high-volume and high-velocity short text streams, a flexible LSTM network is developed and implemented in a distributed mode for classifying short text data streams. Third, a concept drift factor is introduced for adapting to the concept drifts caused by the changing of data distributions. Finally, experiments conducted on three real short text data sets demonstrate that as compared with several state-of-the-art short text (stream) classification approaches, the proposed approach can classify short text streams effectively and efficiently while adapting to concept drifts.

An Online Semantic-Enhanced Graphical Model for Evolving Short Text Stream Clustering.

Due to the popularity of social media and online fora, such as Twitter, Reddit, Facebook, and Wechat, short text stream clustering has gained significant attention in recent years. However, most existing short text stream clustering approaches usually work on static data and tend to cause a "term ambiguity" problem due to the sparse word representation. Beyond, they often exploit short text streams in a batch way and are difficult to find evolving topics in term-changing subspaces. In this article, we propose an online semantic-enhanced graphical model for evolving short text stream clustering (OSGM), by exploiting the word-occurrence semantic information and dynamically maintaining evolving active topics in term-changing subspaces in an online way. Compared to the existing approaches, our online model is not only free of determining the optimal batch size but also lends itself to handling large-scale data streams efficiently. It is also able to handle the "term ambiguity" problem without incorporating features from external resources. More importantly, to the best of our knowledge, it is the first work to extract evolving topics in term-changing subspaces automatically in an online way. Extensive experiments demonstrate that the proposed model yields better performance compared to many state-of-the-art algorithms on both synthetic and real-world datasets.

Balancing stability and plasticity when learning topic models from short and noisy text streams

Learning hidden topics from short text streams has become crucial in modern applications such as social networks, instant messages, question and answer forums, etc. Building an effective learning method poses two main challenges: Short and noisy data, as well as the stability-plasticity dilemma. In this paper, we investigate carefully how existing methods face these challenges. From our theoretical and empirical analyses, they often deal well with a challenge but ineffectively handle the other. In particular, they often suffer from catastrophic forgetting, because they impose a constraint on the learned knowledge from only the previous time step in the streaming duration. In this paper, we propose a novel method, namely BSP, which has a regularization term based on second-order Taylor expansion to accumulate information from all former minibatches. Moreover, external knowledge and Dropout technique will be combined at each time step to handle better short and noisy texts as well as enhance the model’s plasticity. We empirically evaluate BSP, compared to other state-of-the-art streaming methods in terms of dealing with stability-plasticity dilemma and handling short and noisy texts. The extensive experiments show superior effectiveness of BSP.

GOWSeqStream: an integrated sequential embedding and graph-of-words for short text stream clustering

GOWSeqStream: an integrated sequential embedding and graph-of-words for short text stream clustering

A graph convolutional topic model for short and noisy text streams

Learning hidden topics from data streams has become absolutely necessary but posed challenging problems such as concept drift as well as short and noisy data. Using prior knowledge to enrich a topic model is one of potential solutions to cope with these challenges. Prior knowledge that is derived from human knowledge (e.g. Wordnet) or a pre-trained model (e.g. Word2vec) is very valuable and useful to help topic models work better. However, in a streaming environment where data arrives continually and infinitely, existing studies are limited to exploiting these resources effectively. Especially, a knowledge graph, that contains meaningful word relations, is ignored. In this paper, to aim at exploiting a knowledge graph effectively, we propose a novel graph convolutional topic model (GCTM) which integrates graph convolutional networks (GCN) into a topic model and a learning method which learns the networks and the topic model simultaneously for data streams. In each minibatch, our method not only can exploit an external knowledge graph but also can balance the external and old knowledge to perform well on new data. We conduct extensive experiments to evaluate our method with both a human knowledge graph (Wordnet) and a graph built from pre-trained word embeddings (Word2vec). The experimental results show that our method achieves significantly better performances than state-of-the-art baselines in terms of probabilistic predictive measure and topic coherence. In particular, our method can work well when dealing with short texts as well as concept drift.

Dynamic clustering for short text stream based on Dirichlet process.

Due to the explosive growth of short text on various social media platforms, short text stream clustering has become an increasingly prominent issue. Unlike traditional text streams, short text stream data present the following characteristics: short length, weak signal, high volume, high velocity, topic drift, etc. Existing methods cannot simultaneously address two major problems very well: inferring the number of topics and topic drift. Therefore, we propose a dynamic clustering algorithm for short text streams based on the Dirichlet process (DCSS), which can automatically learn the number of topics in documents and solve the topic drift problem of short text streams. To solve the sparsity problem of short texts, DCSS considers the correlation of the topic distribution at neighbouring time points and uses the inferred topic distribution of past documents as a prior of the topic distribution at the current moment while simultaneously allowing newly streamed documents to change the posterior distribution of topics. We conduct experiments on two widely used datasets, and the results show that DCSS outperforms existing methods and has better stability.

EWNStream+: Effective and Real-time Clustering of Short Text Streams Using Evolutionary Word Relation Network

The real-time clustering of short text streams has various applications, such as event tracking, text summarization and sentimental analysis. However, accurately and efficiently clustering short text streams is challenging due to the sparsity problem (i.e., the limited information comprised in a single short text document leads to high-dimensional and sparse vectors when we represent short texts using traditional vector space models), topic drift and the fast generated text streams. In this paper, we provide an effective and real-time Evolutionary Word relation Network for short text streams clustering (EWNStream[Formula: see text]) method. The EWNStream[Formula: see text] method constructs a bi-weighted word relation network using the aggregated term frequencies and term co-occurrence statistics at corpus level to overcome the sparsity problem and topic drift of short texts. Better still, as the query window in the stream shifts to the newly arriving data, EWNStream[Formula: see text] is capable of incrementally updating the word relation network by incorporating new word statistics and decaying the old ones to naturally capture the underlying topic drift in the data streams and reduce the size of the network. The experimental results on a real-world dataset show that EWNStream[Formula: see text] can achieve better clustering accuracy and time efficiency than several counterpart methods.

Lifelong Learning Augmented Short Text Stream Clustering Method

Depending on the scanning mode, existing short text stream clustering methods can be divided into the following two kinds of methods: one-pass-based and batch-based. The one-pass-based method handles each text only one time, but cannot deal with the sparseness problem very well. The batch-based method obtains better results by allowing multiple iterations of each batch, but the efficiency is relatively low. To overcome these problems, this paper presents Lifelong learning Augmented Short Text stream clustering method (LAST), which incorporates the episodic memory module and sparse experience replay module of lifelong learning into the clustering process. Specifically, LAST processes each text one time, but at a certain interval it randomly samples some previously seen texts of the episodic memory to update cluster features by performing sparse experience replay. Empirical studies on two public datasets demonstrate that the performance of the LAST-based method is on a par with the batch-based method, and runs close to the speed of the one-pass-based method.

Boosting prior knowledge in streaming variational Bayes

Exploiting prior/human knowledge is an effective way to enhance Bayesian models, especially in cases of sparse or noisy data, for which building an entirely new model is not always possible. There is a lack of studies on the effect of external prior knowledge in streaming environments, where the data come sequentially and infinitely. In this work, we show the problem of vanishing prior knowledge in streaming variational Bayes. This is a serious drawback in various applications. We then develop a simple framework to boost the external prior when learning a Bayesian model from data streams. By boosting, the prior knowledge can be maintained and efficiently exploited through each minibatch of streaming data. We evaluate the performance of our framework in four scenarios: streaming in synthetic data, streaming sentiment analysis, streaming learning for latent Dirichlet allocation, and streaming text classification, in comparison with the methods that do not keep priors. From extensive experiments, we find that when provided good external knowledge, our framework can improve the performance of a Bayesian model, often by a significant margin for noisy and short text streams.

A Dirichlet process biterm-based mixture model for short text stream clustering

Short text stream clustering has become an important problem for mining textual data in diverse social media platforms (e.g., Twitter). However, most of the existing clustering methods (e.g., LDA and PLSA) are developed based on the assumption of a static corpus of long texts, while little attention has been given to short text streams. Different from the long texts, the clustering of short texts is more challenging since their word co-occurrence pattern easily suffers from a sparsity problem. In this paper, we propose a Dirichlet process biterm-based mixture model (DP-BMM), which can deal with the topic drift problem and the sparsity problem in short text stream clustering. The major advantages of DP-BMM include (1) DP-BMM explicitly exploits the word-pairs constructed from each document to enhance the word co-occurrence pattern in short texts; (2) DP-BMM can deal with the topic drift problem of short text streams naturally. Moreover, we further propose an improved algorithm of DP-BMM with forgetting property called DP-BMM-FP, which can efficiently delete biterms of outdated documents by deleting clusters of outdated batches. To perform inference, we adopt an online Gibbs sampling method for parameter estimation. Our extensive experimental results on real-world datasets show that DP-BMM and DP-BMM-FP can achieve a better performance than the state-of-the-art methods in terms of NMI metrics.

Generation of topic evolution graphs from short text streams

Topic evolution mining on short texts is an important research topic in natural language processing. Existing methods have been focused either on the topic evolution of normal documents or on the evolution of topics along a timeline. In this paper, we aim to generate topic evolutionary graphs from short texts, which not only capture the main topic timeline, but also reveal the correlations between related subtopics. Firstly, we propose an Encoder-only Transformer Language Model (ETLM) to quantify the relationship between words. Then we propose a novel topic model, referred as weighted Conditional random field regularized Correlated Topic Model (CCTM), which leverages semantic correlations to discover meaningful topics and topic correlations. Finally, topic evolutionary graphs are generated by an Online version of CCTM (OCCTM) to capture the evolutionary patterns of main topics and related subtopics. Experimental results on real-world datasets demonstrate our method outperforms baselines on quality of topics and presents motivated patterns for topic evolution mining.

A nonparametric model for online topic discovery with word embeddings

With the explosive growth of short documents generated from streaming textual sources (e.g., Twitter), latent topic discovery has become a critical task for short text stream clustering. However, most online clustering models determine the probability of producing a new topic by manually setting some hyper-parameter/threshold, which becomes barrier to achieve better topic discovery results. Moreover, topics generated by using existing models often involve a wide coverage of the vocabulary which is not suitable for online social media analysis. Therefore, we propose a nonparametric model (NPMM) which exploits auxiliary word embeddings to infer the topic number and employs a “spike and slab” function to alleviate the sparsity problem of topic-word distributions in online short text analyses. NPMM can automatically decide whether a given document belongs to existing topics, measured by the squared Mahalanobis distance. Hence, the proposed model is free from tuning the hyper-parameter to obtain the probability of generating new topics. Additionally, we propose a nonparametric sampling strategy to discover representative terms for each topic. To perform inference, we introduce a one-pass Gibbs sampling algorithm based on Cholesky decomposition of covariance matrices, which can further be sped up using a Metropolis-Hastings step. Our experiments demonstrate that NPMM significantly outperforms the state-of-the-art algorithms.

Collaboratively Tracking Interests for User Clustering in Streams of Short Texts

In this paper, we aim at tackling the problem of user clustering in the context of their published short text streams. Clustering users by short text streams is more challenging than in the case of long documents associated with them as it is difficult to track users' dynamic interests in streaming sparse data. To obtain better user clustering performance, we propose two user collaborative interest tracking models that aim at tracking changes of each user's dynamic topic distributions in collaboration with their followees' dynamic topic distributions, based both on the content of current short texts and the previously estimated distributions. Our models can be either short-term or long-term dependency topic models. Short-term dependency model collaboratively tracks users' interests based on users' topic distributions at the previous time period only, whereas long-term dependency model collaboratively tracks users' interests based on users' topic distributions at multiple time periods in the past. We also propose two collapsed Gibbs sampling algorithms for collaboratively inferring users' dynamic interests for their clustering in our short-term and long-term dependency topic models, respectively. We evaluate our proposed models via a benchmark dataset consisting of Twitter users and their tweets. Experimental results validate the effectiveness of our proposed models that integrate both users' and their collaborative interests for user clustering by short text streams.

Burst Hotspots Dynamic Detection and Tracking on Large-Scale Text Stream

The online social network has become an important communication tool for people and forms a virtual society interacting with the real world. The numerous events rapidly spread through social networks and may become hotspots in a short period of time. Especially, the negative events vibrate national security and social stability, potentially causing a series of social problems. Therefore, detection and tracking burst hotspots on social networks are of great significance. However, this problem is non-trivial because of the challenges of massive noise, sparsity, high-dimensionality, and dynamic changing. To address these challenges, this paper proposes a distributed method of burst hotspots dynamic detection and tracking based on Map/Reduce. To judge the relevance of the current text and previous texts, the keyword similarity matrix is calculated by using Word2Vec and context information. The keyword weights are modified according to the association model to further reduce noise. The proposed method is not only robust to the sparsity of short text but also overcomes the curse of dimensionality. Finally, it can dynamically detect and track burst hotspots from large-scale short text stream on the Hadoop platform. The experiments have shown that the proposed method outperforms state-of-the-art algorithms.

Dynamic topic modeling via self-aggregation for short text streams

Social networks such as Twitter, Facebook, and Sina microblogs have emerged as major sources for discovering and sharing the latest topics. Because social network topics change quickly, developing an effective method to model such topics is urgently needed. However, topic modeling is challenging due to the sparsity problem and the dynamic change of topics in microblog streams. In this study, we propose dynamic topic modeling via a self-aggregation method (SADTM) that can capture the time-varying aspect of topic distributions and resolve the sparsity problem. The SADTM aggregates the observable and unordered short texts as the aggregated document without leveraging an external context to overcome the sparsity problem of short text. Furthermore, we exploit word pairs instead of words for each microblog to generate more word co-occurrence patterns. The SADTM models temporal dynamics by using the topic distribution at previous time steps in microblog streams to infer the current topic from sequential data. Extensive experiments on a real-world Sina microblog dataset demonstrate that our SADTM algorithm outperforms several state-of-the-art methods in topic coherence and cluster quality. Additionally, when applied in a search scene, our SADTM significantly outperforms all baseline methods in terms of the quality of the search results.

Online Biterm Topic Model based short text stream classification using short text expansion and concept drifting detection

Short text stream classification suffers from enormous challenges, due to the sparsity, high dimension and rapid variability of the short text stream. In this paper, we present a short text stream classification approach refined from online Biterm Topic Model (BTM) using short text expansion and concept drifting detection. Specifically, in our method, we firstly extend short text streams from an external resource to make up for the sparsity of data, and use online BTM to select representative topics instead of the word vector to represent the feature of short texts. Secondly, we propose a concept drift detection method based on the topic model to detect the hidden concept drifts in short text streams. Thirdly, we build an ensemble model using several data chunks and update with the newest data chunk and results of the concept drift detection. Finally, extensive experimental results demonstrate that compared to well-known baselines, our approach achieves a better performance in the classification and concept drifting detection.

Learning From Short Text Streams With Topic Drifts

Short text streams such as search snippets and micro blogs have been popular on the Web with the emergence of social media. Unlike traditional normal text streams, these data present the characteristics of short length, weak signal, high volume, high velocity, topic drift, etc. Short text stream classification is hence a very challenging and significant task. However, this challenge has received little attention from the research community. Therefore, a new feature extension approach is proposed for short text stream classification with the help of a large-scale semantic network obtained from a Web corpus. It is built on an incremental ensemble classification model for efficiency. First, more semantic contexts based on the senses of terms in short texts are introduced to make up of the data sparsity using the open semantic network, in which all terms are disambiguated by their semantics to reduce the noise impact. Second, a concept cluster-based topic drifting detection method is proposed to effectively track hidden topic drifts. Finally, extensive studies demonstrate that as compared to several well-known concept drifting detection methods in data stream, our approach can detect topic drifts effectively, and it enables handling short text streams effectively while maintaining the efficiency as compared to several state-of-the-art short text classification approaches.