Single-pass Clustering Algorithm Research Articles

Intelligent Analysis and Positioning of Political Public Opinion in Universities

With the rapid development of Internet technology, the network has become an indispensable way of life for undergraduates. The correct guidance of public opinion has also become an important thing in the ideological work of universities. Undergraduates are in an important period of formation and development of thoughts that they are easily to be incited by cyber-rumors. Therefore, it is particularly important to obtain the data of political public opinion in universities and position the hot topics for early detection of political public opinion tendency, which can also avoid the outbreak of major security incidents. With such consideration, this paper obtains multi-source political public opinion data from BBS, Tieba and Weibo of SUN YAT-SEN UNIVERSITY (SYSU) through crawler. We study a text feature extraction method based on Word2Vec & LDA (Latent Dirichlet Allocation), which improves the high-dimensional sparsity in traditional Vector Space Model (VSM) text representation. Meanwhile, based on the classical Single-pass clustering algorithm, this paper studies the Single-pass & HAC clustering algorithm. In addition, a measurement method of hot topic is defined to calculate the heat value of political public opinion. Dictionary and rule based method is used to improve the accuracy of sentiment tendency analysis. The experimental results demonstrate that the effect of topic detection and positioning based on LDA & Word2Vec and Single-pass & HAC algorithm is better than other methods.

Text Clustering Incremental Algorithm in Sensitive Topic Detection

With the rapid development of Internet technology, the influence of online consensus continues to expand. How to quickly and effectively discover sensitive topics and keep track of those topics has become an important research recently. Text clustering can aggregate news texts with the same or similar content to achieve the purpose of discovering topics automatically. Make improvement to clustering algorithm according to different media types is the main research direction. Although the existing typical clustering algorithms have certain advantages, they all face constraints on data size and data characteristics in specific applications. There is no existing algorithm can fully adapt to these characteristics. Although the application of more Single-pass algorithms in the (TDT) field can realize the discovery and tracking of topics, there are disadvantages of poor accuracy and slow speed under massive data. According to the dynamic evolution characteristics of online consensus, this paper proposes an incremental text clustering algorithm based on Single-pass, which optimizes the clustering accuracy and efficiency of massive news. Based on the real online news texts from the online consensus analysis system, we conduct an experiment to test and verify the feasibility and effectiveness of the algorithm we proposed. The result shows that the new algorithm is much more efficient compared to the original Single-pass clustering algorithm. In the real application, the new incremental text clustering algorithm basically meets the real-time demand of online topic detection and has a certain practical value.

Single-Pass Clustering Algorithm Based on Storm

The dramatically increasing volume of data makes the computational complexity of traditional clustering algorithm rise rapidly accordingly, which leads to the longer time. So as to improve the efficiency of the stream data clustering, a distributed real-time clustering algorithm (S-Single-Pass) based on the classic Single-Pass [1] algorithm and Storm [2] computation framework was designed in this paper. By employing this kind of method in the Topic Detection and Tracking (TDT) [3], the real-time performance of topic detection arises effectively. The proposed method splits the clustering process into two parts: one part is to form clusters for the multi-thread parallel clustering, the other part is to merge the generated clusters in the previous process and update the global clusters. Through the experimental results, the conclusion can be drawn that the proposed method have the nearly same clustering accuracy as the traditional Single-Pass algorithm and the clustering accuracy remains steady, computing rate increases linearly when increasing the number of cluster machines and nodes (processing threads).

基于LDA模型的微信图书馆热点话题检测

为使图书馆工作人员免受大量冗余信息的困扰，实时了解广大师生的需求及关注热点，面向微信图书馆，本文给出一种基于LDA模型的微信热点话题检测方法。该方法首先通过构建图书馆领域专业词典合并特征词，其次应用LDA模型表示微信文本信息，最后采用主题相似度计算文本间的相似度，进而利用Single-Pass聚类算法识别热点话题。实验结果表明，该方法能够有效地对微信图书馆上的数据进行话题检测，在准确率、召回率和F1值上均有不错的效果。 In order to make the library staff relieve from a large amount of redundant information and real-time understanding of the needs of teachers and students, for WeChat library, in the paper, the method of hotspot topic detection based on model Latent Dirichlet Allocation (LDA) was pro-posed. The method first merged the characteristic words by constructing the professional dictionary in the library field, and then all the texts of WeChat were described by model LDA. Finally, the similarity between texts was calculated by topic similarity, and then the Single-Pass clustering algorithm was used to cluster WeChat data and found hotspot topics. The experimental results show that this method can effectively identify hotspot topics, and achieve good results in precision, recall and F-measure.

Online Hot Topic Detection Based on Segmented Timeline and Aging Theory

With a great deal of digitized textual information now available on the internet, it is almost impossible for people to assimilate all the information timely. Therefore, the technologies of topic detection and tracking are used for constructing news topics from news stories in order to bring convenience to people. However, traditional topic detection methods are not always so effective in detecting emerging hot news topics in a short period of time, and most topic detection methods use single-pass clustering algorithm which is with low accuracy and very sensitive to the input sequence of news documents. In order to improve clustering accuracy, we utilize a temporal distance factor to segment timeline into equal parts and propose a novel two-times single-pass clustering algorithm to deal with news stories in each part of timeline separately. Moreover, the aging theory is combined with our approach to build life-span model of topics from which we can obtain variation trend of hotness value of topics. The results of experiments show that our approach is effective and the life circle model of topics established by our method can conform to reality well.

Hierarchy Topic Detection and Hot Topic Identification

Due to the problem that the existing topic detection algorithms can not satisfy accuracy,real time and topic hierarchical clustering at the same time, this article builds a hierarchy topic detection algorithm based on improved single pass clustering algorithm. In addition, using public opinion evaluation indexes to analyze topic temperature,the method proposed in this paper can detect hot topics accurately and timely while showing the hierarchical structure of the topic .

Clustering Analysis on E-commerce Transaction Based on K-means Clustering

Based on the density, increment and grid etc, shortcomings like the bad elasticity, weak handling ability of high-dimensional data, sensitive to time sequence of data, bad independence of parameters and weak handling ability of noise are usually existed in clustering algorithm when facing a large number of high-dimensional transaction data. Making experiments by sampling data samples of the 300 mobile phones of Taobao, the following conclusions can be obtained: compared with Single-pass clustering algorithm, the K-means clustering algorithm has a high intra-class dissimilarity and inter-class similarity when analyzing e-commerce transaction. In addition, the K-means clustering algorithm has very high efficiency and strong elasticity when dealing with a large number of data items. However, clustering effects of this algorithm are affected by clustering number and initial positions of clustering center. Therefore, it is easy to show the local optimization for clustering results. Therefore, how to determine clustering number and initial positions of the clustering center of this algorithm is still the important job to be researched in the future.

Micro-blog hot topics detection method based on user role orientation

To solve the low extraction efficiency for extracting hot topics in huge amounts of micro-blog data, a new topics detection method based on user role orientation was proposed. Firstly, some noise data of parts of users were filtered out by user role orientation. Secondly, the feature weight was calculated by the Term Frequency-Inverse Document Frequency( TFIDF) function combined with semantic similarity to reduce the error caused by semantic expression. Then, the improved Single-Pass clustering algorithm was used to extract the topics of micro-blog. Lastly, the heat evaluation of micro-blog topics was made according to the number of reposts and comments, thus the hot topics were found. The results show that the average missing rate and false detection rate respectively decrease by 12. 09% and 2. 37%, and further indicate the topic detection accuracy rate is effectively improved and the method is feasible.

Automatic workflow for the classification of local DNA conformations

BackgroundA growing number of crystal and NMR structures reveals a considerable structural polymorphism of DNA architecture going well beyond the usual image of a double helical molecule. DNA is highly variable with dinucleotide steps exhibiting a substantial flexibility in a sequence-dependent manner. An analysis of the conformational space of the DNA backbone and the enhancement of our understanding of the conformational dependencies in DNA are therefore important for full comprehension of DNA structural polymorphism.ResultsA detailed classification of local DNA conformations based on the technique of Fourier averaging was published in our previous work. However, this procedure requires a considerable amount of manual work. To overcome this limitation we developed an automatic classification method consisting of the combination of supervised and unsupervised approaches. A proposed workflow is composed of k-NN method followed by a non-hierarchical single-pass clustering algorithm. We applied this workflow to analyze 816 X-ray and 664 NMR DNA structures released till February 2013. We identified and annotated six new conformers, and we assigned four of these conformers to two structurally important DNA families: guanine quadruplexes and Holliday (four-way) junctions. We also compared populations of the assigned conformers in the dataset of X-ray and NMR structures.ConclusionsIn the present work we developed a machine learning workflow for the automatic classification of dinucleotide conformations. Dinucleotides with unassigned conformations can be either classified into one of already known 24 classes or they can be flagged as unclassifiable. The proposed machine learning workflow permits identification of new classes among so far unclassifiable data, and we identified and annotated six new conformations in the X-ray structures released since our previous analysis. The results illustrate the utility of machine learning approaches in the classification of local DNA conformations.

Orientation Mining-Driven Approach to Analyze Web Public Sentiment

In recent years, Internet provides a unique opportunity to express and spread public sentiment, which makes the web contents becoming the largest information source of public sentiment. Since web public sentiment reflects people’s attitude to society and politics, the public opinion’s orientation is significant to decision-makers. In this paper, we utilize VSM (vector space model) to present the text orientation of web information and offer data-mining approaches to analyze public opinion’s orientation, which can assist decision-makers to steer social information and guide the web public sentiment. To achieve the goal of text orientation analysis, two ways are proposed. Firstly, a novel text orientation analysis method is described to analyze the orientation of original web postings and their replies. Secondly, an improved single-pass clustering algorithm is introduced to cluster the subject of web discussion and discover the hot topics.We also construct a prototype system, named WPSAS (web public sentiment analysis system), as experimental platform to validate the presented methodology. The experimental results show that our methods are effective and efficient.

Contextual correlation based thread detection in short text message streams

Short text message streams are produced by Instant Messaging and Short Message Service which are wildly used nowadays. Each stream contains more than one thread usually. Detecting threads in the streams is helpful to various applications, such as business intelligence, investigation of crime and public opinion analysis. Existing works which are mainly based on text similarity encounter many challenges including the sparse eigenvector and anomaly of short text message. This paper introduces a novel concept of contextual correlation instead of the traditional text similarity into single-pass clustering algorithm to cover the challenges of thread detection. We firstly analyze the contextually correlative nature of conversations in short text message streams, and then propose an unsupervised method to compute the correlative degree. As a reference, a single-pass algorithm employing the contextual correlation is developed to detect threads in massive short text stream. Experiments on large real-life online chat logs show that our approach improves the performance by 11% when compared with the best similarity-based algorithm in terms of F1 measure.

사건중심 뉴스기사 자동요약을 위한 사건탐지 기법에 관한 연구

This study investigates an event detection method with the aim of generating an event-focused news summary from a set of news articles on a certain event using a multi-document summarization technique. The event detection method first classifies news articles into the event related topic categories by employing a SVM classifier and then creates event clusters containing news articles on an event by a modified single pass clustering algorithm. The clustering algorithm applies a time penalty function as well as cluster partitioning to enhance the clustering performance. It was found that the event detection method proposed in this study showed a satisfactory performance in terms of both the F-measure and the detection cost.

An Aging Theory for Event Life-Cycle Modeling

An event can be described by a sequence of chronological documents from several information sources that together describe a story or happening. The goal of event detection and tracking is to automatically identify events and their associated documents during their life cycles. Conventional document clustering and classification techniques cannot effectively detect and track sequential events, as they ignore the temporal relationships among documents related to an event. The life cycle of an event is analogous to living beings. With abundant nourishment (i.e., related documents for the event), the life cycle is prolonged; conversely, an event or living fades away when nourishment is exhausted. Improper tracking algorithms often unnecessarily prolong or shorten the life cycle of detected events. In this paper, we propose an aging theory to model the life cycle of sequential events, which incorporates a traditional single-pass clustering algorithm to detect and track events. Our experiment results show that the proposed method achieves a better overall performance for both long-running and short-term events than previous approaches. Moreover, we find that the aging parameters of the aging schemes are profile dependent and that using proper profile-specific aging parameters improves the detection and tracking performance further

Multispectral Analysis of IRAS Sky Flux Maps

We have tested the performance of principal components analysis and a single-pass clustering algorithm to identify different components of the cosmic dust. Applying these techniques on a training set of 2500 points extracted from the PL51 IRAS maps we recognized two main components with temperatures of 180 K and 28 K.