Clustering Algorithm For Large Data Sets Research Articles

POFCM: A Parallel Fuzzy Clustering Algorithm for Large Datasets

Clustering algorithms have proven to be a useful tool to extract knowledge and support decision making by processing large volumes of data. Hard and fuzzy clustering algorithms have been used successfully to identify patterns and trends in many areas, such as finance, healthcare, and marketing. However, these algorithms significantly increase their solution time as the size of the datasets to be solved increase, making their use unfeasible. In this sense, the parallel processing of algorithms has proven to be an efficient alternative to reduce their solution time. It has been established that the parallel implementation of algorithms requires its redesign to optimise the hardware resources of the platform that will be used. In this article, we propose a new parallel implementation of the Hybrid OK-Means Fuzzy C-Means (HOFCM) algorithm, which is an efficient variant of Fuzzy C-Means, in OpenMP. An advantage of using OpenMP is its scalability. The efficiency of the implementation is compared against the HOFCM algorithm. The experimental results of processing large real and synthetic datasets show that our implementation tends to more efficiently solve instances with a large number of clusters and dimensions. Additionally, the implementation shows excellent results concerning speedup and parallel efficiency metrics. Our main contribution is a Fuzzy clustering algorithm for large datasets that is scalable and not limited to a specific domain.

An improved density-based single sliding clustering algorithm for large datasets in the cultural information system

This research introduces a technique for using an unsupervised machine learning method to data mine cultural datasets on the cultural information system (CIS). This has been referred to as web data mining for cultural data analytics and it will help web users analyze large, historical cultural dataset objects within web clusters. However, current techniques are ineffective at classifying data points into specific CIS clusters. This issue was resolved by using a Hybridized Single Sliding Centroid with Density-based Clustering (HSSCDC) to optimize data analytic techniques. Density-based clustering improves the clustering precision of the cultural analysis system (CAS) and optimizes window size. Experimental validation shows that the proposed HSSCDC automatically identifies data patterns in CAS web clusters with greater precision, less computational complexity, and less error than previous methods. This will help web users predict historical dataset objects more accurately and minimize computational time.

DHC: A Distributed Hierarchical Clustering Algorithm for Large Datasets

Hierarchical clustering is a classical method to provide a hierarchical representation for the purpose of data analysis. However, in practical applications, it is difficult to deal with massive datasets due to their high computation complexity. To overcome this challenge, this paper presents a novel distributed storage and computation hierarchical clustering algorithm, which has a lower time complexity than the standard hierarchical clustering algorithms. Our proposed approach is suitable for hierarchical clustering on massive datasets, which has the following advantages. First, the algorithm is able to store massive dataset exceeding the main memory space by using distributed storage nodes. Second, the algorithm is able to efficiently process nearest neighbor searching along parallel lines by using distributed computation at each node. Extensive experiments are carried out to validate the effectiveness of the DHC algorithm. Experimental results demonstrate that the algorithm is 10 times faster than the standard hierarchical clustering algorithm, which is an effective and flexible distributed algorithm of hierarchical clustering for massive datasets.

Scalable Clustering by Iterative Partitioning and Point Attractor Representation

Clustering very large datasets while preserving cluster quality remains a challenging data-mining task to date. In this paper, we propose an effective scalable clustering algorithm for large datasets that builds upon the concept of synchronization. Inherited from the powerful concept of synchronization, the proposed algorithm, CIPA (Clustering by Iterative Partitioning and Point Attractor Representations), is capable of handling very large datasets by iteratively partitioning them into thousands of subsets and clustering each subset separately. Using dynamic clustering by synchronization, each subset is then represented by a set of point attractors and outliers. Finally, CIPA identifies the cluster structure of the original dataset by clustering the newly generated dataset consisting of points attractors and outliers from all subsets. We demonstrate that our new scalable clustering approach has several attractive benefits: (a) CIPA faithfully captures the cluster structure of the original data by performing clustering on each separate data iteratively instead of using any sampling or statistical summarization technique. (b) It allows clustering very large datasets efficiently with high cluster quality. (c) CIPA is parallelizable and also suitable for distributed data. Extensive experiments demonstrate the effectiveness and efficiency of our approach.

PySAPC, a python package for sparse affinity propagation clustering: Application to odontogenesis whole genome time series gene-expression data

BackgroundDevelopmental dental anomalies are common forms of congenital defects. The molecular mechanisms of dental anomalies are poorly understood. Systematic approaches such as clustering genes based on similar expression patterns could identify novel genes involved in dental anomalies and provide a framework for understanding molecular regulatory mechanisms of these genes during tooth development (odontogenesis). MethodsA python package (pySAPC) of sparse affinity propagation clustering algorithm for large datasets was developed. Whole genome pair-wise similarity was calculated based on expression pattern similarity based on 45 microarrays of several stages during odontogenesis. ResultspySAPC identified 743 gene clusters based on expression pattern similarity during mouse tooth development. Three clusters are significantly enriched for genes associated with dental anomalies (with FDR <0.1). The three clusters of genes have distinct expression patterns during odontogenesis. ConclusionsClustering genes based on similar expression profiles recovered several known regulatory relationships for genes involved in odontogenesis, as well as many novel genes that may be involved with the same genetic pathways as genes that have already been shown to contribute to dental defects. General significanceBy using sparse similarity matrix, pySAPC use much less memory and CPU time compared with the original affinity propagation program that uses a full similarity matrix. This python package will be useful for many applications where dataset(s) are too large to use full similarity matrix. This article is part of a Special Issue entitled “System Genetics” Guest Editor: Dr. Yudong Cai and Dr. Tao Huang.

A parallel attribute reduction algorithm based on Affinity Propagation clustering

As information technology is developing rapidly, massive and high dimensional data sets have appeared in abundance. The existing attribute reduction methods are encountering bottleneck problem of timeliness and spatiality. AP(Affinity Propagation) is an efficient and fast clustering algorithm for large dataset compared with the existing clustering algorithms. This paper discusses attribute clustering method in order to reduce attributes and provides a kind of parallel attribute reduction algorithm based on Affinity Propagation (APPAR) clustering. The attribute set is clustered into several subsets by Affinity Propagation algorithm first, and then the reductions of these subsets are proposed concurrently in order to get attribute reduction set of the whole data set. The whole algorithm has been improved in the two sides so as to largely increase the algorithm's speed. Experimental results show that the APPAR method is outperforming traditional attribute reduction algorithm for huge and high dimensional dataset processing.

A new clustering algorithm for large datasets

The Circle algorithm was proposed for large datasets. The idea of the algorithm is to find a set of vertices that are close to each other and far from other vertices. This algorithm makes use of the connection between clustering aggregation and the problem of correlation clustering. The best deterministic approximation algorithm was provided for the variation of the correlation of clustering problem, and showed how sampling can be used to scale the algorithms for large datasets. An extensive empirical evaluation was given for the usefulness of the problem and the solutions. The results show that this method achieves more than 50% reduction in the running time without sacrificing the quality of the clustering.

Ensemble of clustering algorithms for large datasets

The ensemble clustering algorithm ECCA (Ensemble of Combined Clustering Algorithms) for processing large datasets is proposed and theoretically substantiated. Results of an experimental study of the algorithm on simulated and real data proving its effectiveness are presented

Scalable Varied Density Clustering Algorithm for Large Datasets

Finding clusters in data is a challenging problem especially when the clusters are being of widely varied shapes, sizes, and densities. Herein a new scalable clustering technique which addresses all these issues is proposed. In data mining, the purpose of data clustering is to identify useful patterns in the underlying dataset. Within the last several years, many clustering algorithms have been proposed in this area of research. Among all these proposed methods, density clustering methods are the most important due to their high ability to detect arbitrary shaped clusters. Moreover these methods often show good noise-handling capabilities, where clusters are defined as regions of typical densities separated by low or no density regions. In this paper, we aim at enhancing the well-known algorithm DBSCAN, to make it scalable and able to discover clusters from uneven datasets in which clusters are regions of homogenous densities. We achieved the scalability of the proposed algorithm by using the k-means algorithm to get initial partition of the dataset, applying the enhanced DBSCAN on each partition, and then using a merging process to get the actual natural number of clusters in the underlying dataset. This means the proposed algorithm consists of three stages. Experimental results using synthetic datasets show that the proposed clustering algorithm is faster and more scalable than the enhanced DBSCAN counterpart.

Semi-Supervised Clustering Based on Affinity Propagation Algorithm

PDF HTML阅读 XML下载 导出引用 引用提醒 基于近邻传播算法的半监督聚类 DOI: 作者: 作者单位: 作者简介: 通讯作者: 中图分类号: 基金项目: Supported by the National Natural Science Foundation of China under Grant No.60875031 (国家自然科学基金); the National Basic Research Program of China under Grant No.2007CB311002 (国家重点基础研究发展计划(973)); the Program for New Century Excellent Talents in University of China under Grant No.NECT-06-0078 (新世纪优秀人才支持计划); the Research Fund for the Doctoral Program of Higher Education of the Ministry of Education of China under Grant No.20050004008 (教育部高等学校博士学科点专项科研基金); the Fok Ying-Tong Education Foundation for Young Teachers in the Higher Education Institutions of China under Grant No.101068 (霍英东教育基金会高等院校青年教师基金) Semi-Supervised Clustering Based on Affinity Propagation Algorithm Author: Affiliation: Fund Project: 摘要 | 图/表 | 访问统计 | 参考文献 | 相似文献 | 引证文献 | 资源附件 | 文章评论 摘要:提出了一种基于近邻传播(affinity propagation,简称AP)算法的半监督聚类方法.AP是在数据点的相似度矩阵的基础上进行聚类.对于规模很大的数据集,AP算法是一种快速、有效的聚类方法,这是其他传统的聚类算法所不能及的,比如:K中心聚类算法.但是,对于一些聚类结构比较复杂的数据集,AP算法往往不能得到很好的聚类结果.使用已知的标签数据或者成对点约束对数据形成的相似度矩阵进行调整,进而达到提高AP算法的聚类性能.实验结果表明,该方法不仅提高了AP对复杂数据的聚类结果,而且在约束对数量较多时,该方法要优于相关比对算法. Abstract:A semi-supervised clustering method based on affinity propagation (AP) algorithm is proposed in this paper. AP takes as input measures of similarity between pairs of data points. AP is an efficient and fast clustering algorithm for large dataset compared with the existing clustering algorithms, such as K-center clustering. But for the datasets with complex cluster structures, it cannot produce good clustering results. It can improve the clustering performance of AP by using the priori known labeled data or pairwise constraints to adjust the similarity matrix. Experimental results show that such method indeed reaches its goal for complex datasets, and this method outperforms the comparative methods when there are a large number of pairwise constraints. 参考文献 相似文献 引证文献