Clustering Of Large Data Sets Research Articles

AI-Enhanced evaluation of YouTube content on post-surgical incontinence following pelvic cancer treatment

BackgroundSeveral pelvic area cancers exhibit high incidence rates, and their surgical treatment can result in adverse effects such as urinary and fecal incontinence, significantly impacting patients' quality of life. Post-surgery incontinence is a significant concern, with prevalence rates ranging from 25 to 45% for urinary incontinence and 9–68% for fecal incontinence. Cancer survivors are increasingly turning to YouTube as a platform to connect with others, yet caution is warranted as misinformation is prevalent. ObjectiveThis study aims to evaluate the information quality in YouTube videos about post-surgical incontinence after pelvic area cancer surgery. MethodsA YouTube search for "Incontinence after cancer surgery" yielded 108 videos, which were subsequently analyzed. To evaluate these videos, several quality assessment tools were utilized, including DISCERN, GQS, JAMA, PEMAT, and MQ-VET. Statistical analyses, such as descriptive statistics and intercorrelation tests, were employed to assess various video attributes, including characteristics, popularity, educational value, quality, and reliability. Also, artificial intelligence techniques like PCA, t-SNE, and UMAP were used for data analysis. HeatMap and Hierarchical Clustering Dendrogram techniques validated the Machine Learning results. ResultsThe quality scales presented a high level of correlation one with each other (p < 0.01) and the Artificial Intelligence-based techniques presented clear clustering representations of the dataset samples, which were reinforced by the Heat Map and Hierarchical Clustering Dendrogram. ConclusionsYouTube videos on "Incontinence after Cancer Surgery" present a "High" quality across multiple scales. The use of AI tools, like PCA, t-SNE, and UMAP, is highlighted for clustering large health datasets, improving data visualization, pattern recognition, and complex healthcare analysis.

Scellpam: an R package/C++ library to perform parallel partitioning around medoids on scRNAseq data sets

Background Partitioning around medoids (PAM) is one of the most widely used and successful clustering method in many fields. One of its key advantages is that it only requires a distance or a dissimilarity between the individuals, and the fact that cluster centers are actual points in the data set means they can be taken as reliable representatives of their classes. However, its wider application is hampered by the large amount of memory needed to store the distance matrix (quadratic on the number of individuals) and also by the high computational cost of computing such distance matrix and, less importantly, by the cost of the clustering algorithm itself.Results Therefore, new software has been provided that addresses these issues. This software, provided under GPL license and usable as either an R package or a C++ library, calculates in parallel the distance matrix for different distances/dissimilarities (L_1, L_2, Pearson, cosine and weighted Euclidean) and also implements a parallel fast version of PAM (FASTPAM1) using any data type to reduce memory usage. Moreover, the parallel implementation uses all the cores available in modern computers which greatly reduces the execution time. Besides its general application, the software is especially useful for processing data of single cell experiments. It has been tested in problems including clustering of single cell experiments with up to 289,000 cells with the expression of about 29,000 genes per cell.Conclusions Comparisons with other current packages in terms of execution time have been made. The method greatly outperforms the available R packages for distance matrix calculation and also improves the packages that implement the PAM itself. The software is available as an R package at https://CRAN.R-project.org/package=scellpam and as C++ libraries at https://github.com/JdMDE/jmatlib and https://github.com/JdMDE/ppamlib The package is useful for single cell RNA-seq studies but it is also applicable in other contexts where clustering of large data sets is required.

Big data: an optimized approach for cluster initialization

The k-means, one of the most widely used clustering algorithm, is not only faster in computation but also produces comparatively better clusters. However, it has two major downsides, first it is sensitive to initialize k value and secondly, especially for larger datasets, the number of iterations could be very large, making it computationally hard. In order to address these issues, we proposed a scalable and cost-effective algorithm, called R-k-means, which provides an optimized solution for better clustering large scale high-dimensional datasets. The algorithm first selects O(R) initial points then reselect O(l) better initial points, using distance probability from dataset. These points are then again clustered into k initial points. An empirical study in a controlled environment was conducted using both simulated and real datasets. Experimental results showed that the proposed approach outperformed as compared to the previous approaches when the size of data increases with increasing number of dimensions.

HEADSS: HiErArchical Data Splitting and Stitching software for non-distributed clustering algorithms

The increase in data volume is challenging the suitability of non-distributed and non-scalable algorithms, despite advancements in hardware. An example of this challenge is clustering. Considering that optimal clustering algorithms scale poorly with increased data volume or are intrinsically non-distributed, accurate clustering of large datasets is increasingly resource-heavy, relying on substantial and expensive compute nodes. This scenario forces users to choose between accuracy and scalability. In this work, we introduce HiErArchical Data Splitting and Stitching (HEADSS), a Python package designed to facilitate clustering at scale. By automating the splitting and stitching, it allows repeatable handling, and removal, of edge effects. We implement HEADSS in conjunction with HDBSCAN, where we achieve orders of magnitude reduction in single node memory requirements for both non-distributed and distributed implementations, with the latter offering similar order of magnitude reductions in total run times while recovering analogous accuracy. Furthermore, our method establishes a hierarchy of features by using a subset of clustering features to split the data.11Source code and examples are available at https://github.com/D-Crake/HEADSS.

Multi-Objective Genetic Algorithm for Cluster Analysis of Single-Cell Transcriptomes

Cells are the basic building blocks of human organisms, and the identification of their types and states in transcriptomic data is an important and challenging task. Many of the existing approaches to cell-type prediction are based on clustering methods that optimize only one criterion. In this paper, a multi-objective Genetic Algorithm for cluster analysis is proposed, implemented, and systematically validated on 48 experimental and 60 synthetic datasets. The results demonstrate that the performance and the accuracy of the proposed algorithm are reproducible, stable, and better than those of single-objective clustering methods. Computational run times of multi-objective clustering of large datasets were studied and used in supervised machine learning to accurately predict the execution times of clustering of new single-cell transcriptomes.

An Efficient Method to Accurately Cluster Large Number of High Dimensional Facial Images

Accurately clustering large, high dimensional datasets is a challenging problem in unsupervised learning. K-means is considered to be a fast, widely used and accurate centroid based data partitioning algorithm for spherical datasets. However, its non-determinism and heavy dependence on the selection of initial cluster centers along with vulnerability to noise make it a poor candidate for clustering large datasets with high dimensionality. To overcome these, we develop a novel, nature inspired, centroid based clustering algorithm, inspired from the principles of particle physics. Our method ensures that the convergence to local optima and non-deterministic outputs are avoided. We experiment the method on large datasets of human face images. Besides, our method addresses the problem of outliers and presence of not well-separated data in these datasets. We use a deep learning model for extracting facial features into a vector of 128 dimensions. We validate the quality and accuracy of our methods using different statistical parameters like <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">f-measure, accuracy, error rate, average in group proportion and normalized cluster size rand index</i> . These evaluations show that our method exhibits better accuracy and quality in clustering large face image datasets, in comparison with other existing mechanisms. The strength of our algorithms is more visible as the size of the dataset grows.

Identification and Potential Use of Clusters of Patients With Colorectal Cancer and Patients With Prostate Cancer in Clinical Practice: Explorative Mixed Methods Study.

A steady increase in colorectal and prostate cancer survivors and patients with these cancers is expected in the upcoming years. As a result of primary cancer treatments, patients have numerous additional complaints, increasing the need for cancer aftercare. However, referrals to appropriate cancer aftercare remain inadequate, despite a wide range of aftercare options. Caregivers and patients often do not know which aftercare is the most appropriate for the individual patient. Since characteristics and complaints of patients within a diagnosis group may differ, predefined patient clusters could provide substantive and efficient support for professionals in the conversation about aftercare. By using advanced data analysis methods, clusters of patients who are different from one another within a diagnosis group can be identified. This study had a 2-fold objective: (1) to identify, visualize, and describe potential patient clusters within the colorectal and prostate cancer population and (2) to explore the potential usability of these clusters in clinical practice. First, we used cross-sectional data from patients with colorectal cancer and patients with prostate cancer provided by the population-based PROFILES (Patient-Reported Outcomes Following Initial Treatment and Long-Term Evaluation of Survivorship) registry, which were originally collected between 2008 and 2012. To identify and visualize different clusters among the 2 patient populations, we conducted cluster analyses by applying the K-means algorithm and multiple-factor analyses. Second, in a qualitative study, we presented the patient clusters to patients with prostate, patients with colorectal cancer, and oncology professionals. To assess the usability of these clusters, we held expert panel group interviews. The interviews were video recorded and transcribed. Three researchers independently performed content-directed data analyses to understand and describe the qualitative data. Quotes illustrate the most important results. We identified 3 patient clusters among colorectal cancer cases (n=3989) and 5 patient clusters among prostate cancer cases (n=696), which were described in tabular form. Patient experts (6/8, 75%) and professional experts (17/20, 85%) recognized the patient clustering based on distinguishing variables. However, the tabular form was evaluated as less applicable in clinical practice. Instead, the experts suggested the development of a conversation tool (eg, decision tree) to guide professionals through the hierarchy of variables. In addition, participants suggested that information about possible aftercare initiatives should be offered and integrated. This would also ensure a good overview and seemed to be a precondition for finding suitable aftercare. This study demonstrates that a fully data-driven approach can be used to identify distinguishable and recognizable (ie, in routine care) patient clusters in large data sets within cancer populations. Patient clusters can be a source of support for health professionals in the aftercare conversation. These clusters, when integrated into a smart digital conversation and referral tool, might be an opportunity to improve referral to cancer aftercare. Netherlands Trial Register NL9226; https://trialsearch.who.int/Trial2.aspx?TrialID=NL9226.

Clustering Algorithm for Big Datasets with Mixed Attribute Features under Spark

This paper presents an in-depth study and analysis of large datasets of mixed and attribute features under Spark using a large dataset clustering algorithm. The classical algorithm K-means based on division and the density-based clustering algorithm DPC, which has become more popular in recent years, are selected as the research objects of this paper. Secondly, the original K-means algorithm is improved by combining holdout validation and K-means++ method to address the shortcomings of the K-means algorithm that the number of class clusters K needs to be set in advance and the initial class cluster centers are chosen randomly, which leads to unstable iterations and slow convergence of clustering results. The similarity matrix will be continuously updated during the iterative process. It mainly refers to the process of dividing objects into multiple classes according to the degree of similarity between objects. After the division, the objects within the class are like each other, while the objects between the classes are different from each other. The comparison experiments of the improved algorithm before and after the MovieLens dataset are conducted to verify that the new algorithm has better performance in terms of clustering accuracy and efficiency. Again, to address the drawback that the clustering results in the DPC algorithm rely heavily on the subjective selection of the truncation distance parameter cd, and it is difficult to handle datasets with complex distribution and large density variation, the algorithm can generate the optimal cd adaptively by combining K-nearest neighbors and introducing the distance comparison quantity, which has a better performance by considering the overall and local distribution of the data. The feasibility of the improved method is verified by validating the algorithm with artificial datasets and UCI datasets as well as separation tests. Finally, the parallelized design and implementation of the improved K-means algorithm and CDPC-KNN algorithm are completed by building a Spark clustering environment, and the parallelized algorithm is verified to have much better data processing capability and be more adaptable to the clustering analysis of large-scale data by comparing algorithm string parallelism experiments.

Credibilistic fuzzy clustering based on evolutionary method of crazy cats

The problem of fuzzy clustering of large datasets that are sent for processing in both batch and online modes, based on a credibilistic approach, is considered. To find the global extremum of the credibilistic fuzzy clustering goal function, the modification of the swarm algorithm of crazy cats swarms was introduced, that combined the advantages of evolutionary algorithms and a global random search. It is shown that different search modes are generated by a unified mathematical procedure, some cases of which are known algorithms for both local and global optimizations. The proposed approach is easy to implement and is characterized by the high speed and reliability in problems of multi-extreme fuzzy clustering.

BIRCHSCAN: A sampling method for applying DBSCAN to large datasets

The DBSCAN algorithm is a traditional density-based clustering method. This algorithm allows to identify clusters of different shapes, with the ability to manage noisy patterns in the data. DBSCAN usually presents good results, however it performs several distance calculations in the clustering process. This leads to a low efficiency and its application in large datasets is not recommended. This work presents a new method to apply DBSCAN to a reduced set of elements in order to cluster the entire dataset. Therefore, the method allows clustering large datasets with similar results to the outcoe of DBSCAN on the entire dataset. Experiments results suggest that the proposed technique presents good results and consistency compared to other algorithms with similar approach.

A Robust Distributed Clustering of Large Data Sets on a Grid of Commodity Machines

Distributed clustering algorithms have proven to be effective in dramatically reducing execution time. However, distributed environments are characterized by a high rate of failure. Nodes can easily become unreachable. Furthermore, it is not guaranteed that messages are delivered to their destination. As a result, fault tolerance mechanisms are of paramount importance to achieve resiliency and guarantee continuous progress. In this paper, a fault-tolerant distributed k-means algorithm is proposed on a grid of commodity machines. Machines in such an environment are connected in a peer-to-peer fashion and managed by a gossip protocol with the actor model used as the concurrency model. The fact that no synchronization is needed makes it a good fit for parallel processing. Using the passive replication technique for the leader node and the active replication technique for the workers, the system exhibited robustness against failures. The results showed that the distributed k-means algorithm with no fault-tolerant mechanisms achieved up to a 34% improvement over the Hadoop-based k-means algorithm, while the robust one achieved up to a 12% improvement. The experiments also showed that the overhead, using such techniques, was negligible. Moreover, the results indicated that losing up to 10% of the messages had no real impact on the overall performance.

A Comparative Review of Incremental Clustering Methods for Large Dataset

Several algorithms have developed for analyzing large incremental datasets. Incremental algorithms are relatively efficient in dynamic evolving environment to seek out small clusters in large datasets. Many algorithms have devised for limiting the search space, building, and updating arbitrary shaped clusters in large incremented datasets. Within the real time visualization of real time data, when data in motion and growing dynamically, new data points arrive that generates instant cluster labels. In this paper, the comparative review of Incremental clustering methods for large dataset has done.

A Comparative Study of the Some Methods Used in Constructing Coresets for Clustering Large Datasets

Coresets can be described as a compact subset such that models trained on coresets will also provide a good fit with models trained on full data set. Using coresets, we can scale down a big data to a tiny one to reduce the computational cost of a machine learning problem. In recent years, data scientists have investigated various methods to create coresets with many techniques and approaches, especially for solving the problem of clustering large datasets. In this paper, we make comparisons among four state-of-the-art algorithms: ProTraS by Ros and Guillaume with improvements, Lightweight Coreset by Bachem et al. Adaptive Sampling Coreset by Feldman et al. and a native Farthest-First-Traversal-based coreset construction. We briefly introduce these four algorithms and compare their performances to find out the benefits and drawbacks of each one.

On the Consistency of k-means++ algorithm

We prove in this paper that the expected value of the objective function of the $k$-means++ algorithm for samples converges to population expected value. As $k$-means++, for samples, provides with constant factor approximation for $k$-means objectives, such an approximation can be achieved for the population with increase of the sample size. This result is of potential practical relevance when one is considering using subsampling when clustering large data sets (large data bases).

LERI: Local Exploration for Rare-Category Identification

To identify the data examples of rare categories that form small compact clusters in large data sets, existing approaches mostly require enough labeled data examples as a training set to learn a classifier, assuming that the rare-category clusters are spherical or nearly spherical. Nonetheless, a large enough training set is usually difficult to obtain in practice, and rare categories in many real-world applications often form small compact clusters with arbitrary shapes. In this paper, we investigate how to identify all data examples of a rare category with an arbitrary shape based on only one seed (i.e., a labeled rare-category data example). Instead of finding a compact and spherical local region around the seed, we locally explore the data set from the seed by continuously searching and visiting the $k$ k -nearest neighbors of each newly visited data example. The local exploration connects the data examples in the objective rare category by the relationship of $k$ k -nearest neighbors, and meanwhile, suspected external data examples are filtered out if they are not close enough to any visited data example. Experimental results on both synthetic and real-world data sets are conducted, and the results verify the effectiveness and efficiency of our approach.

Compartment and hub definitions tune metabolic networks for metabolomic interpretations.

BackgroundMetabolic networks represent all chemical reactions that occur between molecular metabolites in an organism’s cells. They offer biological context in which to integrate, analyze, and interpret omic measurements, but their large scale and extensive connectivity present unique challenges. While it is practical to simplify these networks by placing constraints on compartments and hubs, it is unclear how these simplifications alter the structure of metabolic networks and the interpretation of metabolomic experiments.ResultsWe curated and adapted the latest systemic model of human metabolism and developed customizable tools to define metabolic networks with and without compartmentalization in subcellular organelles and with or without inclusion of prolific metabolite hubs. Compartmentalization made networks larger, less dense, and more modular, whereas hubs made networks larger, more dense, and less modular. When present, these hubs also dominated shortest paths in the network, yet their exclusion exposed the subtler prominence of other metabolites that are typically more relevant to metabolomic experiments. We applied the non-compartmental network without metabolite hubs in a retrospective, exploratory analysis of metabolomic measurements from 5 studies on human tissues. Network clusters identified individual reactions that might experience differential regulation between experimental conditions, several of which were not apparent in the original publications.ConclusionsExclusion of specific metabolite hubs exposes modularity in both compartmental and non-compartmental metabolic networks, improving detection of relevant clusters in omic measurements. Better computational detection of metabolic network clusters in large data sets has potential to identify differential regulation of individual genes, transcripts, and proteins.

Cluster Based Segmentation using K-Means Algorithm

In this paper we present about Data Mining and brief detail about the clustering algorithm which is used to cluster categorical data. The algorithm called K means. The K means algorithm is capable of clustering large data sets. Clustering is used to measure the difference between objects by identifying the distance between the pair of objects. These measures include the Euclidean, Minkowski and Manhattan distance. In this paper we used Euclidean distance to measure the distance between objects. Segmentation is within our overall customer database, which ones have something in common. We need to find the groups of people, understand them and make some commercial value from the different groups

Similarity Based Prediction System using Machine Learning Algorithms in Big Data Analytics

Big Data is a noteworthy environment to maintain the diversity of the huge amount of data. The big data utilizes machine learning algorithms to process large datasets which comes from various places such as histories, weblogs, and data repositories, large datasets and data warehousing, etc. In an existing method, most of the data mining approaches might not be able to maintain the large dataset. Using datamining, the big data are having lack of compatibility with database systems and analysis tools; large dataset clustering and analyzing is a big issue in big data. For this reason, the research work uses machine learning algorithms which are implemented in the Hadoop tool to collect and process the large amount of data which is structured, semi-structured or unstructured in a reasonable amount of time. Also, it gives more accurate prediction system and accurate information. Using Machine Learning Algorithm computational cost and complexities is minimized. The overall research work is implemented in the Hadoop tool with the help of the python programming language and it is compared with some existing algorithms. The proposed work tested with suitable parameters such as accuracy, Kappa T and Kappa M.

A method for k-means-like clustering of categorical data

Despite recent efforts, the challenge in clustering categorical and mixed data in the context of big data still remains due to the lack of inherently meaningful measure of similarity between categorical objects and the high computational complexity of existing clustering techniques. While k-means method is well known for its efficiency in clustering large data sets, working only on numerical data prohibits it from being applied for clustering categorical data. In this paper, we aim to develop a novel extension of k-means method for clustering categorical data, making use of an information theoretic-based dissimilarity measure and a kernel-based method for representation of cluster means for categorical objects. Such an approach allows us to formulate the problem of clustering categorical data in the fashion similar to k-means clustering, while a kernel-based definition of centers also provides an interpretation of cluster means being consistent with the statistical interpretation of the cluster means for numerical data. In order to demonstrate the performance of the new clustering method, a series of experiments on real datasets from UCI Machine Learning Repository are conducted and the obtained results are compared with several previously developed algorithms for clustering categorical data.

Tangle-Tree Duality: In Graphs, Matroids and Beyond

We apply a recent duality theorem for tangles in abstract separation systems to derive tangle-type duality theorems for width-parameters in graphs and matroids. We further derive a duality theorem for the existence of clusters in large data sets. Our applications to graphs include new, tangle-type, duality theorems for tree-width, path-width, and tree-decompositions of small adhesion. Conversely, we show that carving width is dual to edge-tangles. For matroids we obtain a duality theorem for tree-width. Our results can be used to derive short proofs of all the classical duality theorems for width parameters in graph minor theory, such as path-width, tree-width, branch-width and rank-width.