Clustering Large Data Research Articles

Clustering large mixed-type data with ordinal variables

AbstractOne of the most frequently used algorithms for clustering data with both numeric and categorical variables is the k-prototypes algorithm, an extension of the well-known k-means clustering. Gower’s distance denotes another popular approach for dealing with mixed-type data and is suitable not only for numeric and categorical but also for ordinal variables. In the paper a modification of the k-prototypes algorithm to Gower’s distance is proposed that ensures convergence. This provides a tool that allows to take into account ordinal information for clustering and can also be used for large data. A simulation study demonstrates convergence, good clustering results as well as small runtimes.

A ReCon-BCALoD clustering algorithm for field monitoring data of marine structures

Monitoring of the extreme marine environmental loads and hazardous dynamic responses of marine structures is critical to maintaining safe field operations. Field monitoring data from marine structures are high-dimensional with high aggregation and weak modes. Furthermore, monitoring data accumulates rapidly over time, which makes it difficult for conventional clustering algorithms to accurately identify the extreme environmental loads and hazardous response conditions experienced by marine structures during their service life. Cluster analysis is an unsupervised learning algorithm widely used in the fields of machine learning and information recognition. Currently, clustering algorithms are mainly used to calculate and analyze large data clusters, but few studies have focused on small clusters. Therefore, this study proposes a ReCon-BCALoD clustering algorithm suitable for long-term monitoring data of marine structures. The ReCon-BCALoD algorithm consists of up and down processes in which the data are clustered based on the ReCon local densities of data points. Applied to field monitoring data, the ReCon-BCALoD clustering algorithm proved capable of automatically determining cluster quantity and effectively identifying small data clusters. This showed that the ReCon-BCALoD clustering algorithm can be used to identify extreme current profile environment conditions and hazardous response conditions of a soft yoke mooring system. These clustering results can be used to improve the safety of in-service structures and provide valid references for future designs of marine structures.

Cluster and network analysis of non-headache symptoms in migraine patients reveals distinct subgroups based on onset age and vestibular-cochlear symptom interconnection.

The present study endeavors to identify natural subgroups of migraine patients based on the patterns of non-headache symptoms, utilizing cluster analysis. Subsequently, network analysis was performed to estimate the structure of symptoms and explore the potential pathophysiology of these findings. A total of 475 patients who met the diagnostic criteria for migraine were surveyed face-to-face during the period of 2019 to 2022. The survey included collecting demographic and symptom data. Four different solutions were generated by the K-means for mixed large data (KAMILA) clustering algorithm, from which the final cluster solutions were selected based on a series of cluster metrics. Subsequently, we performed network analysis using Bayesian Gaussian graphical models (BGGM) to estimate the symptom structure across subgroups and conducted global and pairwise comparisons between structures. Cluster analysis identified two distinct patient groups, and the onset age of migraine proved to be an effective characteristic differentiating the two patient groups. Participants assigned to late-onset group showed a longer course of migraine, higher frequency of monthly headache attacks, and greater tendency toward medication overuse. In contrast, patients in early-onset group exhibited a higher frequency of nausea, vomiting, and phonophobia compared to their counterparts in the other group. The network analysis revealed a different symptom structure between the two groups globally, while the pairwise differences indicated an increasing connection between tinnitus and dizziness, and a decreasing connection between tinnitus and hearing loss in the early-onset group. Utilizing clustering and network analysis, we have identified two distinct non-headache symptom structures of migraine patients with early-onset age and late-onset age. Our findings suggest that the vestibular-cochlear symptoms may differ in the context of different onset ages of migraine patients, which may contribute to a better understanding of the pathology of vestibular-cochlear symptoms in migraine.

Multivariate semiparametric control charts for mixed-type data.

A useful tool that has gained popularity in the Quality Control area is the control chart which monitors a process over time, identifies potential changes, understands variations, and eventually improves the quality and performance of the process. This article introduces a new class of multivariate semiparametric control charts for monitoring multivariate mixed-type data, which comprise both continuous and discrete random variables (rvs). Our methodology leverages ideas from clustering and Statistical Process Control to develop control charts for MIxed-type data. We propose four control chart schemes based on modified versions of the KAy-means for MIxed LArge KAMILA data clustering algorithm, where we assume that the two existing clusters represent the reference and the test sample. The charts are semiparametric, the continuous rvs follow a distribution that belongs in the class of elliptical distributions. Categorical scale rvs follow a multinomial distribution. We present the algorithmic procedures and study the characteristics of the new control charts. The performance of the proposed schemes is evaluated on the basis of the False Alarm Rate and in-control Average Run Length. Finally, we demonstrate the effectiveness and applicability of our proposed methods utilizing real-world data.

Accelerating Reads With In-Network Consistency-Aware Load Balancing

We present FLAIR, a novel approach for accelerating read operations in leader-based consensus protocols. FLAIR leverages the capabilities of the new generation of programmable switches to serve reads from follower replicas without compromising consistency. The core of the new approach is a packet-processing pipeline that can track client requests and system replies, identify consistent replicas, and at line speed, forward read requests to replicas that can serve the read without sacrificing linearizability. An additional benefit of FLAIR is that it facilitates devising novel consistency-aware load balancing techniques. Following the new approach, we designed FlairKV, a key-value store atop Raft. FlairKV implements the processing pipeline using the P4 programming language. We evaluate the benefits of the proposed approach and compare it to previous approaches using a cluster with a Barefoot Tofino switch. Our evaluation indicates that, compared to state-of-the-art alternatives, the proposed approach can bring significant performance gains: up to 42% higher throughput and 35–97% lower latency for most workloads. Furthermore, our evaluation shows that our novel load balancing techniques can cope with heterogeneous load and hardware to achieve higher performance, and that FLAIR can scale to support large data sets and clusters.

Analysis of Artistic Modeling of Opera Stage Clothing Based on Big Data Clustering Algorithm

In order to deal with the problem that the traditional stage costume artistry analysis method cannot correct the results of big data clustering, which leads to deviations in the extraction of costume artistry features, this paper proposes a clothing artistic modeling method based on big data clustering algorithm. The proposed method provides a database for big data clustering by constructing the attribute set of the big data feature sequence training set and, at the same time, constructing a second-order cone programming model to correct the big data. Aiming at the problem that traditional stage costume art analysis methods cannot correct the clustering results of big data. On this basis, the costume elements of the opera stage are segmented, initialized, and transformed into a binary function. Finally, using the convolutional neural network, combining the element segmentation results and the large data clustering space state vector, a feature extraction model of stage costume art is constructed. Experimental results show that the model has good convergence, short time-consuming, high accuracy, and ideal feature recognition capabilities.

Unsupervised Machine Learning for the Identification of Preflare Spectroscopic Signatures

Abstract The study of the preflare environment is of great importance to understanding what drives solar flares. k-means clustering, an unsupervised machine-learning technique, has the ability to cluster large data set in a way that would be impractical or impossible for a human to do. In this paper we present a study using k-means clustering to identify possible preflare signatures in spectroscopic observations of the Mg ii h and k spectral lines made by NASA's Interface Region Imaging Spectrometer. Our analysis finds that spectral profiles showing single-peak Mg ii h and k and single-peaked emission in the Mg ii UV triplet lines are associated with preflare activity up to 40 minutes prior to flaring. Subsequent inversions of these spectral profiles reveal increased temperature and electron density in the chromosphere, which suggest that significant heating events in the chromosphere may be associated with precursor signals to flares.

BETULA: Fast clustering of large data with improved BIRCH CF-Trees

BIRCH clustering is a widely known approach for clustering that has influenced much subsequent research and commercial products. The key contribution of BIRCH is the Clustering Feature tree (CF-Tree), which is a compressed representation of the input data. As new data arrives, the tree is eventually rebuilt to increase the compression. Afterward, the leaves of the tree are used for clustering. Because of the data compression, this method is very scalable. The idea has been adopted, for example, for k-means, data stream, and density-based clustering. Clustering features used by BIRCH are simple summary statistics that can easily be updated with new data: the number of points, the linear sums, and the sum of squared values. Unfortunately, how the sum of squares is then used in BIRCH is prone to catastrophic cancellation. We introduce a replacement cluster feature that does not have this numeric problem, that is not much more expensive to maintain, and which makes many computations simpler and, hence, more efficient. These cluster features can also easily be used in other work derived from BIRCH such as algorithms for streaming data. In the experiments, we demonstrate the numerical problem and compare the performance of the original algorithm compared to the improved cluster features. We furthermore explain how to improve clustering with different algorithms such as hierarchical clustering, k-means, k-means++ and Gaussian mixture modeling by using the variance information stored in the cluster features to obtain high-quality result approximations in a substantially reduced runtime (up to 500× faster in our experiments).

Visual Clustering Factors in Scatterplots.

Cluster analysis is an important technique in data analysis. However, there is no encompassing theory on scatterplots to evaluate clustering. Human visual perception is regarded as a gold standard to evaluate clustering. The cluster analysis based on human visual perception requires the participation of many probands, to obtain diverse data, and hence is a challenge to do. We contribute an empirical and data-driven study on human perception for visual clustering of large scatterplot data. First, we systematically construct and label a large, publicly available scatterplot dataset. Second, we carry out a qualitative analysis based on the dataset and summarize the influence of visual factors on clustering perception. Third, we use the labeled datasets to train a deep neural network for modeling human visual clustering perception. Our experiments show that the data-driven model successfully models the human visual perception, and outperforms conventional clustering algorithms in synthetic and real datasets.

Direct Digitalization Devices in Today’s Dental Practice: Intra Oral Scanners

Rapidly developing technologies and new changes in modern dentistry have led to more effective solutions to be used in our clinical practice. The digital workflow includes some steps, such as data acquisition, digitalization, designing, and subsequently manufacturing by using digital devices. In this process, acquiring the images from patients mouth, which is named computer-aided impression (CAI), is the first step of the digital workflow and named as digitalization. The need to skip many time-consuming intermediate steps and reduce the possibility of fabrication errors has made the digital workflow a popular choice for dentists and dental technicians. The popularity of intraoral scanners has increased rapidly since the 1980s. With the development of camera systems and software engineering, many companies have presented many different IOS devices, which are based on different software systems. This situation causes dentists to get confused due to the formation of a very large data cluster. IOS devices are used to capture the digital data and they are an effective alternative to conventional impressions in many ways, such as increased patient comfort, time efficiency, hygiene, predictability, and precise results. Clinicians should have enough technical information about the IOS they use to obtain optimal results in fabricated restorations. The aim of this review is to evaluate the scanning technologies of intraoral scanners, their advantages and disadvantages in clinical practice, scanning strategies, accuracy of scanning processes, and software protocols.

MapReduce fuzzy C-means ensemble clustering with gentle AdaBoost for big data analytics

Big data clustering is one of the significant processes employed in numerous application domains. Existing clustering algorithms do not cope with large-scale data, resulting in higher false positive rate. In order to cluster such large datasets with higher accuracy, MapReduce gradient descent gentle AdaBoost clustering (MGDGAC) technique is proposed. The MGDGAC technique designs MapReduce fuzzy C-means (MFCM) clustering where the large dataset is initially subdivided into a number of chunks which are executed in parallel on different nodes to effectively perform clustering processes with minimal time. The data with larger membership value are grouped in the cluster with help of mappers. Then, reducer in MFCM clustering re-estimates the centroid value and iteratively fed to the mapper again until it attains a particular iteration and groups the similar data together. Finally, MGDGAC technique applies gentle AdaBoost with intention of reducing the training error of large data clustering.

Research on network security defence based on big data clustering algorithms

To improve the security of network in big data era, the improved clustering algorithm is applied to carry out network security defence. Firstly, application of large data clustering algorithms in network security defence is analysed. Secondly, network security defence model is studied, and corresponding mathematical model is designed. Thirdly, the improved clustering algorithm based on big data is established through analysis text requirements and data characteristics. Finally, simulation analysis is carried out, and the effectiveness of the proposed algorithm is verified. The theory analysis results show that the proposed model can provide the theoretical basis for designing network information security defence system.

MAPREDUCE FUZZY C-MEANS ENSEMBLE CLUSTERING WITH GENTLE ADABOOST FOR BIG DATA ANALYTICS

Big data clustering is one of the significant processes employed in numerous application domains. Existing clustering algorithms do not cope with large-scale data, resulting in higher false positive rate. In order to cluster such large datasets with higher accuracy, MapReduce gradient descent gentle AdaBoost clustering (MGDGAC) technique is proposed. The MGDGAC technique designs MapReduce fuzzy C-means (MFCM) clustering where the large dataset is initially subdivided into a number of chunks which are executed in parallel on different nodes to effectively perform clustering processes with minimal time. The data with larger membership value are grouped in the cluster with help of mappers. Then, reducer in MFCM clustering re-estimates the centroid value and iteratively fed to the mapper again until it attains a particular iteration and groups the similar data together. Finally, MGDGAC technique applies gentle AdaBoost with intention of reducing the training error of large data clustering.

Data-parallel clustering algorithm based on mutual information mining of joint condition

In order to improve the reliability service ability of the cloud storage database, the data parallel clustering process is carried out, and the data parallel clustering algorithm based on the mutual information mining of the joint condition is proposed. a large data configuration structure model of a cloud environment virtual resource is constructed, data compression and characteristic reconstruction are carried out by adopting an online dictionary learning method, a regression analysis of the cloud environment virtual resource configuration data and a point cloud structure recombination are carried out in combination with a non-linear statistical sequence analysis method, the parallel characteristic scheduling of the large data of the virtual resources of the cloud environment is realized, the mutual information feature quantity of the joint condition is mined, the characteristic quantity of the mining is subjected to the characteristic filtering and the attribute set merging processing by adopting the fuzzy C-means clustering algorithm, by using the self-adaptive optimization algorithm, the automatic retrieval of the fuzzy clustering center is carried out, and the parallel clustering optimization of the large data is realized. The simulation results show that the classification performance of the large data clustering of the cloud environment virtual resources is good, the property classification and fusion capability is high, and the error rate is lower.

Clustering of large scale QoS time series data in federated clouds using improved variable Chromosome Length Genetic Algorithm (CQGA)

Service monitoring in federated clouds generates large scale QoS time series data with various unknown, frequent and abnormal patterns. This could be associated with inaccurate resource provisioning and avoid violations through predictive and preventive actions. A sufficient intelligence in the form of expert system for decision support is needed in such situations. Therefore, the main challenge here is to efficiently discover unknown frequent and abnormal patterns from QoS time series data of federated clouds. On the other hand, QoS time series data in federated clouds is unlabeled and consists of frequent and abnormal structures. Studies showed that clustering is the most common and efficient method to discover interesting patterns and structures from unlabeled data. But, clustering is normally associated with time overhead that should be optimized as well as accuracy issues mainly in connection with convergence and finding an optimum number of clusters. This work proposes a new genetic based clustering algorithm that shows better accuracy and speed in comparison to state-of-the-art methods. Furthermore, the proposed algorithm can find the optimum number of clusters concurrently with the clustering itself. Achieved accuracy and convergence of the proposed method in the experimental results assure its use in expert systems, mainly for resource provisioning and further autonomous decision making situations in federated clouds. In addition to the scientific impact of this paper, the proposed method can be used by federated cloud service providers in practice.

Gene encoder: a feature selection technique through unsupervised deep learning-based clustering for large gene expression data

Cancer is a severe condition of uncontrolled cell division that results in a tumor formation that spreads to other tissues of the body. Therefore, the development of new medication and treatment methods for this is in demand. Classification of microarray data plays a vital role in handling such situations. The relevant gene selection is an important step for the classification of microarray data. This work presents gene encoder, an unsupervised two-stage feature selection technique for the cancer samples’ classification. The first stage aggregates three filter methods, namely principal component analysis, correlation, and spectral-based feature selection techniques. Next, the genetic algorithm is used, which evaluates the chromosome utilizing the autoencoder-based clustering. The resultant feature subset is used for the classification task. Three classifiers, namely support vector machine, k-nearest neighbors, and random forest, are used in this work to avoid the dependency on any one classifier. Six benchmark gene expression datasets are used for the performance evaluation, and a comparison is made with four state-of-the-art related algorithms. Three sets of experiments are carried out to evaluate the proposed method. These experiments are for the evaluation of the selected features based on sample-based clustering, adjusting optimal parameters, and for selecting better performing classifier. The comparison is based on accuracy, recall, false positive rate, precision, F-measure, and entropy. The obtained results suggest better performance of the current proposal.

Depicting More Information in Enriched Squarified Treemaps with Layered Glyphs

The Treemap is one of the most relevant information visualization (InfoVis) techniques to support the analysis of large hierarchical data structures or data clusters. Despite that, Treemap still presents some challenges for data representation, such as the few options for visual data mappings and the inability to represent zero and negative values. Additionally, visualizing high dimensional data requires many hierarchies, which can impair data visualization. Thus, this paper proposes to add layered glyphs to Treemap’s items to mitigate these issues. Layered glyphs are composed of N partially visible layers, and each layer maps one data dimension to a visual variable. Since the area of the upper layers is always smaller than the bottom ones, the layers can be stacked to compose a multidimensional glyph. To validate this proposal, we conducted a user study to compare three scenarios of visual data mappings for Treemaps: only Glyphs (G), Glyphs and Hierarchy (GH), and only Hierarchy (H). Thirty-six volunteers with a background in InfoVis techniques, organized into three groups of twelve (one group per scenario), performed 8 InfoVis tasks using only one of the proposed scenarios. The results point that scenario GH presented the best accuracy while having a task-solving time similar to scenario H, which suggests that representing more data in Treemaps with layered glyphs enriched the Treemap visualization capabilities without impairing the data readability.

On the Consistency of k-means++ algorithm

We prove in this paper that the expected value of the objective function of the $k$-means++ algorithm for samples converges to population expected value. As $k$-means++, for samples, provides with constant factor approximation for $k$-means objectives, such an approximation can be achieved for the population with increase of the sample size. This result is of potential practical relevance when one is considering using subsampling when clustering large data sets (large data bases).

Mean field models for large data–clustering problems

We consider mean-field models for data–clustering problems starting from a generalization of the bounded confidence model for opinion dynamics. The microscopic model includes information on the position as well as on additional features of the particles in order to develop specific clustering effects. The corresponding mean–field limit is derived and properties of the model are investigated analytically. In particular, the mean–field formulation allows the use of a random subsets algorithm for efficient computations of the clusters. Applications to shape detection and image segmentation on standard test images are presented and discussed.

Micro Clustering Methodology for Document Objects using Deep Learning Techniques

Large data clustering and classification is a very challenging task in data mining. Various machine learning and deep learning systems have been proposed by many researchers on a different dataset. Data volume, data size and structure of data may affect the time complexity of the system. This paper described a new document object classification approach using deep learning (DL) and proposed a recurrent neural network (RNN) for classification with a micro-clustering approach.TF-IDF and a density-based approach are used to store the best features. The plane work used supervised learning method and it extracts features set called as BK of the desired classes. once the training part completed then proceeds to figure out the particular test instances with the help of the planned classification algorithm. Recurrent Neural Network categorized the particular test object according to their weights. The system can able to work on heterogeneous data set and generate the micro-clusters according to classified results. The system also carried out experimental analysis with classical machine learning algorithms. The proposed algorithm shows higher accuracy than the existing density-based approach on different data sets.