Large Number Of Clusters Research Articles

Scalable Clustering: Large Scale Unsupervised Learning of Gaussian Mixture Models with Outliers

Clustering is a widely used technique with a long and rich history in a variety of areas. However, most existing algorithms do not scale well to large datasets, or are missing theoretical guarantees of convergence. This paper introduces a provably robust clustering algorithm based on loss minimization that performs well on Gaussian mixture models with outliers. It provides theoretical guarantees that the algorithm obtains high accuracy with high probability under certain assumptions. Moreover, it can also be used as an initialization strategy for k-means clustering. Experiments on real-world large-scale datasets demonstrate the effectiveness of the algorithm when clustering a large number of clusters, and a k-means algorithm initialized by the algorithm outperforms many of the classic clustering methods in both speed and accuracy, while scaling well to large datasets such as ImageNet.

Machine Learning‐Based Clustering of Oceanic Lagrangian Particles: Identification of the Main Pathways of the Labrador Current

AbstractModeled geospatial Lagrangian trajectories are widely used in Earth Science, including in oceanography, atmospheric science and marine biology. The typically large size of these data sets makes them arduous to analyze, and their underlying pathways challenging to identify. Here, we show that we can use a machine learning unsupervised k‐means++ clustering method combined with expert aggregation of clusters to identify the pathways of the Labrador Current from a large set of modeled Lagrangian trajectories. The presented method requires simple pre‐processing of the data, including a Cartesian correction on longitudes and a principal component analysis reduction. The clustering is performed in a kernelized space and uses a larger number of clusters than the number of expected pathways. To identify the main pathways, similar clusters are grouped into pathway categories by experts in the circulation of the region of interest. We find that the Labrador Current mainly follows a westward‐flowing and an eastward retroflecting pathway (20% and 50% of the flow, respectively) that compensate each other through time in a see‐saw behavior. These pathways experience a strong variability (representing through time 4%–42% and 24%–73% of the flow, respectively). Two thirds of the retroflection occurs at the tip of the Grand Banks, and one quarter at Flemish Cap. The westward pathway is mostly fed by the on‐shelf branch of the Labrador Current, and the eastward pathway by the shelf‐break branch. Among the pathways of secondary importance, we identify a previously unreported one that feeds the subtropics across the Gulf Stream.

Is Management and Organizational Studies divided into (micro-)tribes?

Many claims have been made in the past that Management and Organization Studies (MOS) is becoming increasingly fragmented, and that this fragmentation is causing it to drift into self-reference and irrelevance. Despite the weight of this claim, it has not yet been subjected to a systematic empirical test. This paper addresses this research gap using the tribalization approach and diachronic co-citation analyses. Based on 22,430 papers published in 14 MOS journals between 1980 and 2019, we calculate local and global centrality measures and the flow of cited articles between co-citation communities over time. In addition, we use a node-removal strategy to test whether only ritualized citations ensure MOS cohesion. Rather than tribalization, our results suggest a center–periphery structure. Furthermore, more peripheral papers are integrated into the central co-citation communities, but the lion's share of the flow of cited papers occurs over time to only a small number of large clusters. An increase of fragmentation and crowding-out of smaller clusters in MOS in seen in the polycentrically organized core 2014–2019.

Robust analysis of stepped wedge trials using composite likelihood models.

Stepped wedge trials (SWTs) are a type of cluster randomized trial that involve repeated measures on clusters and design-induced confounding between time and treatment. Although mixed models are commonly used to analyze SWTs, they are susceptible to misspecification particularly for cluster-longitudinal designs such as SWTs. Mixed model estimation leverages both "horizontal" or within-cluster information and "vertical" or between-cluster information. To use horizontal information in a mixed model, both the mean model and correlation structure must be correctly specified or accounted for, since time is confounded with treatment and measurements are likely correlated within clusters. Alternative non-parametric methods have been proposed that use only vertical information; these are more robust because between-cluster comparisons in a SWT preserve randomization, but these non-parametric methods are not very efficient. We propose a composite likelihood method that focuses on vertical information, but has the flexibility to recover efficiency by using additional horizontal information. We compare the properties and performance of various methods, using simulations based on COVID-19 data and a demonstration of application to the LIRE trial. We found that a vertical composite likelihood model that leverages baseline data is more robust than traditional methods, and more efficient than methods that use only vertical information. We hope that these results demonstrate the potential value of model-based vertical methods for SWTs with a large number of clusters, and that these new tools are useful to researchers who are concerned about misspecification of traditional models.

Constraining f(R) gravity using future galaxy cluster abundance and weak-lensing mass calibration datasets

We present forecasts for constraints on the Hu and Sawicki f(R) modified gravity model using realistic mock data representative of future cluster and weak lensing surveys. We create mock thermal Sunyaev-Zel’dovich effect selected cluster samples for SPT-3G and CMB-S4 and the corresponding weak gravitational lensing data from next-generation weak-lensing (ngWL) surveys like Euclid and Rubin. We employ a state-of-the-art Bayesian likelihood approach that includes all observational effects and systematic uncertainties to obtain constraints on the f(R) gravity parameter log10|fR0|. In this analysis we vary the cosmological parameters [Ωm,Ωνh2,h,As,ns,log10|fR0|], which allows us to account for possible degeneracies between cosmological parameters and f(R) modified gravity. The analysis accounts for f(R) gravity via its effect on the halo mass function which is enhanced on cluster mass scales compared to the expectations within general relativity (GR). Assuming a fiducial GR model, the upcoming cluster dataset SPT−3G×ngWL is expected to obtain an upper limit of log10|fR0|<−5.95 at 95% credibility, which significantly improves upon the current best bounds. The CMB−S4×ngWL dataset is expected to improve this even further to log10|fR0|<−6.23. Furthermore, f(R) gravity models with log10|fR0|≥−6, which have larger numbers of clusters, would be distinguishable from GR with both datasets. We also report degeneracies between log10|fR0| and Ωm as well as σ8 for log10|fR0|>−6 and log10|fR0|>−5 respectively. Our forecasts indicate that future cluster abundance studies of f(R) gravity will enable substantially improved constraints that are competitive with other cosmological probes. Published by the American Physical Society 2024

Understanding and including 'pink-collar' workers in employment-based travel demand models.

The segmentation of commuters into either blue or white-collar workers remains is still common in urban transport models. Internationally, models have started to use more elaborate segmentations, more reflective of changes in labour markets, such as increased female participation. Finding appropriate labour market segmentations for commute trip modelling remains a challenge. This paper harnesses a data-driven approach using unsupervised clustering-applied to 2017-20 South East Queensland Travel Survey (SEQTS) data. Commuter types are grouped by occupational, industry, and socio-demographic variables (i.e., gender, age, household size, household vehicle ownership and worker skill score). The results show that at a large number of clusters (i.e., k = 8) a highly distinct set of commuter types can be observed. But model run times tend to require a much smaller number of market segments. When only three clusters are formed (k = 3) a market segmentation emerges with one female-dominated type ('pink collar'), one male-dominated type ('blue collar') and one with both genders almost equally involved ('white collar'). There are nuances as to which workers are included in each segment, and differences in travel behaviours across the three types. 'Pink collar' workers are mostly comprised of female clerical and administrative workers, community and personal service workers and sales workers. They have the shortest median commutes for both private motorised and active transport modes. The approach and methods should assist transport planners to derive more accurate and robust market segmentations for use in large urban transport models, and, better predict the value of alternative transport projects and policies for all types of commuters.

Assessing Oral Health in U.S. Children and Its Impact on Disparities of Overall Health and Health Expenditures: An Application Using Two-Stage Clustering

We assess children’s oral and overall health using an innovative clustering methodology. Prior studies have documented the link between oral health and overall health, such as dental caries, which could lead to infections and pain, or periodontal disease, which is associated with other conditions, such as diabetes, heart disease, cardiovascular disease, and adverse pregnancy outcomes. Using the nationally representative Medical Expenditure Panel Study data, we link children’s dental care services and other control variables with various health outcomes using a two-stage clustering method. In the first stage, we cluster similar children using their social determinants of health, medical conditions, and dental care utilization as input variables. In the second stage, we identify groups of similar clusters using a metric called cosine similarity. External variables, rather than those used to create the clusters, are used to assess the effectiveness of the clusters. An outcome score is created based on children’s favorable overall and mental health statuses and absence of behavioral issues at home, school, and in general. We find that the first-stage clusters are separated into three main groups in the second stage, where one cluster from the first stage is very different from all of the other clusters, a second group has more favorable outcome scores, and the third group has lower outcome scores. Lower dental expenditures are associated with lower outcome scores. Clusters are useful for cost management purposes to identify subgroups for interventions to improve population health. Comparisons of access and outcomes of children between different insurance types can provide beneficial information for insurance design and public policy. Our approach can be applied to other areas of practice where there is an interest in grouping similar individuals and relating these groups to other quantities. The two-stage clustering method is a versatile method with a broader appeal to empirical research, especially when there are big data, a large number of clusters, and many input variables.

Correcting for Sampling Error in between-Cluster Effects: An Empirical Bayes Cluster-Mean Approach with Finite Population Corrections

With clustered data, such as where students are nested within schools or employees are nested within organizations, it is often of interest to estimate and compare associations among variables separately for each level. While researchers routinely estimate between-cluster effects using the sample cluster means of a predictor, previous research has shown that such practice leads to biased estimates of coefficients at the between level, and recent research has recommended the use of latent cluster means with the multilevel structural equation modeling framework. However, the latent cluster mean approach may not always be the best choice as it (a) relies on the assumption that the population cluster sizes are close to infinite, (b) requires a relatively large number of clusters, and (c) is currently only implemented in specialized software such as Mplus. In this paper, we show how using empirical Bayes estimates of the cluster means can also lead to consistent estimates of between-level coefficients, and illustrate how the empirical Bayes estimate can incorporate finite population corrections when information on population cluster sizes is available. Through a series of Monte Carlo simulation studies, we show that the empirical Bayes cluster-mean approach performs similarly to the latent cluster mean approach for estimating the between-cluster coefficients in most conditions when the infinite-population assumption holds, and applying the finite population correction provides reasonable point and interval estimates when the population is finite. The performance of EBM can be further improved with restricted maximum likelihood estimation and likelihood-based confidence intervals. We also provide an R function that implements the empirical Bayes cluster-mean approach, and illustrate it using data from the classic High School and Beyond Study.

Searching for Structure: Characterizing the Protein Conformational Landscape with Clustering-Based Algorithms.

The identification and characterization of the main conformations from a protein population are a challenging and inherently high-dimensional problem. Here, we evaluate the performance of the Secondary sTructural Ensembles with machine LeArning (StELa) double-clustering method, which clusters protein structures based on the relationship between the φ and ψ dihedral angles in a protein backbone and the secondary structure of the protein, thus focusing on the local properties of protein structures. The classification of states as vectors composed of the clusters' indices arising naturally from the Ramachandran plot is followed by the hierarchical clustering of the vectors to allow for the identification of the main features of the corresponding free energy landscape (FEL). We compare the performance of StELa with the established root-mean-squared-deviation (RMSD)-based clustering algorithm, which focuses on global properties of protein structures and with Combinatorial Averaged Transient Structure (CATS), the combinatorial averaged transient structure clustering method based on distributions of the φ and ψ dihedral angle coordinates. Using ensembles of conformations from molecular dynamics simulations of intrinsically disordered proteins (IDPs) of various lengths (tau protein fragments) or short fragments from a globular protein, we show that StELa is the clustering method that identifies many of the minima and relevant energy states around the minima from the corresponding FELs. In contrast, the RMSD-based algorithm yields a large number of clusters that usually cover most of the FEL, thus being unable to distinguish between states, while CATS does not sample well the FELs for long IDPs and fragments from globular proteins.

MAIT Cell Frequencies within PBSC Grafts Are Associated with Donor CMV Serostatus and Age: An Initial Analysis from the DKMS and NMDP Graft Composition Study

Introduction We have established an international collaboration between the DKMS (Germany) and the National Marrow Donor Program (NMDP; USA) to characterize more than 2000 donor peripheral blood stem cell (PBSC) grafts. We aim to correlate graft immunophenotype with patient outcomes and perform additional analyses on biobanked aliquots of selected samples to deepen our understanding of the relationships between features of the donor graft and key patient outcomes, such as graft-versus-host disease, relapse, and overall survival. Here, we present an interim analysis, focused on the CD8 T cell compartment, describing relationships between donor characteristics and graft composition. We have developed a novel computational clustering method to integrate cytometry data from both the US and German analytical laboratories. Methods 300 donor PBSC graft samples were collected, processed, freshly stained (34-color panel for immunophenotyping of lymphocytes and hematopoietic stem cells), and analyzed using an Aurora Cytek instrument at the NMDP contract laboratory (Roswell Park, Buffalo, NY). 53 additional samples were processed in the DKMS laboratory in Dresden, Germany (equivalent antibody panel and instrument). We have developed a novel analytical pipeline for integrating these data. First, we group samples by facility, and by month run for computational convenience, to form “batches” for initial clustering. We perform Leiden clustering, and subclustering, to generate a large number of clusters per batch. To integrate data from all batches, we then cluster all subcluster centroids from all batch-specific clusterings. This gives a global clustering for each stream ( e.g. the CD8+ T cells). Cluster frequencies per sample provide features for quantifying links between immunophenotype and donor metadata ( e.g., CMV serostatus, age, and sex). Our approach offers an important complement to traditional gating and allows full exploration of the data in an unbiased fashion. Results 353 donor products were included in the analysis. The median donor age was 28 years (range 18 - 60), 59% were female and 65% were CMV positive. Initial metaclustering demonstrated that the 53 DKMS samples and the 300 NMDP samples are each well represented in all clusters, and that site-driven batch effects, while present, are mild. We initially focused on the CD8 T cell compartment (UMAP in Fig 1A) and discovered several phenotypic differences when we compared CMV seronegative and seropositive donors. CD8 T cell clusters 10 and 11 (both TCRαβ+ CD45RA+ CCR7+ HLA-DR+ CD57+; cluster 10 CCR9+; cluster 11 CCR9-) were present at higher frequency in CMV+ donors (n = 116) compared with CMV- donors (n = 220; p <0.0001; Fig 1B). Conversely, clusters 9 and 12, enriched for mucosal-associated invariant T (MAIT) cells, were significantly reduced in CMV+ donors (p <0.001). Of note, CMV serostatus was not significantly associated with age (median age in the CMV+ group was 29.4 years; vs 28.4 years in CMV- donors). We also observed an association between cluster 8 (TCRαβ+ CD45RA+ CCR7+ HLA-DR+ CD38+; consistent with an activated phenotype) and age (p = 0.0009; Fig1B). There was a trend toward decreased MAIT frequencies with age (Cluster 9; p = 0.058). When we analyzed the MAIT cell frequency as defined by traditional flow cytometry gating (CD3+ CD8+ CD161+ Vα7.2+), we observed a statistically significant relationship between younger donor age and increased MAIT cell frequency (p = 0.0002), consistent with the trend seen in cluster 9. Sex at birth was not linked with immunophenotype in our analyses thus far. Conclusions In this initial analysis of our multi-center observational study of PBSC graft products, we demonstrate the development and application of new computational tools for large-cohort flow cytometric data analysis. CMV-positive serostatus and younger donor age are both associated with an increased frequency of MAIT cells in the graft. We also found differences in CD8 T cell activation status in association with age and CMV. In the future, we will assess the relationships between these populations and patient outcomes, as well as explore cluster frequencies in association with key patient outcomes including GVHD, relapse, and overall survival. Sample collection and analysis are ongoing at both US and German sites as an ongoing collaborative effort between the NMDP and the DKMS.

The effects of physical treatment on physicochemical and biological properties of water and aqueous solutions

Objectives. Changes to the properties of water caused by factors such as pressure or temperature, can only be explained by its structural changes. Scientists study changes to the properties of water due to various physical stimuli only without the addition of any substances. Examples of stimuli are acoustic exposure, thermal exposure, pressure variation, shaking, intensive vibration treatment followed by dilutions, vortexing, bubble generation, inter alia.The aim of the present review article is to summarize the available data on how the above processes affect the physicochemical and biological properties of water and aqueous solutions.Results. It has been shown that heating makes water less compressible and decreases air solubility in water, while cooling enhances its viscosity. Acoustic exposure makes the structure of water become coarse-grained, followed by an increase the number of large clusters, pH and temperature inside a cavitation bubble. High pressure enhances the viscosity, self-diffusion, and compressibility of water. For bubble processed water, there are changes in the spin-spin and spin-lattice relaxation times. Reactive oxygen species are formed, as well as increased solubility of gases in liquids and reduced friction. Vortex process technology causes an increase of electrical conductivity of water and reduced viscosity. Intensive vibration treatment and dilution processes result in changes in electrical conductivity of water, dissolved gas concentration, ultrasonic wave velocity, рН, surface tension, dielectric constant, and spectral response. There is also data to support the biological effects of different types of physical treatment of solutions.Conclusions. This review shows that physical treatment of water can induce changes both in physicochemical and biological properties of water and aqueous solutions.

Energy Efficient CH Selection Scheme Based on ABC and Q-Learning Approaches for IoUT Applications

Nowadays, the Internet of Underwater Things (IoUT) provides many marine 5G applications. However, it has some issues with energy efficiency and network lifetime. The network clustering approach is efficient for optimizing energy consumption, especially for underwater acoustic communications. Recently, many algorithms have been developed related to clustering-based underwater communications for energy efficiency. However, these algorithms have drawbacks when considered for heterogeneous IoUT applications. Clustering efficiency in heterogeneous IoUT is influenced by the uniform distribution of cluster heads (CHs). As a result, conventional schemes are inefficient when CHs are arranged in large and dense nodes since they are unable to optimize the right number of CHs. Consequently, the clustering approach cannot improve the IoUT network, and many underwater nodes will rapidly consume their energies and be exhausted because of the large number of clusters. In this paper, we developed an efficient clustering scheme to effectively select the best CHs based on artificial bee colony (ABC) and Q-learning optimization approaches. The proposed scheme enables an effective selection of the CHs based on four factors, the residual energy level, the depth and the distance from the base station, and the signal quality. We first evaluate the most suitable swarm algorithms and their impact on improving the CH selection mechanism. The evaluated algorithms are generic algorithm (GA), particle swarm optimization (PSO), ant colony optimization (ACO), and ABC. Then, the ABC algorithm process is improved by using the Q-learning approach to improve the process of ABC and its fitness function to optimize the CH selection. We observed from the simulation performance result that an improved ABC-QL scheme enables efficient selection of the best CHs to increase the network lifetime and reduce average energy consumption by 40% compared to the conventional ABC.

Self-supervised deep clustering of single-cell RNA-seq data to hierarchically detect rare cell populations.

Single-cell RNA sequencing (scRNA-seq) is a widely used technique for characterizing individual cells and studying gene expression at the single-cell level. Clustering plays a vital role in grouping similar cells together for various downstream analyses. However, the high sparsity and dimensionality of large scRNA-seq data pose challenges to clustering performance. Although several deep learning-based clustering algorithms have been proposed, most existing clustering methods have limitations in capturing the precise distribution types of the data or fully utilizing the relationships between cells, leaving a considerable scope for improving the clustering performance, particularly in detecting rare cell populations from large scRNA-seq data. We introduce DeepScena, a novel single-cell hierarchical clustering tool that fully incorporates nonlinear dimension reduction, negative binomial-based convolutional autoencoder for data fitting, and a self-supervision model for cell similarity enhancement. In comprehensive evaluation using multiple large-scale scRNA-seq datasets, DeepScena consistently outperformed seven popular clustering tools in terms of accuracy. Notably, DeepScena exhibits high proficiency in identifying rare cell populations within large datasets that contain large numbers of clusters. When applied to scRNA-seq data of multiple myeloma cells, DeepScena successfully identified not only previously labeled large cell types but also subpopulations in CD14 monocytes, T cells and natural killer cells, respectively.

Empirical Configuration Study of a Common Document Clustering Pipeline

Document clustering is frequently used in applications of natural language processing, e.g. to classify news articles or creating topic models. In this paper, we study document clustering with the common clustering pipeline that includes vectorization with BERT or Doc2Vec, dimension reduction with PCA or UMAP, and clustering with K-Means or HDBSCAN. We discuss the inter- actions of the different components in the pipeline, parameter settings, and how to determine an appropriate number of dimensions. The results suggest that BERT embeddings combined with UMAP dimension reduction to no less than 15 dimensions provides a good basis for clustering, regardless of the specific clustering algorithm used. Moreover, while UMAP performed better than PCA in our experiments, tuning the UMAP settings showed little impact on the overall performance. Hence, we recommend configuring UMAP so as to optimize its time efficiency. According to our topic model evaluation, the combination of BERT and UMAP, also used in BERTopic, performs best. A topic model based on this pipeline typically benefits from a large number of clusters.

Variational quantum and quantum-inspired clustering

Here we present a quantum algorithm for clustering data based on a variational quantum circuit. The algorithm allows to classify data into many clusters, and can easily be implemented in few-qubit Noisy Intermediate-Scale Quantum devices. The idea of the algorithm relies on reducing the clustering problem to an optimization, and then solving it via a Variational Quantum Eigensolver combined with non-orthogonal qubit states. In practice, the method uses maximally-orthogonal states of the target Hilbert space instead of the usual computational basis, allowing for a large number of clusters to be considered even with few qubits. We benchmark the algorithm with numerical simulations using real datasets, showing excellent performance even with one single qubit. Moreover, a tensor network simulation of the algorithm implements, by construction, a quantum-inspired clustering algorithm that can run on current classical hardware.

Arabinoxylan in Water through SANS: Single-Chain Conformation, Chain Overlap, and Clustering.

Using small-angle neutron scattering (SANS), we examine the structure and conformational behavior of wheat arabinoxylan (AX) prepared at various concentrations in a sodium phosphate aqueous buffer. As for another major hemicellulose, xyloglucan, we observe a small number of large clusters surrounded by AX chains that behave exactly as a polymer in good solvent with a Flory exponent ν = 0.588. The fit of the data at high q-values to a standard worm-like chain model gives the persistence length lp = 45 Å and cross section of the chains 2Rc = 11-12 Å. In addition, using a dedicated modeling approach, we extract from the SANS data at the intermediate q-range the correlation length ξ of the solutions in the semidilute regime. The decay of ξ with concentration follows a scaling law that further confirms the self-avoiding statistical behavior of the AX chains. This first comprehensive study about the properties of water-soluble AX at different length scales may help in the development of products and processes involving AX as a substitute for fossil carbon molecules.

Genetic Diversity for Yield and Yield Contributing Characters in Sesame (Sesamum indicum L.)

In the present investigation, At RARS, Jagtial, 68 genotypes, including three checks were evaluated for genetic diversity for selection of diverse parents. Divergence studies through D2 statistic indicated the presence of substantial diversity by forming large number of clusters with wide range of inter-cluster distance.The 68 genotypes were distributed into eight clusters based on the D2 values. Among the eight clusters, cluster I was the largest comprising of 30 genotypes followed by cluster II with 22 genotypes and Cluster III and V with six genotypes in each cluster remaining clusters IV, VI, VII, VIII were solitary. The data on character means for eight clusters indicated that, cluster III was having highest mean value for number of capsules per plant, capsule length, capsule width, number of seeds per capsule, seed yield per plant and 1000 seed weight. Cluster VIII for days to 50% flowering, plant height, days to maturity and number of branches per plant. The genotypes JCS 2611, JCS 2454, and JCS 3599 have a high cluster distance and might be employed directly used for adaptation or may be used as parents in future hybridization programme.

A double-Cox model for non-proportional hazards survival analysis with frailty.

The Cox regression, a semi-parametric method of survival analysis, is extremely popular in biomedical applications. The proportional hazards assumption is a key requirement in the Cox model. To accommodate non-proportional hazards, we propose to parameterize the shape parameter of the baseline hazard function using the additional, separate Cox-regression term which depends on the vector of the covariates. This parametrization retains the general form of the hazard function over the strata and is similar to one in Devarajan and Ebrahimi (Comput Stat Data Anal. 2011;55:667-676) in the case of the Weibull distribution, but differs for other hazard functions. We call this model the double-Cox model. We formally introduce the double-Cox model with shared frailty and investigate, by simulation, the estimation bias and the coverage of the proposed point and interval estimation methods for the Gompertz and the Weibull baseline hazards. For real-life applications with low frailty variance and a large number of clusters, the marginal likelihood estimation is almost unbiased and the profile likelihood-based confidence intervals provide good coverage for all model parameters. We also compare the results from the over-parametrized double-Cox model to those from the standard Cox model with frailty in the case of the scale-only proportional hazards. The model is illustrated on an example of the survival after a diagnosis of type 2 diabetes mellitus. The R programs for fitting the double-Cox model are available on Github.

Minimal distance index — A new clustering performance metrics

We define a new index for measuring clustering performance called the Minimal Distance Index. The index is based on representing clusters by characteristic objects containing the majority of cluster points. It performs well for both spherical and ellipsoidal clusters. This method can recognize all acceptable partitions with well-separated clusters. Among such partitions, our minimal distance index may identify the most appropriate one. The proposed index is compared with other most frequently used indexes in numerous examples with spherical and ellipsoidal clusters. It turned out that our proposed minimal distance index always recognizes the most appropriate partition, whereas the same cannot be said for other indexes found in the literature.Furthermore, among all acceptable partitions, the one with the largest number of clusters, not necessarily the most appropriate ones, has a special significance in image analysis. Namely, following Mahalanobis image segmentation, our index recognizes partitions that might not be the most appropriate ones but are the ones using colors that significantly differ from each other. The minimal distance index recognizes partitions with dominant colors, thus making it possible to select specific details in the image. We apply this approach to some real-world applications such as the plant rows detection problem, painting analysis, and iris detection. This may also be useful for medical image analysis.

POFCM: A Parallel Fuzzy Clustering Algorithm for Large Datasets

Clustering algorithms have proven to be a useful tool to extract knowledge and support decision making by processing large volumes of data. Hard and fuzzy clustering algorithms have been used successfully to identify patterns and trends in many areas, such as finance, healthcare, and marketing. However, these algorithms significantly increase their solution time as the size of the datasets to be solved increase, making their use unfeasible. In this sense, the parallel processing of algorithms has proven to be an efficient alternative to reduce their solution time. It has been established that the parallel implementation of algorithms requires its redesign to optimise the hardware resources of the platform that will be used. In this article, we propose a new parallel implementation of the Hybrid OK-Means Fuzzy C-Means (HOFCM) algorithm, which is an efficient variant of Fuzzy C-Means, in OpenMP. An advantage of using OpenMP is its scalability. The efficiency of the implementation is compared against the HOFCM algorithm. The experimental results of processing large real and synthetic datasets show that our implementation tends to more efficiently solve instances with a large number of clusters and dimensions. Additionally, the implementation shows excellent results concerning speedup and parallel efficiency metrics. Our main contribution is a Fuzzy clustering algorithm for large datasets that is scalable and not limited to a specific domain.