Minimum Sum Of Squares Clustering Research Articles

Distributed random swap: An efficient algorithm for minimum sum-of-squares clustering

The clustering model known as Minimum Sum-of-Squares Clustering (MSSC) is widely used, with the popular k-means algorithm serving as its local minimizer. It is well-known that solutions of k-means can result in substantial deviations from the true global optimum of MSSC. While numerous heuristics and metaheuristics have been proposed to overcome this limitation, none have gained dominant acceptance in academic literature. This is likely related to challenges such as intricate implementations and a multitude of tunable parameters.In this paper, we dispute the belief that simplifying an algorithm for MSSC inherently means sacrificing quality. We present the Distributed Random Swap (DRS-means) algorithm, which is designed to enhance clustering performance for the MSSC problem. This algorithm can be interpreted as an iterative method that refines the solution generated by the k-means algorithm during each iteration. The enhancement is achieved by selecting points based on specific probability distributions. These distributions are carefully designed to improve and speed up the exploration phase. The proposed algorithm is straightforward to implement. DRS-means offers a user-friendly solution with state-of-the-art results, making it suitable for a wide range of research fields.

Second‐order cone programming models for the unitary weighted Weber problem and for the minimum sum of the squares clustering problem

AbstractIn this work, new mixed integer nonlinear optimization models are proposed for two clustering problems: the unitary weighted Weber problem and the minimum sum of squares clustering. The proposed formulations are convex quadratic models with linear and second‐order cone constraints that can be efficiently solved by interior point algorithms. Their continuous relaxation is convex and differentiable. The numerical experiments show the proposed models are more efficient than some classical models for these problems known in the literature.

Global optimization for cardinality-constrained minimum sum-of-squares clustering via semidefinite programming

AbstractThe minimum sum-of-squares clustering (MSSC), or k-means type clustering, has been recently extended to exploit prior knowledge on the cardinality of each cluster. Such knowledge is used to increase performance as well as solution quality. In this paper, we propose a global optimization approach based on the branch-and-cut technique to solve the cardinality-constrained MSSC. For the lower bound routine, we use the semidefinite programming (SDP) relaxation recently proposed by Rujeerapaiboon et al. (SIAM J Optim 29(2):1211–1239, 2019). However, this relaxation can be used in a branch-and-cut method only for small-size instances. Therefore, we derive a new SDP relaxation that scales better with the instance size and the number of clusters. In both cases, we strengthen the bound by adding polyhedral cuts. Benefiting from a tailored branching strategy which enforces pairwise constraints, we reduce the complexity of the problems arising in the children nodes. For the upper bound, instead, we present a local search procedure that exploits the solution of the SDP relaxation solved at each node. Computational results show that the proposed algorithm globally solves, for the first time, real-world instances of size 10 times larger than those solved by state-of-the-art exact methods.

How to Use K-means for Big Data Clustering?

K-means plays a vital role in data mining and is the simplest and most widely used algorithm under the Euclidean Minimum Sum-of-Squares Clustering (MSSC) model. However, its performance drastically drops when applied to vast amounts of data. Therefore, it is crucial to improve K-means by scaling it to big data using as few of the following computational resources as possible: data, time, and algorithmic ingredients. We propose a new parallel scheme of using K-means and K-means++ algorithms for big data clustering that satisfies the properties of a “true big data” algorithm and outperforms the classical and recent state-of-the-art MSSC approaches in terms of solution quality and runtime. The new approach naturally implements global search by decomposing the MSSC problem without using additional metaheuristics. This work shows that data decomposition is the basic approach to solve the big data clustering problem. The empirical success of the new algorithm allowed us to challenge the common belief that more data is required to obtain a good clustering solution. Moreover, the present work questions the established trend that more sophisticated hybrid approaches and algorithms are required to obtain a better clustering solution.

An exact algorithm for semi-supervised minimum sum-of-squares clustering

The minimum sum-of-squares clustering (MSSC), or k-means type clustering, is traditionally considered an unsupervised learning task. In recent years, the use of background knowledge to improve the cluster quality and promote interpretability of the clustering process has become a hot research topic at the intersection of mathematical optimization and machine learning research. The problem of taking advantage of background information in data clustering is called semi-supervised or constrained clustering. In this paper, we present branch-and-cut algorithm for semi-supervised MSSC, where background knowledge is incorporated as pairwise must-link and cannot-link constraints. For the lower bound procedure, we solve the semidefinite programming relaxation of the MSSC discrete optimization model, and we use a cutting-plane procedure for strengthening the bound. For the upper bound, instead, by using integer programming tools, we use an adaptation of the k-means algorithm to the constrained case. For the first time, the proposed global optimization algorithm efficiently manages to solve real-world instances up to 800 data points with different combinations of must-link and cannot-link constraints and with a generic number of features. This problem size is about four times larger than the one of the instances solved by state-of-the-art exact algorithms.

Responsive threshold search based memetic algorithm for balanced minimum sum-of-squares clustering

Clustering is a common task in data mining for constructing well-separated groups (clusters) from a large set of data points. The balanced minimum sum-of-squares clustering problem is a variant of the classic minimum sum-of-squares clustering (MSSC) problem and arises from broad real-life applications where the cardinalities of any two clusters differ by at most one. This study presents the first memetic algorithm for solving the balanced MSSC problem. The proposed algorithm combines a backbone-based crossover operator for generating offspring solutions and a responsive threshold search that alternates between a threshold-based exploration procedure and a descent-based improvement procedure for improving new offspring solutions. Numerical results on 16 real-life datasets show that the proposed algorithm competes very favorably with several state-of-the-art methods from the literature. Key components of the proposed algorithm are investigated to understand their effects on the performance of the algorithm.

Memetic differential evolution methods for clustering problems

The Euclidean Minimum Sum-of-Squares Clustering(MSSC) is one of the most important models for the clustering problem. Due to its NP-hardness, the problem continues to receive much attention in the scientific literature and several heuristic procedures have been proposed. Recent research has been devoted to the improvement of the classical K-MEANS algorithm, either by suitably selecting its starting configuration or by using it as a local search method within a global optimization algorithm. This paper follows this last approach by proposing a new implementation of a Memetic Differential Evolution (MDE) algorithm specifically designed for the MSSC problem and based on the repeated execution of K-MEANS from selected configurations. In this paper we describe how to adapt MDE to the clustering problem and we show, through a vast set of numerical experiments, that the proposed method has very good quality, measured in terms of the minimization of the objective function, as well as a very good efficiency, measured in the number of calls to the local optimization routine, with respect to state of the art methods.

HG-means: A scalable hybrid genetic algorithm for minimum sum-of-squares clustering

Minimum sum-of-squares clustering (MSSC) is a widely used clustering model, of which the popular K-means algorithm constitutes a local minimizer. It is well known that the solutions of K-means can be arbitrarily distant from the true MSSC global optimum, and dozens of alternative heuristics have been proposed for this problem. However, no other algorithm has been predominantly adopted in the literature. This may be related to differences of computational effort, or to the assumption that a near-optimal solution of the MSSC has only a marginal impact on clustering validity.In this article, we dispute this belief. We introduce an efficient population-based metaheuristic that uses K-means as a local search in combination with problem-tailored crossover, mutation, and diversification operators. This algorithm can be interpreted as a multi-start K-means, in which the initial center positions are carefully sampled based on the search history. The approach is scalable and accurate, outperforming all recent state-of-the-art algorithms for MSSC in terms of solution quality, measured by the depth of local minima. This enhanced accuracy leads to clusters which are significantly closer to the ground truth than those of other algorithms, for overlapping Gaussian-mixture datasets with a large number of features. Therefore, improved global optimization methods appear to be essential to better exploit the MSSC model in high dimension.

Algorithms for a minimum sum of squares clustering

Cluster analysis aims at solving the following very general problem: given a set of entities, find subsets (also called clusters) which are homogeneous and well separated. Homogeneity means that similar entities should belong to the same cluster. Separation means that dissimilar entities should belong to different clusters. This concept can be expressed mathematically in many ways.

Grouping cities based of their socio-economic indicators

In this paper a new approach in analyzing city behaviour and its dynamic developments is proposed. Usual attributes are divided into two larger groups: social and economic. This idea makes possible analyzing relative performances of cities from different angles, such as: two-dimensional clustering; city ranking by simple calculating their distance from origin; minimum sum of squares clustering, etc. Such an analysis is helpful to show good and week points in developing city strategy in long term period. Grouping of 77 capital cities of regions in Russia is considered. The averages of both groups, for each city are calculated, covering 12 years. Minimum sum-of-squares criterion is used for clustering and solved by variable neighborhood search. It appeared that there are a few outlier cities, two of them being non dominated or efficient. Detailed comparative analysis of the results before and after crisis (economic sanctions) are also provided.

Coordination of Cluster Ensembles via Exact Methods

We present a novel optimization-based method for the combination of cluster ensembles for the class of problems with intracluster criteria, such as Minimum-Sum-of-Squares-Clustering (MSSC). We propose a simple and efficient algorithm-called EXAMCE-for this class of problems that is inspired from a Set-Partitioning formulation of the original clustering problem. We prove some theoretical properties of the solutions produced by our algorithm, and in particular that, under general assumptions, though the algorithm recombines solution fragments so as to find the solution of a Set-Covering relaxation of the original formulation, it is guaranteed to find better solutions than the ones in the ensemble. For the MSSC problem in particular, a prototype implementation of our algorithm found a new better solution than the previously best known for 21 of the test instances of the 40-instance TSPLIB benchmark data sets used in [1], [2], and [3], and found a worse-quality solution than the best known only five times. For other published benchmark data sets where the optimal MSSC solution is known, we match them. The algorithm is particularly effective when the number of clusters is large, in which case it is able to escape the local minima found by K-means type algorithms by recombining the solutions in a Set-Covering context. We also establish the stability of the algorithm with extensive computational experiments, by showing that multiple runs of EXAMCE for the same clustering problem instance produce high-quality solutions whose Adjusted Rand Index is consistently above 0.95. Finally, in experiments utilizing external criteria to compute the validity of clustering, EXAMCE is capable of producing high-quality results that are comparable in quality to those of the best known clustering algorithms.

Variable neighborhood search for harmonic means clustering

Harmonic means clustering is a variant of minimum sum of squares clustering (which is sometimes called K-means clustering), designed to alleviate the dependance of the results on the choice of the initial solution. In the harmonic means clustering problem, the sum of harmonic averages of the distances from the data points to all cluster centroids is minimized. In this paper, we propose a variable neighborhood search heuristic for solving it. This heuristic has been tested on numerous datasets from the literature. It appears that our results compare favorably with recent ones from tabu search and simulated annealing heuristics.

Solving the minimum sum-of-squares clustering problem by hyperbolic smoothing and partition into boundary and gravitational regions

This article considers the minimum sum-of-squares clustering (MSSC) problem. The mathematical modeling of this problem leads to a min-sum-min formulation which, in addition to its intrinsic bi-level nature, has the significant characteristic of being strongly nondifferentiable. To overcome these difficulties, the proposed resolution method, called hyperbolic smoothing, adopts a smoothing strategy using a special C ∞ differentiable class function. The final solution is obtained by solving a sequence of low dimension differentiable unconstrained optimization subproblems which gradually approach the original problem. This paper introduces the method of partition of the set of observations into two nonoverlapping groups: “data in frontier” and “data in gravitational regions”. The resulting combination of the two methodologies for the MSSC problem has interesting properties, which drastically simplify the computational tasks.

An improved column generation algorithm for minimum sum-of-squares clustering

Given a set of entities associated with points in Euclidean space, minimum sum-of-squares clustering (MSSC) consists in partitioning this set into clusters such that the sum of squared distances from each point to the centroid of its cluster is minimized. A column generation algorithm for MSSC was given by du Merle et al. in SIAM Journal Scientific Computing 21:1485–1505. The bottleneck of that algorithm is the resolution of the auxiliary problem of finding a column with negative reduced cost. We propose a new way to solve this auxiliary problem based on geometric arguments. This greatly improves the efficiency of the whole algorithm and leads to exact solution of instances with over 2,300 entities, i.e., more than 10 times as much as previously done.

A branch-and-cut SDP-based algorithm for minimum sum-of-squares clustering

Minimum sum-of-squares clustering (MSSC) consists in partitioning a given set of n points into k clusters in order to minimize the sum of squared distances from the points to the centroid of their cluster. Recently, Peng & Xia (2005) established the equivalence between 0-1 semidefinite programming (SDP) and MSSC. In this paper, we propose a branch-and-cut algorithm for the underlying 0-1 SDP model. The algorithm obtains exact solutions for fairly large data sets with computing times comparable with those of the best exact method found in the literature.

CLUSTERING USING AN IMPROVED HYBRID GENETIC ALGORITHM

In this article, a new genetic clustering algorithm called the Improved Hybrid Genetic Clustering Algorithm (IHGCA) is proposed to deal with the clustering problem under the criterion of minimum sum of squares clustering. In IHGCA, the improvement operation including five local iteration methods is developed to tune the individual and accelerate the convergence speed of the clustering algorithm, and the partition-absorption mutation operation is designed to reassign objects among different clusters. By experimental simulations, its superiority over some known genetic clustering methods is demonstrated.

Analysis of gene expression profiles: an application of memetic algorithms to the minimum sum-of-squares clustering problem

Microarrays have become a key technology in experimental molecular biology since they allow monitoring of gene expression for more than 10,000 genes in parallel producing huge amounts of data. In the exploration of transcriptional regulatory networks, an important task is to cluster gene expression data to identify groups of genes with similar patterns and hence similar function. In this paper, memetic algorithms (MAs)—evolutionary algorithms incorporating local search—are proposed for minimum sum-of-squares clustering (MSSC). In a fitness landscape analysis, it is shown that the MSSC problem has correlation structure exploitable by MAs. The proposed MAs are shown to be superior to multi-start k-means as well as five other clustering algorithms from the bioinformatics literature including hierarchical algorithms and self-organizing maps. Although the fitness values of the different clustering solutions lie close together, it is shown that the solutions differ significantly from each other in terms of cluster memberships which is extremely important for the biological interpretation of the clustering results.

Minimum Sum of Squares Clustering in a Low Dimensional Space

Clustering with a criterion which minimizes the sum of squared distances to cluster centroids is usually done in a heuristic way. An exact polynomial algorithm, with a complexity in O(N p+1 logN), is proposed for minimum sum of squares hierarchical divisive clustering of points in a p-dimensional space with small p. Empirical complexity is one order of magnitude lower. Data sets with N = 20000 for p = 2, N = 1000 for p = 3, and N = 200 for p = 4 are clustered in a reasonable computing time.