Cannot-link Constraints Research Articles

Code and Data Repository for An Algorithm for Clustering with Confidence-Based Must-Link and Cannot-Link Constraints

The PCCC algorithm is a clustering method that incorporates both hard and soft must-link and cannot-link constraints.

An Algorithm for Clustering with Confidence-Based Must-Link and Cannot-Link Constraints

We study here the semisupervised k-clustering problem where information is available on whether pairs of objects are in the same or different clusters. This information is available either with certainty or with a limited level of confidence. We introduce the pair-wise confidence constraints clustering (PCCC) algorithm, which iteratively assigns objects to clusters while accounting for the information provided on the pairs of objects. Our algorithm uses integer programming for the assignment of objects, which allows us to include relationships as hard constraints that are guaranteed to be satisfied or as soft constraints that can be violated subject to a penalty. This flexibility distinguishes our algorithm from the state of the art, in which all pair-wise constraints are considered hard or all are considered soft. We developed an enhanced multistart approach and a model-size reduction technique for the integer program that contribute to the effectiveness and efficiency of the algorithm. Unlike existing algorithms, our algorithm scales to large-scale instances with up to 60,000 objects, 100 clusters, and millions of cannot-link constraints (which are the most challenging constraints to incorporate). We compare the PCCC algorithm with state-of-the-art approaches in an extensive computational study. Even though the PCCC algorithm is more general than the state-of-the-art approaches in its applicability, it outperforms the state-of-the-art approaches on instances with all hard or all soft constraints in terms of both run time and various metrics of solution quality. The code of the PCCC algorithm is publicly available on GitHub. History: Accepted by Ram Ramesh, Area Editor for Data Science and Machine Learning. Funding: The research of D. S. Hochbaum was supported by the AI Institute NSF Award [Grant 2112533]. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2023.0419 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2023.0419 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .

Effective semi-supervised graph clustering with pairwise constraints

Semi-supervised graph clustering with constraints has received considerable attention in the last decade. They use pre-given constraints to guide the clustering process and improve the performance. Nonetheless, most of related research works still face two main issues: (1) cannot-link problem: how to ensure that instances under the cannot-link constraint are located in different clusters as much as possible. (2) The high computational cost caused by eigendecomposition. To tackle these problems, this paper proposes a novel method for directly solving the clustering indicator matrix without high complexity computational steps. Then, we propose an effective and simple algorithm guided by the pre-given constraints, which is able to simultaneously optimize the associated multiple pairs of constraints, allowing the associated constraints to achieve the related co-optimal solution. Extensive experiments are conducted to verify that our approach significantly reduces the violation rate of the prior constraints and the clustering performance of the proposed method outperforms seven other state-of-the-art semi-supervised clustering methods on 16 real benchmark datasets.

Ship trajectory segmentation and semisupervised clustering via geospatial background knowledge

Cluster analysis is a common application of maritime traffic pattern recognition. However, most current methods are unsupervised clustering algorithms that rely solely on the features of the trajectories themselves for clustering, making them susceptible to uneven density distributions. To address this issue, we propose a semisupervised clustering algorithm that utilizes geospatial data to segment trajectories and generate clustering constraints. First, shoreline and anchorage information are employed for trajectory segmentation to enhance the consistency of local trajectory features. Second, cannot-link constraints derived from shoreline data are used to modify the distance between two trajectories, thereby preventing the clustering of trajectories separated by nonnavigable waters into a single cluster. Last, experimental verification is conducted using automatic identification system data, and the results are compared with those of existing algorithms. The results demonstrate more concentrated trajectory segmentation points, easier selection of clustering parameters, and significantly improved evaluation outcomes based on external clustering validity indices.

Efficient Constrained K-center Clustering with Background Knowledge

Center-based clustering has attracted significant research interest from both theory and practice. In many practical applications, input data often contain background knowledge that can be used to improve clustering results. In this work, we build on widely adopted k-center clustering and model its input background knowledge as must-link (ML) and cannot-link (CL) constraint sets. However, most clustering problems including k-center are inherently NP-hard, while the more complex constrained variants are known to suffer severer approximation and computation barriers that significantly limit their applicability. By employing a suite of techniques including reverse dominating sets, linear programming (LP) integral polyhedron, and LP duality, we arrive at the first efficient approximation algorithm for constrained k-center with the best possible ratio of 2. We also construct competitive baseline algorithms and empirically evaluate our approximation algorithm against them on a variety of real datasets. The results validate our theoretical findings and demonstrate the great advantages of our algorithm in terms of clustering cost, clustering quality, and running time.

Side-constrained graph fusion for semi-supervised multi-view clustering

In these years, semi-supervised learning arouses ongoing attentions due to its appealing avail and economic expenditure of guiding model training. Semi-supervised multi-view clustering takes advantages of heterogeneous features and a small number of side constraints (i.e., must-link constraints and cannot-link constraints) to partition data points. However, most of existing approaches are very limited in absorbing available side constraints since it is difficult to optimize cannot-link constraints in a provable way, and thus greatly waste the value of prior information. To address it, we innovatively pose a side-constrained multi-view graph clustering method, where the pairwise constraints are flexibly incorporated into the multiple graph fusion framework. The technique definitely formulates the pairwise constraints in the graph clustering model by designing a semi-supervised graph regularization term. In this way, the structured optimal graph that satisfies multiple perspectives and specified pairwise relations is obtained. By virtue of graph fusion, the self-adaptive weight of each single-view is optimally determined without partiality. We demonstrate theoretical feasibility of the proposed method. Extensive experimental results in four multi-view data sets witness our superiority compared to the state-of-the-art approaches.

Double-Constrained Consensus Clustering with Application to Online Anti-Counterfeiting

Semi-supervised consensus clustering is a promising strategy to compensate for the subjectivity of clustering and its sensitivity to design factors, with various techniques being recently proposed to integrate domain knowledge and multiple clustering partitions. In this article, we present a new approach that makes double use of domain knowledge, namely to build the initial partitions, as well as to combine them. In particular, we show how to model and integrate must-link and cannot-link constraints into the objective function of a generic consensus clustering (CC) framework that maximizes the similarity between the consensus partition and the input partitions, which have, in turn, been enriched with the same constraints. In addition, borrowing from the theory of functional dependencies, the integrated framework exploits the notions of deductive closure and minimal cover to take full advantage of the logical implication between constraints. Using standard UCI benchmarks, we found that the resulting algorithm, termed CCC double-constrained consensus clustering), was more effective than plain CC at combining base-constrained partitions, with an average performance improvement of 5.54%. We then argue that CCC is especially well-suited for profiling counterfeit e-commerce websites, as constraints can be acquired by leveraging specific domain features, and demonstrate its potential for detecting affiliate marketing programs. Taken together, our experiments suggest that CCC makes the process of clustering more robust and able to withstand changes in clustering algorithms, datasets, and features, with a remarkable improvement in average performance.

Nonsmooth Optimization-Based Model and Algorithm for Semisupervised Clustering.

Using a nonconvex nonsmooth optimization approach, we introduce a model for semisupervised clustering (SSC) with pairwise constraints. In this model, the objective function is represented as a sum of three terms: the first term reflects the clustering error for unlabeled data points, the second term expresses the error for data points with must-link (ML) constraints, and the third term represents the error for data points with cannot-link (CL) constraints. This function is nonconvex and nonsmooth. To find its optimal solutions, we introduce an adaptive SSC (A-SSC) algorithm. This algorithm is based on the combination of the nonsmooth optimization method and an incremental approach, which involves the auxiliary SSC problem. The algorithm constructs clusters incrementally starting from one cluster and gradually adding one cluster center at each iteration. The solutions to the auxiliary SSC problem are utilized as starting points for solving the nonconvex SSC problem. The discrete gradient method (DGM) of nonsmooth optimization is applied to solve the underlying nonsmooth optimization problems. This method does not require subgradient evaluations and uses only function values. The performance of the A-SSC algorithm is evaluated and compared with four benchmarking SSC algorithms on one synthetic and 12 real-world datasets. Results demonstrate that the proposed algorithm outperforms the other four algorithms in identifying compact and well-separated clusters while satisfying most constraints.

A constrained spectral clustering method for lane identification using trajectory data

The rapid development of information and communications technologies acts as an enabler for the successful implementation of vehicle automation and advanced traffic management applications. In particular, the appearance of new sources of high-resolution trajectory data, such as videos obtained from drones, provides an opportunity to build accurate maps and enrich applications in traffic research at an unprecedented resolution. However, existing methods cannot handle certain features, such as, among others, accurate of road lane identification.This paper proposes a constrained spectral clustering method to identify lane information from high-resolution trajectory data. Contrary to state-of-the-art methods, such as the Gaussian mixture model, the proposed method is directly applicable to two-dimensional trajectory data, without assuming a constant number of lanes characterised by the same lane width. The trajectory data is clustered by taking into account the neighbourhood distances and prior knowledge via defining so-called must-link and cannot-link constraints, which significantly improve the clustering results, especially in cases where the number of lanes or the lane width changes. The proposed method has been evaluated through numerical experiments using data obtained from drone videos, and the results indicate that the method performs well on complex road segments, even in the presence of a varying number of lanes or lane-changing manoeuvres.

Constrained Clustering: General Pairwise and Cardinality Constraints

In this work, we study constrained clustering, where constraints are utilized to guide the clustering process. In existing works, two categories of constraints have been widely explored, namely pairwise and cardinality constraints. Pairwise constraints enforce the cluster labels of two instances to be the same (must-link constraints) or different (cannot-link constraints). Cardinality constraints encourage cluster sizes to satisfy a user-specified distribution. However, most existing constrained clustering models can only utilize one category of constraints at a time. In this paper, we enforce the above two categories into a unified clustering model starting with the integer program formulation of the standard K-means. As these two categories provide useful information at different levels, utilizing both of them is expected to allow for better clustering performance. However, the optimization is difficult due to the binary and quadratic constraints in the proposed unified formulation. To alleviate this difficulty, we utilize two techniques: equivalently replacing the binary constraints by the intersection of two continuous constraints; the other is transforming the quadratic constraints into bi-linear constraints by introducing extra variables. Then we derive an equivalent continuous reformulation with simple constraints, which can be efficiently solved by Alternating Direction Method of Multipliers (ADMM) algorithm. Extensive experiments on both synthetic and real data demonstrate: 1) when utilizing a single category of constraint, the proposed model is superior to or competitive with state-of-the-art constrained clustering models, and 2) when utilizing both categories of constraints jointly, the proposed model shows better performance than the case of the single category. The experimental results show that the proposed method exploits the constraints to achieve perfect clustering performance with improved clustering to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2-5$ </tex-math></inline-formula> % in classical clustering metrics, e.g., Adjusted Random Index (ARI), Mirkin’s Index (MI), and Huber’s Index (HI), outerperfomring all compared-againts methods across the board. Moreover, we show that our method is robust to initialization.

Active Pairwise Constraint Learning in Constrained Time-Series Clustering for Crop Mapping from Airborne SAR Imagery

Airborne SAR is an important data source for crop mapping and has important applications in agricultural monitoring and food safety. However, the incidence-angle effects of airborne SAR imagery decrease the crop mapping accuracy. An active pairwise constraint learning method (APCL) is proposed for constrained time-series clustering to address this problem. APCL constructs two types of instance-level pairwise constraints based on the incidence angles of the samples and a non-iterative batch-mode active selection scheme: the must-link constraint, which links two objects of the same crop type with large differences in backscattering coefficients and the shapes of time-series curves; the cannot-link constraint, which links two objects of different crop types with only small differences in the values of backscattering coefficients. Experiments were conducted using 12 time-series images with incidence angles ranging from 21.2° to 64.3°, and the experimental results prove the effectiveness of APCL in improving crop mapping accuracy. More specifically, when using dynamic time warping (DTW) as the similarity measure, the kappa coefficient obtained by APCL was increased by 9.5%, 8.7%, and 5.2% compared to the results of the three other methods. It provides a new solution for reducing the incidence-angle effects in the crop mapping of airborne SAR time-series images.

Towards more efficient local search algorithms for constrained clustering

Constrained clustering extends clustering by integrating user constraints, and aims to determine an optimal assignment under the constraints. In this paper, we propose a local search algorithm called FastCCP to solve the constrained clustering problem. In the algorithm, instances connected by must-link constraints are first merged into nodes, and then, a local search method is performed to handle the cannot-link constraints while minimizing the Within-Cluster Sum of Squares (WCSS). Several strategies are proposed to enhance the solution diversity and achieve a trade-off between constraint satisfaction and WCSS minimization during the search. Furthermore, a node-filtering strategy is proposed to improve the efficiency of the algorithm. Experiments are performed on benchmark datasets to evaluate our algorithm. The comparative results indicate that our algorithm outperforms state-of-the-art algorithms in terms of both the solution quality and CPU runtime.

Egocentric video co-summarization using transfer learning and refined random walk on a constrained graph

In this paper, we address the problem of egocentric video co-summarization. We show how a shot level accurate summary can be obtained in a time-efficient manner using random walk on a constrained graph in transfer learned feature space with label refinement. While applying transfer learning, we propose a new loss function capturing egocentric characteristics in a pre-trained ResNet on the set of auxiliary egocentric videos. Transfer learning is used to generate i) an improved feature space and ii) a set of labels to be used as seeds for the test egocentric video. A complete weighted graph is created for a test video in the new transfer learned feature space with shots as the vertices. We derive two types of cluster label constraints in form of Must-Link (ML) and Cannot-link (CL) based on the similarity of the shots. ML constraints are used to prune the complete graph which is shown to result in substantial computational advantage, especially, for the long duration videos. We derive expressions for the number of vertices and edges for the ML-constrained graph and show that this graph remains connected. Random walk is applied to obtain labels of the unmarked shots in this new graph. CL constraints are applied to refine the cluster labels. Finally, shots closest to individual cluster centres are used to build the summary. Experiments on the short duration videos as in CoSum and TVSum datasets and long duration videos as in ADL and EPIC-Kitchens datasets clearly demonstrate the advantage of our solution over several state-of-the-art methods.

Deep semi-supervised clustering for multi-variate time-series

Huge amount of data are nowadays produced by a large and disparate family of sensors, which typically measure multiple variables over time. Such rich information can be profitably organized as multivariate time-series. Collect enough labelled samples to set up supervised analysis for such kind of data is challenging while a reasonable assumption is to dispose of a limited background knowledge that can be injected in the analysis process. In this context, semi-supervised clustering methods represent a well suited tool to get the most out of such reduced amount of knowledge. With the aim to deal with multivariate time-series analysis under a limited background knowledge setting, we propose a semi-supervised (constrained) deep embedding time-series clustering framework that exploits knowledge supervision modeled as Must- and Cannot-link constraints. More in detail, our proposal, named conDetSEC (constrained Deep embedding time SEries Clustering), is based on Gated Recurrent Units (GRUs) with the aim to explicitly manage the temporal dimension associated to multi-variate time series data. conDetSEC implements a procedure in which an embedding generation step is combined with a clustering refinement step. Both steps exploit the small amount of available knowledge provided by Must- and Cannot-link constraints. More specifically, during the data embedding generation the constraints are used by jointly optimizing the network parameters via both unsupervised and semi-supervised tasks, while at the refinement step they are used in conjunction with the goal to stretch the embedding manifold towards the clustering centroids to recover a more clear cluster structure. Experimental evaluation on real-world benchmarks coming from diverse domains has highlighted the effectiveness of our proposal in comparison with state-of-the-art unsupervised and semi-supervised time-series clustering methods.

An exact algorithm for semi-supervised minimum sum-of-squares clustering

The minimum sum-of-squares clustering (MSSC), or k-means type clustering, is traditionally considered an unsupervised learning task. In recent years, the use of background knowledge to improve the cluster quality and promote interpretability of the clustering process has become a hot research topic at the intersection of mathematical optimization and machine learning research. The problem of taking advantage of background information in data clustering is called semi-supervised or constrained clustering. In this paper, we present branch-and-cut algorithm for semi-supervised MSSC, where background knowledge is incorporated as pairwise must-link and cannot-link constraints. For the lower bound procedure, we solve the semidefinite programming relaxation of the MSSC discrete optimization model, and we use a cutting-plane procedure for strengthening the bound. For the upper bound, instead, by using integer programming tools, we use an adaptation of the k-means algorithm to the constrained case. For the first time, the proposed global optimization algorithm efficiently manages to solve real-world instances up to 800 data points with different combinations of must-link and cannot-link constraints and with a generic number of features. This problem size is about four times larger than the one of the instances solved by state-of-the-art exact algorithms.

On mathematical optimization for clustering categories in contingency tables

Many applications in data analysis study whether two categorical variables are independent using a function of the entries of their contingency table. Often, the categories of the variables, associated with the rows and columns of the table, are grouped, yielding a less granular representation of the categorical variables. The purpose of this is to attain reasonable sample sizes in the cells of the table and, more importantly, to incorporate expert knowledge on the allowable groupings. However, it is known that the conclusions on independence depend, in general, on the chosen granularity, as in the Simpson paradox. In this paper we propose a methodology to, for a given contingency table and a fixed granularity, find a clustered table with the highest chi ^2 statistic. Repeating this procedure for different values of the granularity, we can either identify an extreme grouping, namely the largest granularity for which the statistical dependence is still detected, or conclude that it does not exist and that the two variables are dependent regardless of the size of the clustered table. For this problem, we propose an assignment mathematical formulation and a set partitioning one. Our approach is flexible enough to include constraints on the desirable structure of the clusters, such as must-link or cannot-link constraints on the categories that can, or cannot, be merged together, and ensure reasonable sample sizes in the cells of the clustered table from which trustful statistical conclusions can be derived. We illustrate the usefulness of our methodology using a dataset of a medical study.

SOS-SDP: An Exact Solver for Minimum Sum-of-Squares Clustering

The minimum sum-of-squares clustering problem (MSSC) consists of partitioning n observations into k clusters in order to minimize the sum of squared distances from the points to the centroid of their cluster. In this paper, we propose an exact algorithm for the MSSC problem based on the branch-and-bound technique. The lower bound is computed by using a cutting-plane procedure in which valid inequalities are iteratively added to the Peng–Wei semidefinite programming (SDP) relaxation. The upper bound is computed with the constrained version of k-means in which the initial centroids are extracted from the solution of the SDP relaxation. In the branch-and-bound procedure, we incorporate instance-level must-link and cannot-link constraints to express knowledge about which data points should or should not be grouped together. We manage to reduce the size of the problem at each level, preserving the structure of the SDP problem itself. To the best of our knowledge, the obtained results show that the approach allows us to successfully solve, for the first time, real-world instances up to 4,000 data points.

A Fast Semi-Supervised Clustering Framework for Large-Scale Time Series Data

Semi-supervised clustering algorithms have several limitations: 1) the computation complexity of them is very high, because calculating the similarity distances of pairs of examples is time-consuming; 2) traditional semi-supervised clustering methods have not considered how to make full use of must-link and cannot-link constraints. In the clustering, the contribution of a few pairwise constraints to the clustering performance is very limited, and some may negatively affect the outcome; and 3) these methods are not effective to handle high dimensional data, especially for time series data. Up to now, few work touched semi-supervised clustering on time series data. To efficiently cluster large-scale time series data, we first tackle contract time series clustering to produce the most accurate clustering results under a contracted time. We propose a semi-supervised time series clustering framework (STSC), which integrates a fast similarity measure and a constraint propagation approach. Based on the proposed framework, two valid semi-supervised clustering algorithms including fssK-means and fssDBSCAN are designed. Experiments on 11 datasets show that our proposed method is efficient and effective for clustering large-scale time series data.

Enhancing instance-level constrained clustering through differential evolution

Clustering has always been a powerful tool in knowledge discovery. Traditionally unsupervised, it received renewed attention when it was shown to produce better results when provided with new types of information, thus leading to a new kind of semi-supervised learning: constrained clustering. This technique is a generalization of traditional clustering that considers additional information encoded by constraints. Constraints can be given in the form of instance-level must-link and cannot-link constraints, which this paper focuses on. We propose the first application of Differential Evolution to the constrained clustering problem, which has proven to produce a better exploration–exploitation trade-off when comparing with previous approaches. We will compare the results obtained by this proposal to those obtained by previous nature-inspired techniques and by some of the state-of-the-art algorithms on 25 datasets with incremental levels of constraint-based information, supporting our conclusions with the aid of Bayesian statistical tests.

Sleep Staging Using Plausibility Score: A Novel Feature Selection Method Based on Metric Learning.

As an effective method, feature selection can reduce computational complexity and improve classification performance. A number of criteria exist for feature selection using labeled data, unlabeled data and pairwise constraints, most of which are based on the Euclidean distance. In this paper, we propose a filter method for feature selection with pairwise constraints, aiming to jointly evaluate a feature subset based on metric learning. Two criteria are designed based on the well-known Kullback-Leibler divergence for measuring the difference between must-link constraints and cannot-link constraints that can indicate the feature subset discrimination based on Keep It Simple and Straightforward (KISS) metric learning and Cross-view Quadratic Discriminant Analysis (XQDA) metric learning. To address the challenging feature selection problem, we formulate a sequential search algorithm guided by indicators that are simplified from the proposed criteria. Furthermore, we conducted several experiments on sleep staging based on electroencephalogram (EEG) recordings from the Sleep-EDF Database Expanded. The experimental results demonstrate the effectiveness of the proposed method compared with nine representative feature selection methods. On the data set from healthy volunteers and the data set from volunteers that had mild difficulty falling asleep, the classification average accuracies achieve 97.66% and 93.57% by using the proposed method, respectively.