Hierarchical Density-based Clustering Research Articles

Statistical and density-based clustering of geographical flows for crowd movement patterns recognition

With the rapid development of sensors and communication technologies, it has become easy to collect large-scale and long-term crowd movement positioning data, which brings new opportunities for studying crowd movement patterns. This paper proposes a novel Statistical and Density-Based Clustering algorithm (SDBC) to identify implicit significant spatial aggregation patterns in geographical flow data. Unlike existing flow clustering algorithms, this method identifies the hot spots of origin-destination (OD) flows based on local spatial statistics and density-growing clustering. It also evaluates the significance of the identified geographic flow clusters, effectively reducing the identification of spurious clusters generated by chance in data. In our method, the spatial neighborhood of each flow is first obtained based on spatial proximity, temporal similarity, and directional similarity. Then, the number of flows in the spatial neighborhood of each flow is calculated and used as the density measure. Based on this, high-density flows are automatically detected using local spatial aggregation statistics, and a hierarchical density-based clustering strategy is developed to merge adjacent high-density flows to generate candidate flow clusters. Finally, we perform permutation tests to infer the statistical significance of each flow cluster and eliminate the candidate clusters generated by chance. Experiments on synthetic data and real-world taxi trajectory data were conducted to evaluate the effectiveness of the proposed method. Results show that the proposed method can accurately identify the statistically significant flow clusters of different shapes and densities and performs better than the available state-of-the-art flow clustering algorithms.

Document retrieval using clustering-based Aquila hash-Q optimization with query expansion based on pseudo relevance feedback

A document retrieval system helps users to retrieve the relevant documents corresponding to their query quickly and easily. In the real world, document retrieval is a difficult task due to high volumes of data, unstructured data, and different formats of data. Even though many research techniques are introduced, major problems like vocabulary mismatch and non-linear matching still need to be solved. In this work, the Aquila hash-q optimizer is the proposed matching technique with the clustering technique to retrieve the document in a time-efficient manner for the user query without collision. First, preprocessing is done by eliminating the stop words from the document, stemming, and grouping documents in a cluster into a single document using Hierarchical Density-based Sampling Spatial Cluster of Applications with Noise (HDBSSCAN) clustering. This clustering algorithm is powerful, robust to noise, and scalable and identifies clusters of documents that are related to each other. Additionally, the sampling technique used in this clustering algorithm increases the clustering speed by reducing the size of the document which improves the performance of document retrieval systems. Secondly, the queries are searched using the Aquila hash-q optimizer matching technique by which the relevant documents are retrieved. The Aquila hash-q optimization works by pre-computing a hash table of the terms in a document collection and then using this hash table to quickly identify the relevant documents from the given query. This can significantly improve the speed of document retrieval, especially for large document collections. Aquila hash-q optimization can improve the accuracy, efficiency, and scalability of document retrieval systems. The effectiveness of the Hierarchical Density-Based Clustering Aquila Optimization approach is determined by various analyses through NPL, LISA, and CACM data in terms of precision @ 5 (0.497), precision @ 10 (0.425), Mean Average Precision (MAP) (0.462) by comparing our approach with various methods. As a result, the Aquila hash-q optimizer is the proposed matching technique to retrieve the document in a time-efficient manner for the user query without collision.

Advanced Clustering Techniques for Emotional Grouping in Learning Environments Using an AR-Sandbox

Recently, it has been proven that the emotional aspect directly influences the learning process, so that, based on data mining techniques, this behavior has been sought to be characterized. This has made clustering techniques become one of the most used techniques for this purpose. However, studies where emotional data obtained from a person’s brain activity are used, are rare. For this reason, the present study aims to implement and compare advanced clustering techniques based on emotional metrics obtained through Brain-Computer Interfaces, captured in an AR-Sandbox, which fulfills the role of a learning environment. The evaluation of these techniques is carried out using internal criteria such as silhouette coefficient, Composed Density Between and within, Calinski-Harabasz and other statistical measures. When carrying out this study, it was obtained as a result that, the Density-Based Spatial Clustering of Application with Noise and Density-Based Hierarchical Spatial Clustering of Noisy Applications algorithms as the Density-based clustering methods, presented a better level of well-separation, cohesion and compaction, in comparison to the rest of the techniques implemented.

A Clustering-based Approach for Predicting the Future Location of a Vessel

In recent years the number of vessel accidents has increased with the growing number of vessels in the Ocean. A substantial number of these accidents are due to human errors. However, a vessel could avoid some accidents if it knows future locations or trajectories of surrounding vessels. The future location (or trajectory) prediction of vessels is also valuable for maritime traffic management and planning. Existing approaches for predicting the future location are not vessel-specific, but the behavioural pattern of each type of vessel is different. On the other hand, clustering-based approaches are computationally expensive due to computing the similarity measures among vessel trajectories. In this paper, we propose a clustering-based framework that consists of three modules: dataset preparation, route clustering, and location prediction model. The main novelty of this proposal is that the prediction models are built using only trajectories of vessels of the same type, which we hypothesise will increase their accuracy. The other main contribution of our work is the evaluation of different strategies to improve computational efficiency, which can be critical for real-time vessel location prediction. We have used a density-based hierarchical clustering algorithm, HDBSCAN, where two fast implementations of trajectory similarity measures (Hausdorff Distance and Dynamic Time Warping) are evaluated for clustering route patterns. Also, the effect of using a trajectory compression algorithm (Ramer–Douglas–Peucker) to expedite the clustering process is analyzed. The performance of our approach is evaluated using real-world AIS data. We have compared our proposal against two baseline models. Evaluation results confirm the validity of the proposed approach for prediction of the future location of vessels

CASPIAN: A method to identify chromatin topological associated domains based on spatial density cluster

With the development of Hi-C technology, the detection of topologically associated domains (TADs) boundaries plays an important role in exploring the relationship between gene structure and expression. However, a method that can identify accurate TAD boundaries from the Hi-C contact matrix with different resolutions is currently lacking. We proposed a method named CASPIAN that can identify chromatin TAD boundaries based on the spatial density clustering algorithm. CASPIAN requires few parameters to call TADs. This method is realized using the hierarchical density-based clustering method HDBSCAN, where the distance of pairwise bins is calculated based on three distance metrics (Euclidean, Manhattan, and Chebyshev distance metric) to adapt to the characteristics of the Hi-C contact matrix generated from simulation experiments or normalized methods. Our results show that, same as standard methods (e.g., Insulation Score, TopDom), CASPIAN can enrich factors related to promoting the gene expression, such as CTCF, H3K4me1, H3K4me3, RAD21, POLR2A, and SMC3. We also calculated the approximate proportion of various factors anchored at the TAD boundaries to observe the distribution of these factors surrounding the TAD boundaries. In conclusion, CASPIAN is an easy method to explore the relationship between transcription factors and TAD boundaries. CASPIAN is available online (https://gitee.com/ghaiyan/caspian).

Segmentation of EM showers for neutrino experiments with deep graph neural networks

We introduce a first-ever algorithm for the reconstruction of multiple showers from the data collected with electromagnetic (EM) sampling calorimeters. Such detectors are widely used in High Energy Physics to measure the energy and kinematics of in-going particles. In this work, we consider the case when many electrons pass through an Emulsion Cloud Chamber (ECC) brick, initiating electron-induced electromagnetic showers, which can be the case with long exposure times or large input particle flux. For example, SHiP experiment is planning to use emulsion detectors for dark matter search and neutrino physics investigation. The expected full flux of SHiP experiment is about 1020 particles over five years. To reduce the cost of the experiment associated with the replacement of the ECC brick and off-line data taking (emulsion scanning), it is decided to increase exposure time. Thus, we expect to observe a lot of overlapping showers, which turn EM showers reconstruction into a challenging point cloud segmentation problem. Our reconstruction pipeline consists of a Graph Neural Network that predicts an adjacency matrix and a clustering algorithm. We propose a new layer type (EmulsionConv) that takes into account geometrical properties of shower development in ECC brick. For the clustering of overlapping showers, we use a modified hierarchical density-based clustering algorithm. Our method does not use any prior information about the incoming particles and identifies up to 87% of electromagnetic showers in emulsion detectors. The achieved energy resolution over 16,577 showers is σE/E = (0.095 ± 0.005) + (0.134 ± 0.011)/√(E). The main test bench for the algorithm for reconstructing electromagnetic showers is going to be SND@LHC.

Airborne Lidar Survey, Density-Based Clustering, and Ancient Maya Settlement in the Upper Usumacinta River Region of Mexico and Guatemala

We present results from the archaeological analysis of 331 km2 of high-resolution airborne lidar data collected in the Upper Usumacinta River basin of Mexico and Guatemala. Multiple visualizations of the DEM and multi-spectral data from four lidar transects crossing the Classic period (AD 350–900) Maya kingdoms centered on the sites of Piedras Negras, La Mar, and Lacanja Tzeltal permitted the identification of ancient settlement and associated features of agricultural infrastructure. HDBSCAN (hierarchical density-based clustering of applications with noise) cluster analysis was applied to the distribution of ancient structures to define urban, peri-urban, sub-urban, and rural settlement zones. Interpretations of these remotely sensed data are informed by decades of ground-based archaeological survey and excavations, as well as a rich historical record drawn from inscribed stone monuments. Our results demonstrate that these neighboring kingdoms in three adjacent valleys exhibit divergent patterns of structure clustering and low-density urbanism, distributions of agricultural infrastructure, and economic practices during the Classic period. Beyond meeting basic subsistence needs, agricultural production in multiple areas permitted surpluses likely for the purposes of tribute, taxation, and marketing. More broadly, this research highlights the strengths of HDBSCAN to the archaeological study of settlement distributions when compared to more commonly applied methods of density-based cluster analysis.

Hierarchical clustering that takes advantage of both density-peak and density-connectivity

This paper focuses on density-based clustering, particularly the Density Peak (DP) algorithm and the one based on density-connectivity DBSCAN; and proposes a new method which takes advantage of the individual strengths of these two methods to yield a density-based hierarchical clustering algorithm. We first formally define the types of clusters DP and DBSCAN are designed to detect; and then identify the kinds of distributions that DP and DBSCAN individually fail to detect all clusters in a dataset. These identified weaknesses inspire us to formally define a new kind of clusters and propose a new method called DC-HDP to overcome these weaknesses to identify clusters with arbitrary shapes and varied densities. In addition, the new method produces a richer clustering result in terms of hierarchy or dendrogram for a better understanding of cluster structures. Our empirical evaluation results show that DC-HDP produces the best clustering results on 28 datasets in comparison with 8 state-of-the-art clustering algorithms.

Multi-label classification with local pairwise and high-order label correlations using graph partitioning

In multi-label learning problems, the class labels are correlated and the label correlations can be leveraged to improve the predictive performance of a classifier. Methods that consider high-order correlations in the label space, usually do not utilize pairwise correlations. In most of these methods, label correlations are considered as prior knowledge, which can be misleading in problems with noisy or missing labels. In such cases, learning the label correlation as part of the model training task is more effective. In this paper, a rule-based evolutionary multi-label classification method is proposed that incorporates the local label correlations through the high-order label subsets and pairwise dependencies. Graph structures are employed to model the label dependencies and the estimated label similarities are used to obtain more accurate label sets for the classification rules. To refine the high-order label relations, a novel hierarchical density-based clustering method is proposed to obtain a k-way partitioning for the label graphs based on their pairwise correlations. The effectiveness of the proposed method is experimented on multiple benchmark datasets from different domains and compared with several well-known multi-label classification algorithms. The proposed method has shown the highest average rank along multiple metrics and the results are consistently better than or similar to the compared methods with statistical significance.

Markov state models from hierarchical density-based assignment

Markov state models (MSMs) have become one of the preferred methods for the analysis and interpretation of molecular dynamics (MD) simulations of conformational transitions in biopolymers. While there is great variation in terms of implementation, a well-defined workflow involving multiple steps is often adopted. Typically, molecular coordinates are first subjected to dimensionality reduction and then clustered into small "microstates," which are subsequently lumped into "macrostates" using the information from the slowest eigenmodes. However, the microstate dynamics is often non-Markovian, and long lag times are required to converge the relevant slow dynamics in the MSM. Here, we propose a variation on this typical workflow, taking advantage of hierarchical density-based clustering. When applied to simulation data, this type of clustering separates high population regions of conformational space from others that are rarely visited. In this way, density-based clustering naturally implements assignment of the data based on transitions between metastable states, resulting in a core-set MSM. As a result, the state definition becomes more consistent with the assumption of Markovianity, and the timescales of the slow dynamics of the system are recovered more effectively. We present results of this simplified workflow for a model potential and MD simulations of the alanine dipeptide and the FiP35 WW domain.

Marker-Less UAV-LiDAR Strip Alignment in Plantation Forests Based on Topological Persistence Analysis of Clustered Canopy Cover

A holistic strategy is established for automated UAV-LiDAR strip adjustment for plantation forests, based on hierarchical density-based clustering analysis of the canopy cover. The method involves three key stages: keypoint extraction, feature similarity and correspondence, and rigid transformation estimation. Initially, the HDBSCAN algorithm is used to cluster the scanned canopy cover, and the keypoints are marked using topological persistence analysis of the individual clusters. Afterward, the feature similarity is calculated by considering the linear and angular relationships between each point and the pointset centroid. The one-to-one feature correspondence is retrieved by solving the assignment problem on the similarity score function using the Kuhn–Munkres algorithm, generating a set of matching pairs. Finally, 3D rigid transformation parameters are determined by permutations over all conceivable pair combinations within the correspondences, whereas the best pair combination is that which yields the maximum count of matched points achieving distance residuals within the specified tolerance. Experimental data covering eighteen subtropical forest plots acquired from the GreenValley and Riegl UAV-LiDAR platforms in two scan modes are used to validate the method. The results are extremely promising for redwood and poplar tree species from both the Velodyne and Riegl UAV-LiDAR datasets. The minimal mean distance residuals of 31 cm and 36 cm are achieved for the coniferous and deciduous plots of the Velodyne data, respectively, whereas their corresponding values are 32 cm and 38 cm for the Riegl plots. Moreover, the method achieves both higher matching percentages and lower mean distance residuals by up to 28% and 14 cm, respectively, compared to the baseline method, except in the case of plots with extremely low tree height. Nevertheless, the mean planimetric distance residual achieved by the proposed method is lower by 13 cm.

Logistics service provider selection decision making for healthcare industry based on a novel weighted density-based hierarchical clustering

In recent years, Industry 4.0 makes a significant impact on the manufacturing industry, which enables the business more intelligent and efficient, all while minimizing costs. As known, the logistics concerns in the supply chain always play an important role to a manufacturing company, and decision on the selection of logistics service provider is a key point, especially for healthcare manufacture whose products are the medical devices or equipment of fragility and high cost. Practically there are so many logistics service providers with varieties in service quality, effectiveness, punctuality and reliability, that the manufacturers often encounter the challenge on the provider selection, and healthcare industry is no exception. However, the research on provider selection for healthcare manufacturers is quite limited. In order to help them to make the decision, this paper designs a logistics service provider selection scheme based on a novel weighted density-based hierarchical cluster analysis with integration of the analytic hierarchy process (AHP) for healthcare industry. Initially an evaluation index system reflecting the capability of the candidate providers in all aspects is established. To improve the clustering within the scheme, the density concept and the obtained weights are introduced into the traditional hierarchical cluster analysis (HCA) to shape a novel Weighted Density-Based HCA (WDBHCA). To validate the feasibility of the scheme, a case study on a specified healthcare industry manufacturer is carried out, and results fulfill the case company’s requirement which shows the feasibility of the proposed provider selection scheme. In addition, this scheme can be applied to the provider selection in other fields, as well.

Hierarchical Density-Based Clustering Using MapReduce

Hierarchical density-based clustering is a powerful tool for exploratory data analysis, which can play an important role in the understanding and organization of datasets. However, its applicability to large datasets is limited because the computational complexity of hierarchical clustering methods has a quadratic lower bound in the number of objects to be clustered. MapReduce is a popular programming model to speed up data mining and machine learning algorithms operating on large, possibly distributed datasets. In the literature, there have been attempts to parallelize algorithms such as Single-Linkage, which in principle can also be extended to the broader scope of hierarchical density-based clustering, but hierarchical clustering algorithms are inherently difficult to parallelize with MapReduce. In this paper, we discuss why adapting previous approaches to parallelize Single-Linkage clustering using MapReduce leads to very inefficient solutions when one wants to compute density-based clustering hierarchies. Preliminarily, we discuss one such solution, which is based on an exact, yet very computationally demanding, random blocks parallelization scheme. To be able to efficiently apply hierarchical density-based clustering to large datasets using MapReduce, we then propose a different parallelization scheme that computes an approximate clustering hierarchy based on a much faster, recursive sampling approach. This approach is based on HDBSCAN*, the state-of-the-art hierarchical density-based clustering algorithm, combined with a data summarization technique called data bubbles. The proposed method is evaluated in terms of both runtime and quality of the approximation on a number of datasets, showing its effectiveness and scalability.

Rapid sorting of radio galaxy morphology using Haralick features

ABSTRACT We demonstrate the use of Haralick features for the automated classification of radio galaxies. The set of thirteen Haralick features represent an extremely compact non-parametric representation of image texture, and are calculated directly from imagery using the Grey Level Co-occurrence Matrix (GLCM). The GLCM is an encoding of the relationship between the intensity of neighbouring pixels in an image. Using 10 000 sources detected in the first data release of the LOFAR Two-metre Sky Survey (LoTSS), we demonstrate that Haralick features are highly efficient, rotationally invariant descriptors of radio galaxy morphology. After calculating Haralick features for LoTSS sources, we employ the fast density-based hierarchical clustering algorithm hdbscan to group radio sources into a sequence of morphological classes, illustrating a simple methodology to classify and label new, unseen galaxies in large samples. By adopting a ‘soft’ clustering approach, we can assign each galaxy a probability of belonging to a given cluster, allowing for more flexibility in the selection of galaxies according to combinations of morphological characteristics and for easily identifying outliers: those objects with a low probability of belonging to any cluster in the Haralick space. Although our demonstration focuses on radio galaxies, Haralick features can be calculated for any image, making this approach also relevant to large optical imaging galaxy surveys.

Hierarchical density-based clustering methods for tolling zone definition and their impact on distance-based toll optimization

Real-time network control strategies such as congestion pricing have been used in a number of metropolitan areas around the world for traffic congestion mitigation. Recent advances in Global Navigation Satellite System (GNSS) technology have led to increasing interest in distance- or usage-based road pricing as an effective alternative to traditional facility-, cordon- and area-based pricing that typically rely on fixed infrastructure. In this paper, we propose the use of feature-variant clustering methods, OPTICS and HDBSCAN*, as a systematic approach for tolling zone definition to operationalize distance-based tolling schemes. Subsequently, we develop a framework for predictive distance-based toll optimization to evaluate network performance for the various tolling zone definitions derived from the aforementioned feature-variant clustering methods. In this framework, for a specific tolling zone definition, tolling function parameters are optimized using a simulation-based Dynamic Traffic Assignment (DTA) model operating within a rolling horizon scheme. Predictive optimization is integrated with the guidance information generation. Behavioral models capture drivers’ responses to the tolls in terms of trip cancellation and choices of mode, route and departure time. Experiments on the real-world Expressway and Major Arterials network of Singapore demonstrate improved effectiveness of distance-based toll optimization given tolling zone definitions derived from feature-variant clustering, compared to fixed cordon-based pricing, adaptive cordon-based pricing, as well as distance-based pricing with ad hoc tolling zone definitions. Further, the results indicate that the use of the marginal link cost tolls as a clustering feature produces the most robust tolling zone definitions and yields significant improvements in social welfare over ad hoc zone definitions and cordon-pricing. Finally, experiments on the Boston CBD network also demonstrate the effectiveness of distance-based toll optimization schemes on urban traffic networks.

Recursive density-based hierarchical clustering in gaussian distributed sensor network

Sensor networks are data-centric networks constrained with limited battery power and processing capabilities. One of the crucial challenges in sensor network is energy hole problem. In order to deal with the challenge, there exists several mechanisms, of which clustering is considered an energy-efficient solution. In general, clustering refers to the technique of grouping nodes on the basis of similarity in spatial arrangement. An appropriately clustered network helps in processing and aggregation of sensed data before routing the information to destined location. This paper proposes a soft computing based recursive approach for implementing density-based hierarchical clustering, for Gaussian distributed sensor network. Our proposed probabilistic approach creates hierarchical clusters recursively, which not only addresses the problem of energy hole, but also reduces transmission delay, thereby maintaining data freshness.

Determining the Levels of Urbanization in Iran Using Hierar-chical Clustering

In this study, we used a variety of factors that affect urbanization in Iran to evaluate different provinces in Iran in terms of the level of urbanization. Using information from census 2011, we collected data on 33 indicators related to urbanization in 31 provinces in Iran. To rank the provinces we used density-based hierarchical clustering scheme. To determine similarities or differences between the provinces, the square of the Euclidean distance dissimilarity coefficient; Ward's algorithm was used to merge the provinces to minimize intra-cluster variance. One-way analysis of variance (ANOVA) was used to determine the variance between the variables used to rank the provinces in terms of different levels of urbanization. Statistical analysis was performed using SPSS. The provinces in Iran were combined with each other in 30 stages and classified into four levels. Taking into account the variables used to rank the level of urbanization, Tehran, and Alborz provinces were at the highest level of urbanization. On the other hand, the provinces of Sistan and Baluchistan, Kerman, North Khorasan, South Khorasan, Hormozgan, and Bushehr were at the lowest level of urbanization. Identification of provinces at the same level of urbanization can help us to discover the strengths and weaknesses in the infrastructures of each of them. Given the differences between various levels of urbanization, the identification of factors that are effective in the process of urbanization can help to access more information required for designing plans for the years to come.

Efficient Computation and Visualization of Multiple Density-Based Clustering Hierarchies

HDBSCAN*, a state-of-the-art density-based hierarchical clustering method, produces a hierarchical organization of clusters in a dataset w.r.t. a parameter mpts. While a small change in mpts typically leads to a small change in the clustering structure, choosing a “good” mpts value can be challenging: depending on the data distribution, a high or low mpts value may be more appropriate, and certain clusters may reveal themselves at different values. To explore results for a range of mpts values, one has to run HDBSCAN* for each value independently, which can be computationally impractical. In this paper, we propose an approach to efficiently compute all HDBSCAN* hierarchies for a range of mpts values by building upon results from computational geometry to replace HDBSCAN*'s complete graph with a smaller equivalent graph. An experimental evaluation shows that our approach can obtain over one hundred hierarchies for the computational cost equivalent to running HDBSCAN* about twice, which corresponds to a speedup of more than 60 times, compared to running HDBSCAN* independently that many times. We also propose a series of visualizations that allow users to analyze a collection of hierarchies for a range of mpts values, along with case studies that illustrate how these analyses are performed.

Quantification of Solute Topology in Atom Probe Tomography Data: Application to the Microstructure of a Proton-Irradiated Alloy 625

The analysis of solute clustering in atom probe tomography (APT) has almost exclusively relied on a simple algorithm based on the simple friend-of-friend analysis where a threshold distance or maximum separation defines whether atoms are part of a cluster or part of the matrix. This method is however limited to very specific microstructures and is very sensitive to parameter selection. To expand the range and applicability of current APT analysis tools, we introduce new quantitative data analysis methods based on density-based hierarchical clustering algorithms and relevant to solute clustering and segregation. We demonstrate the methods’ performance on the complex microstructure developing in a proton-irradiated Alloy 625, specifically focusing on the analyses of nanoscale Al clusters, Si clusters, and Si-decorated dislocation loops.

STLP-GSM: a method to predict future locations of individuals based on geotagged social media data

ABSTRACTAn increasing number of social media users are becoming used to disseminate activities through geotagged posts. The massive available geotagged posts enable collections of users’ footprints over time and offer effective opportunities for mobility prediction. Using geotagged posts for spatio-temporal prediction of future location, however, is challenging. Previous studies either focus on next-place prediction or rely on dense data sources such as GPS data. Introduced in this article is a novel method for future location prediction of individuals based on geotagged social media data. This method employs the hierarchical density-based clustering algorithm with adaptive parameter selection to identify the regions frequently visited by a social media user. A multi-feature weighted Bayesian model is then developed to forecast users’ spatio-temporal locations by combining multiple factors affecting human mobility patterns. Further, an updating strategy is designed to efficiently adjust, over time, the proposed model to the dynamics in users’ mobility patterns. Based on two real-life datasets, the proposed approach outperforms a state-of-the-art method in prediction accuracy by up to 5.34% and 3.30%. Tests show prediction reliability is high with quality predictions, but low in the identification of erroneous locations.