Incremental Clustering Algorithm Research Articles

Online Self-Learning-Based Raw Material Proportioning for Rotary Hearth Furnace and Intelligent Batching System Development

With the increasing awareness of environmental protection, the rotary hearth furnace system has emerged as a key technology that facilitates a win-win situation for both environmental protection and enterprise economic benefits. This is attributed to its high flexibility in raw material utilization, capability of directly supplying blast furnaces, low energy consumption, and high zinc removal rate. However, the complexity of the raw material proportioning process coupled with the rotary hearth furnace system’s reliance on human labor results in a time-consuming and inefficient process. This paper innovatively introduces an intelligent formula method for proportioning raw materials based on online clustering algorithms and develops an intelligent batching system for rotary hearth furnaces. Firstly, the ingredients of raw materials undergo data preprocessing, which involves using the local outlier factor (LOF) method to detect any abnormal values, using Kalman filtering to smooth the data, and performing one-hot encoding to represent the different kinds of raw materials. Afterwards, the affinity propagation (AP) clustering method is used to evaluate past data on the ingredients of raw materials and their ratios. This analysis aims to extract information based on human experience with ratios and create a library of machine learning formulas. The incremental AP clustering algorithm is utilized to learn new ratio data and continuously update the machine learning formula library. To ensure that the formula meets the actual production performance requirements of the rotary hearth furnace, the machine learning formula is fine-tuned based on expert experience. The integration of machine learning and expert experience demonstrates good flexibility and satisfactory performance in the practical application of intelligent formulas for rotary hearth furnaces. An intelligent batching system is developed and executed at a steel plant in China. It shows an excellent user interface and significantly enhances batching efficiency and product quality.

Research on a Train Safety Driving Method Based on Fusion of an Incremental Clustering Algorithm and Lightweight Shared Convolution.

This paper addresses the challenge of detecting unknown or unforeseen obstacles in railway track transportation, proposing an innovative detection strategy that integrates an incremental clustering algorithm with lightweight segmentation techniques. In the detection phase, the paper innovatively employs the incremental clustering algorithm as a core method, combined with dilation and erosion theories, to expand the boundaries of point cloud clusters, merging adjacent point cloud elements into unified clusters. This method effectively identifies and connects spatially adjacent point cloud clusters while efficiently eliminating noise from target object point clouds, thereby achieving more precise recognition of unknown obstacles on the track. Furthermore, the effective integration of this algorithm with lightweight shared convolutional semantic segmentation algorithms enables accurate localization of obstacles. Experimental results using two combined public datasets demonstrate that the obstacle detection average recall rate of the proposed method reaches 90.3%, significantly enhancing system reliability. These findings indicate that the proposed detection strategy effectively improves the accuracy and real-time performance of obstacle recognition, thereby presenting important practical application value for ensuring the safe operation of railway tracks.

Planning, Design, and Execution of a Quantum Incremental Clustering Algorithm System for the Examination of Unsupervised Data

The exponential growth of data in various domains has underscored the need for efficient unsupervised learning algorithms to discover hidden patterns and structures within large datasets. Traditional clustering algorithms often face challenges in handling dynamic datasets and require periodic retraining, making them less suitable for real-time applications. In response to these challenges, this research introduces a Quantum Incremental Clustering Algorithm System (QICAS) designed to adapt to evolving datasets and provide timely insights into the underlying structures. The proposed QICAS is based on quantum computing principles, leveraging the unique properties of qubits and quantum superposition to perform clustering tasks more efficiently than classical counterparts. The research focuses on the planning, design, and execution of the QICAS, ensuring a comprehensive understanding of the algorithm's capabilities and performance.

An incremental clustering algorithm based on semantic concepts

An incremental clustering algorithm based on semantic concepts

Key grids based batch-incremental CLIQUE clustering algorithm considering cluster structure changes

In the network environment, data from various industries is dynamic and large-scale. Traditional clustering algorithms struggle to effectively utilize existing clustering results when faced with continuously evolving data, which makes the incremental grid-based clustering highly regarded. However, the existing incremental grid-based clustering algorithms fail to adequately consider the impact of newly added data on the original cluster structure. To address this issue, the key grids based batch-incremental CLIQUE clustering algorithm is proposed. The algorithm designates the incremental data mapping grids, which are or their neighbour girds are mixed with original data, as key grids to fully consider the cluster structure changes caused by the incremental data. Moreover, the cluster similarity coefficient based on grid features is introduced to measure density differences between the incremental data and the original clusters, and the cluster membership degree is defined to further consider the cluster membership of boundary sparse grid data and the identification of noise points. All of which ensures that the algorithm can adaptively create, merge or split clusters with the arrival of new data. Experimental results show that the proposed algorithm can adaptively adjust the cluster structure during incremental clustering, outperforming in accuracy and efficiency when clustering large-scale, dynamically changing data.

An Improved Jaccard Coefficient‐Based Clustering Approach with Application to Diagnosis and RUL Estimation

Sample clustering techniques play a crucial role in the data‐driven state evaluation of electromechanical equipment, and selecting an appropriate similarity measurement method for sample sets helps improve the clustering performance. The Jaccard coefficient is a commonly employed indicator of similarity for scalar set‐type samples. In this paper, we propose an incremental clustering algorithm for matrix‐type samples by defining an improved Jaccard coefficient. First, a new binary relation is formulated to derive a relationship matrix between samples. Second, an undirected graph is given by using the relationship matrix, and an improved pruning operation is provided to simplify the graph by eliminating redundant edges. Then, a new relationship matrix is generated according to the modified graph, which enables the calculation of the improved Jaccard coefficient. By using the improved Jaccard coefficient, the improved incremental clustering algorithm updates cluster centers by selecting a particular sample to maximize the sum of similarities between the selected sample and other samples within the same cluster. Finally, the effectiveness of the proposed incremental clustering algorithm is demonstrated in fault diagnosis and remaining useful life estimation application scenarios, respectively. The experimental results indicate that the improved algorithm outperforms traditional clustering methods.

An Incremental Clustering Algorithm for Arbitrary Shaped in IoT Data

An Incremental Clustering Algorithm for Arbitrary Shaped in IoT Data

Incremental density clustering framework based on dynamic microlocal clusters

With the prevailing development of the internet and sensors, various streaming raw data are generated continually. However, traditional clustering algorithms are unfavorable for discovering the underlying patterns of incremental data in time; clustering accuracy cannot be assured if fixed parameters clustering algorithms are used to handle incremental data. In this paper, an Incremental-Density-Micro-Clustering (IDMC) framework is proposed to address this concern. To reduce the succeeding clustering computation, we design the Dynamic-microlocal-clustering method to merge samples from streaming data into dynamic microlocal clusters. Beyond that, the Density-center-based neighborhood search method is proposed for periodically merging microlocal clusters to global clusters automatically; at the same time, these global clusters are updated by the Dynamic-cluster-increasing method with data streaming in each period. In this way, IDMC processes sensor data with less computational time and memory, improves the clustering performance, and simplifies the parameter choosing in conventional and stream data clustering. Finally, experiments are conducted to validate the proposed clustering framework on UCI datasets and streaming data generated by IoT sensors. As a result, this work advances the state-of-the-art of incremental clustering algorithms in the field of sensors’ streaming data analysis.

Statement of Retraction: Incremental Outlier Feature Clustering Algorithm in Blockchain Networks Based on Big Data Analysis

Statement of Retraction: Incremental Outlier Feature Clustering Algorithm in Blockchain Networks Based on Big Data Analysis

A big data exploration approach to exploit in-vehicle data for smart road maintenance

In modern Smart Cities, pervasive collection of sensor-based and IoT data streams is a challenging opportunity for improving mobility resilience. Among the potential applications, sensor-based data streams provide valuable information about the quality of the area-wide road surface. Modern vehicle black boxes are also able to estimate the type of anomaly (e.g., bump, hole, rough ground, depression), based on real-time analysis of acceleration data streams. Road maintainers may use all this information to improve monitoring and maintenance activities. However, the volume of data streams, the variety of road network and different degrees of seriousness of detected anomalies call for methods to support maintainers in the exploration of available data. To this aim, in this paper, we propose a methodological approach, based on big data exploration techniques. The approach is grounded on: (i) a multi-dimensional model, apt to organise data streams according to different dimensions and enable data exploration; (ii) data summarisation techniques, based on an incremental clustering algorithm, to simplify the overall view over massive data streams and to cope with their dynamic nature; (iii) a measure of relevance, to focus the attention on road portions that present critical conditions. The innovative contributions regard the formalisation of the exploration methodology, the definition of exploration scenarios, based on road maintainers’ goals and the measure of relevance, and an extensive experimentation on a real world case study, addressed in a research project on smart and resilient mobility. Experimental results show how relevance evaluation is able to efficiently attract the road maintainers’ attention on road portions that present the most critical conditions and the proposed incremental clustering algorithm outperforms existing algorithms in the literature.

Incremental Deinterleaving of Radar Emitters

In this paper, we present a new paradigm for the deinterleaving of radar signals in electronic support systems. Traditional systems observe the electromagnetic spectrum for a longer period of time and perform the deinterleaving operation batch-wise. This paper presents a new approach, where the deinterleaving is performed after each detected radar pulse, which we call pulse-to-pulse incremental deinterleaving. It enables minimum reaction time of the system for the detection of radar emitters. For that purpose the paper presents three incremental clustering algorithms based on DBSCAN, Adaptive Resonance Theory and the Leader approach. The algorithms are evaluated using simulated PDW data of realistic complex radar scenes. The results show, that incremental approaches can achieve a similar accuracy as non-incremental methods, but have much faster reaction times to the occurrence of new emitters.

Dynamic conditional score model-based weighted incremental fuzzy clustering of consumer power load data

This study proposes a weighted incremental fuzzy C-mean power load clustering algorithm based on the dynamic conditional score model to solve the problems that the predominant power load data mining method only captures the mean characteristics of time series and ignores the heteroscedasticity and sequence correlations. Consequently, this method has unsatisfactory time series clustering and low clustering accuracy. A dynamic conditional score model is constructed to analyze and extract statistical characteristic parameters of a time series to calculate the autocorrelation value of the parameter series. A weighted fuzzy C-mean clustering analysis is performed, and the obtained data weight information is used as input for incremental clustering to improve the clustering accuracy. The DCS model parameter dataset and data weight information are combined, and the clustering analysis of the consumer power load data stream is performed. The power load time series of different companies is given, and the clustering validity indices are defined for the performance analysis to verify the proposed clustering algorithm. The experimental results show that the proposed algorithm achieves satisfactory clustering and improves the performance.

Incremental clustering algorithm based on representative points and covariance for large data

Incremental clustering algorithm based on representative points and covariance for large data

Incremental clustering algorithm based on representative points and covariance for large data

Incremental clustering algorithm based on representative points and covariance for large data

A novel optimization approach towards improving separability of clusters

The objective functions in optimization models of the sum-of-squares clustering problem reflect intra-cluster similarity and inter-cluster dissimilarities and in general, optimal values of these functions can be considered as appropriate measures for compactness of clusters. However, the use of the objective function alone may not lead to the finding of separable clusters. To address this shortcoming in existing models for clustering, we develop a new optimization model where the objective function is represented as a sum of two terms reflecting the compactness and separability of clusters. Based on this model we develop a two-phase incremental clustering algorithm. In the first phase, the clustering function is minimized to find compact clusters and in the second phase, a new model is applied to improve the separability of clusters. The Davies–Bouldin cluster validity index is applied as an additional measure to compare the compactness of clusters and silhouette coefficients are used to estimate the separability of clusters. The performance of the proposed algorithm is demonstrated and compared with that of four other algorithms using synthetic and real-world data sets. Numerical results clearly show that in comparison with other algorithms the new algorithm is able to find clusters with better separability and similar compactness.

Research on incremental clustering algorithm for big data

Abstract As the scale of data becomes larger and larger, clustering processing, a key step in data mining, has important practical significance. Aiming at the problems of time consumption and high clustering errors when the current clustering algorithms deal with massive and dynamic big data, an incremental clustering algorithm is proposed by taking big data as the research object. By exploring the attribute characteristics of big data, four characteristics such as scale, diversity, high speed and value are summarised. For large-scale data streams that have multiple attributes and are acquired one by one, optimise the setting method of the K-means clustering algorithm category centre point, combine the K-means clustering algorithm and the Kalman filter algorithm and measure the distance between data point pairs. Instead of Mahalanobis distance, an incremental clustering algorithm suitable for big data is constructed. Five data sets are selected to carry out example analysis. The results of the algorithm are verified by the algorithm. The proposed algorithm has obvious advantages in the incremental clustering effect of big data. At the same time, it also has efficient and stable computing performance, which meets the expected design requirements and goals.

Context-aware incremental clustering of alerts in monitoring systems

The highly complex nature of today’s modern hybrid IT applications continues to present an increasing challenge for operation teams relying on traditional monitoring approaches. In monitoring systems, incidents occur frequently due to a variety of causes, from updates to software and hardware, to changes in operation environment. These incidents could significantly degrade the system’s availability and customers’ satisfaction. In many cases, investigating an incident in such an environment could feel like looking for a needle in a haystack - and you may not even know how the needle looks like until you see it. In that regard, one of the main challenges is how to efficiently analyze multiple sets of alert messages stemming from disparate monitoring tools and collectors across the application stack, in real-time. Such an analysis can provide trustworthy detection of system states at various critical points, thus helping teams to detect, frame, analyze and resolve incidents or failures in a relatively short time, especially if an accurate system’s topological dependencies are absent. In this work, we suggest a new approach to determining relations among alerts – forming “events”. The suggested approach directly models the event’s likelihood, by first embedding alerts’ corresponding metrics into a common latent space where the distance among metrics can be naturally defined, using a word2vec model, and then cluster alerts by employing a tailored incremental clustering algorithm. The suggested approach allows controlling the trade-off between the model’s sensitivity to clusters’ noise-robustness, thus spanning a wide range of clustering mechanisms, as well as adapting clusters’ outcomes to the level and properties of the noise expected in input data.

ITL-IDS: Incremental Transfer Learning for Intrusion Detection Systems

Utilizing machine learning methods to detect intrusion into computer networks is a trending topic in information security research. The limitation of labeled samples is one of the challenges in this area. This challenge makes it difficult to build accurate learning models for intrusion detection. Transfer learning is one of the methods to counter such a challenge in machine learning topics. On the other hand, the emergence of new technologies and applications might bring new vulnerabilities to computer networks. Therefore, the learning process cannot occur all at once. Incremental learning is a practical standpoint to confront this challenge. This research presents a new framework for intrusion detection systems called ITL-IDS that can potentially start learning in a network without prior knowledge. It begins with an incremental clustering algorithm to detect clusters’ numbers and shape without prior assumptions about the attacks. The outcomes are candidates to transfer knowledge between other instances of ITL-IDS. In each iteration, transfer learning provides target environments with incremental knowledge. Our evaluation shows that this method can combine incremental and transfer learning to identify new attacks.

Analysis of the Running Ability Mining Model of Football Trainers Based on Dynamic Incremental Clustering Algorithm.

Fast-running ability is a very important basic quality of football players. However, players are dynamic. It is difficult for coaches to grasp the running speed, instantaneous acceleration, and other indicators of small athletes in real time with the naked eye. Therefore, to accurately test the performance of athletes in fast-running ability, this paper studies the running ability mining model of football coaches based on the dynamic incremental clustering algorithm. According to scientific procedures and methods, the evaluation model and standard of running ability of Chinese elite female football players are established. The effectiveness of the model is 0.83, as verified by the standard recognition method, which shows that the evaluation model is efficient. The research considers the denoising of the original data. The model has rich data and standard test methods and procedures. It can be used as a measure of the running ability of China's elite female football players in a certain period and range. The research solves the problem of the insufficient running ability of domestic football players. It provides an important reference for training the next generation of excellent national football players.

NGIA: A novel Greedy Incremental Alignment based algorithm for gene sequence clustering

Gene sequence clustering is very basic and important in computational biology and bioinformatics for the study of phylogenetic relationships and gene function prediction, etc. With the rapid growth of the amount of biological data (gene/protein sequences), gene sequence clustering algorithms face more challenges in low precision and efficiency. The growing redundant sequences in gene sequence databases usually contribute to the increasing memory and computing demand for most clustering methods. For example, the original greedy incremental alignment-based (GIA) clustering algorithm obtains high precision clustering results, but with very poor efficiency. Efficient greedy incremental clustering algorithms have been developed with a cost of precision reduction, which usually trade clustering precision off for speed improvement. Algorithms with a better balance between precision and speed are needed. This paper proposes a novel Greedy Incremental Alignment-based algorithm called nGIA for gene clustering with high efficiency and precision. nGIA consists of a pre-filter, a modified short word filter, a new data packing strategy, a modified greedy incremental method, and is parallelized via GPU. The experimental evaluations on four independent datasets show that the proposed tool can cluster datasets with high precisions of 99.99%. Compared with the results of CD-HIT, Vsearch, and Uclust, nGIA is on average 13.6x, 6.2x, and 1.7x faster. In addition, we have developed a multi-node version to handle large data sets. The strong scalability test shows that the multi-node version of nGIA can scale up to 32 threads with a 31% parallel efficiency. The software is available at https://github.com/SIAT-HPCC/gene-sequence-clustering.