Stream Clustering Research Articles

Multi-view streaming clustering with incremental anchor learning

Multi-view streaming clustering with incremental anchor learning

Direct observation of particle clustering and gas breakdown in charged granular streams

The dust layer formed by charged particles deposition significantly limit the collection efficiency and stable application of electrostatic precipitators (ESPs). However, the deposition characteristics and re-entrainment mechanism of charged particles in electric field still remain unclear. In this study, a transparent ESP was designed to investigate the dynamic process of charged particle deposition, clustering and gas breakdown in electric field. The results showed that deposited particles will aggregate and gradually grow in chains in high-voltage electric field. The length of chain agglomerates can exceed 10 mm. And interelectrode gas breakdown could suddenly occur, which significantly changed the particle charging characteristics and increased particle emission at the outlet. In addition, a perforated plate was proposed to inhibit the gas breakdown and particle re-entrainment. The mass concentrations of particle re-entrainment decreased by 85.2 % and 79.1 % for the spike electrode and the arista electrode at the applied voltage of 30 kV, respectively.

IDS-DEC: A novel intrusion detection for CAN bus traffic based on deep embedded clustering

As the automotive industry advances towards greater automation, the proliferation of electronic control units (ECUs) has led to a substantial increase in the connectivity of in-vehicle networks with the external environment. However, the widely used Controller Area Network (CAN), which serves as the standard for in-vehicle networks, lacks robust security features, such as authentication or encrypted information transmission. This poses a significant challenge to the security of these networks. Despite the availability of powerful intrusion detection methods based on machine learning and deep learning, there are notable limitations in terms of stability and accuracy in the absence of a supervised learning process with labeled data. To address this issue, this paper introduces a novel in-vehicle intrusion detection system, termed IDS-DEC. This system combines a spatiotemporal self-coder employing LSTM and CNN (LCAE) with an entropy-based deep embedding clustering. Specifically, our approach involves encoding in-vehicle network traffic into windowed messages using a stream builder, designed to adapt to high-frequency traffic. These messages are then fed into the LCAE to extract a low-dimensional nonlinear spatiotemporal mapping from the initially high-dimensional data. The resulting low-dimensional mapping is subjected to a dual constraint in conjunction with our entropy-based pure deep embedding clustering module. This creates a bidirectional learning objective, addressing the optimization problem and facilitating an end-to-end training pattern for our model to adapt to diverse attack environments. The effectiveness of IDS-DEC is validated using both the benchmark Car Hacking dataset and the Car Hacking-Attack & Defense Challenge dataset. Experimental results demonstrate the model's high detection accuracy across various attacks, stabilizing at approximately 99% accuracy with a 0.5% false alarm rate. The F1 score also stabilizes at around 99%. In comparison with unsupervised methods based on deep stream clustering, LSTM-based self-encoder, and classification-based methods, IDS-DEC exhibits significant improvements across all performance metrics.

Statistical analysis of water quality change by total maximum daily load policy stage

This study aimed to derive major pollutants for standard watersheds with relatively high pollution levels and identify their trends. Hence, the water quality index (WQI) and multivariate statistical techniques were used to analyze the water quality evaluation of standard watershed during the total maximum daily load (TMDL) policy period implemented since 2004. The WQI was calculated for 41 standard watersheds, divided into 14 main streams and 27 tributaries. Consequently, in the main stream, WQI decreased as it moved downstream after the MS5 site. In the tributary, WQI values were found to be low at TS11, TS15, TS16, and TS17 sites. However, the Mann–Kendall test results indicated that the overall WQI value was increasing, suggesting that the water quality of the study basin was improving as the TMDL policy phase progressed. Principal component analysis of the group with relatively low WQI among the standard watersheds revealed that water temperature, dissolved oxygen, total phosphorus, chemical oxygen demand, biochemical oxygen demand, and electrical conductivity were the main variables in the main stream cluster. In the tributary stream cluster, total phosphorus, dissolved oxygen, total suspended solids, total nitrogen, and electrical conductivity showed high loading values. This study proposed a series of methods to provide the basic data required for watershed management through detailed water quality assessment methods, statistical techniques for deriving major pollutants, and trend analysis of water quality evaluation.

Action and energy clustering of stellar streams in deforming Milky Way dark matter haloes

ABSTRACT We investigate the non-adiabatic effect of time-dependent deformations in the Milky Way (MW) halo potential on stellar streams. Specifically, we consider the MW’s response to the infall of the Large Magellanic Cloud (LMC) and how this impacts our ability to recover the spherically averaged MW mass profile from observation using stream actions. Previously, action clustering methods have only been applied to static or adiabatic MW systems to constrain the properties of the host system. We use a time-evolving MW–LMC simulation described by basis function expansions. We find that for streams with realistic observational uncertainties on shorter orbital periods and without close encounters with the LMC, e.g. GD-1, the radial action distribution is sufficiently clustered to locally recover the spherical MW mass profile across the stream radial range within a $2\sigma$ confidence interval determined using a Fisher information approach. For streams with longer orbital periods and close encounters with the LMC, e.g. Orphan–Chenab (OC), the radial action distribution disperses as the MW halo has deformed non-adiabatically. Hence, for OC streams generated in potentials that include an MW halo with any deformations, action clustering methods will fail to recover the spherical mass profile within a $2\sigma$ uncertainty. Finally, we investigate whether the clustering of stream energies can provide similar constraints. Surprisingly, we find for OC-like streams, the recovered spherically averaged mass profiles demonstrate less sensitivity to the time-dependent deformations in the potential.

Fractional Adaptive Resonance Theory (FRA-ART): An Extension for a Stream Clustering Method with Enhanced Data Representation

Clustering data streams has become a hot topic and has been extensively applied to many real-world applications. Compared with traditional clustering, data stream clustering is more challenging. Adaptive Resonance Theory (ART) is a powerful (online) clustering method, it can automatically adjust to learn both abstract and concrete information, and can respond to arbitrarily large non-stationary databases while having fewer parameters, low computational complexity, and less sensitivity to noise, but its limited feature representation hinders its application to complex data streams. In this paper, considering its advantages and disadvantages, we present its flexible extension for stream clustering, called fractional adaptive resonance theory (FRA-ART). FRA-ART enhances data representation by fractionally exponentiating input features using self-interactive basis functions (SIBFs) and incorporating feature interaction through cross-interactive basis functions (CIBFs) at the cost only of introducing an additionally adjustable fractional order. Both SIBFs and CIBFs can be precomputed using existing algorithms, making FRA-ART easily adaptable to any ART variant. Finally, comparative experiments on five data stream datasets, including artificial and real-world datasets, demonstrate FRA-ART’s superior robustness and comparable or improved performance in terms of accuracy, normalized mutual information, rand index, and cluster stability compared to ART and the state-of-the-art G-Stream algorithm.

Design and Implementation of an Intensive Care Unit Command Center for Medical Data Fusion.

The rapid advancements in Artificial Intelligence of Things (AIoT) are pivotal for the healthcare sector, especially as the world approaches an aging society which will be reached by 2050. This paper presents an innovative AIoT-enabled data fusion system implemented at the CMUH Respiratory Intensive Care Unit (RICU) to address the high incidence of medical errors in ICUs, which are among the top three causes of mortality in healthcare facilities. ICU patients are particularly vulnerable to medical errors due to the complexity of their conditions and the critical nature of their care. We introduce a four-layer AIoT architecture designed to manage and deliver both real-time and non-real-time medical data within the CMUH-RICU. Our system demonstrates the capability to handle 22 TB of medical data annually with an average delay of 1.72 ms and a bandwidth of 65.66 Mbps. Additionally, we ensure the uninterrupted operation of the CMUH-RICU with a three-node streaming cluster (called Kafka), provided a failed node is repaired within 9 h, assuming a one-year node lifespan. A case study is presented where the AI application of acute respiratory distress syndrome (ARDS), leveraging our AIoT data fusion approach, significantly improved the medical diagnosis rate from 52.2% to 93.3% and reduced mortality from 56.5% to 39.5%. The results underscore the potential of AIoT in enhancing patient outcomes and operational efficiency in the ICU setting.

Study on Spatio-Temporal Indexing Model of Geohazard Monitoring Data Based on Data Stream Clustering Algorithm

Yunnan Province, residing in the eastern segment of the Qinghai–Tibet Plateau and the western part of the Yunnan–Guizhou Plateau, faces significant challenges due to its intricate geological structures and frequent geohazards. These pose monumental risks to community safety and infrastructure. Unfortunately, conventional spatial indexing methods struggle with the enormous influx of geohazard data, exhibiting inadequacies in efficient spatio-temporal querying and failing to meet the swift response imperatives for real-time geohazard monitoring and early warning mechanisms. In response to these challenges, this study proffers a cutting-edge spatio-temporal indexing model, the BCHR-index, undergirded by data stream clustering algorithms. The operational schema of the BCHR-index model is bifurcated into two stages: real-time and offline. The real-time phase proficiently uses micro-clusters shaped by the CluStream algorithm in unison with a B+ tree to construct indices in memory, thereby satisfying the exigent response necessities for geohazard data streams. Conversely, the offline stage employs the CluStream algorithm and the Hilbert curve to manage heterogeneously distributed spatial objects. Paired with a B+ tree, this framework promotes efficient spatio-temporal querying of geohazard data. The empirical results indicate that the indexing model implemented in this study affords millisecond-level responses when faced with query requests from real-time geohazard data streams. Moreover, in aspects of spatial query efficiency and data-insertion performance, it demonstrates superior results compared to the R-tree and Hilbert-R tree models.

A topic-enhanced dirichlet model for short text stream clustering

A topic-enhanced dirichlet model for short text stream clustering

MCMSTStream: applying minimum spanning tree to KD-tree-based micro-clusters to define arbitrary-shaped clusters in streaming data

Stream clustering has emerged as a vital area for processing streaming data in real-time, facilitating the extraction of meaningful information. While efficient approaches for defining and updating clusters based on similarity criteria have been proposed, outliers and noisy data within stream clustering areas pose a significant threat to the overall performance of clustering algorithms. Moreover, the limitation of existing methods in generating non-spherical clusters underscores the need for improved clustering quality. As a new methodology, we propose a new stream clustering approach, MCMSTStream, to overcome the abovementioned challenges. The algorithm applies MST to micro-clusters defined by using the KD-Tree data structure to define macro-clusters. MCMSTStream is robust against outliers and noisy data and has the ability to define clusters with arbitrary shapes. Furthermore, the proposed algorithm exhibits notable speed and can handling high-dimensional data. ARI and Purity indices are used to prove the clustering success of the MCMSTStream. The evaluation results reveal the superior performance of MCMSTStream compared to state-of-the-art stream clustering algorithms such as DenStream, DBSTREAM, and KD-AR Stream. The proposed method obtained a Purity value of 0.9780 and an ARI value of 0.7509, the highest scores for the KDD dataset. In the other 11 datasets, it obtained much higher results than its competitors. As a result, the proposed method is an effective stream clustering algorithm on datasets with outliers, high-dimensional, and arbitrary-shaped clusters. In addition, its runtime performance is also quite reasonable.

Stream clustering guided supervised learning for classifying NIDS alerts

A Network Intrusion Detection System (NIDS) is a network monitoring technology for identifying cyber attacks, botnet command and control traffic, and other unwanted network activity. Unfortunately, organizational NIDS solutions can often generate tens or hundreds of thousands of alerts on a daily basis, with a significant part of them having low importance or being false positives. Therefore, high priority alerts become hard to spot, which overloads security analysts and complicates their work. The current paper addresses this problem and introduces a machine learning framework for classifying NIDS alerts with the help of stream clustering and supervised learning. We propose a stream-clustering-guided method for creating labeled NIDS alert data sets. The small data sets created using this method can be used for training high-performance supervised NIDS alert classifiers. This significantly reduces the human labeling effort and eases the application of supervised machine learning for NIDS alert classification. The proposed machine learning framework was evaluated on NIDS alerts collected over 2 months from the network of a large academic organization. The experimental results indicate that combining stream clustering and supervised learning into a NIDS alert classification framework significantly decreases the number of false positives, and thus reduces the workload of human security analysts. The framework also features low CPU time and memory consumption and can thus be run on commodity hardware. In conclusion, the proposed framework provides a cost-effective means of integrating machine learning into Security Operation Centers (SOCs). This enables the identification of critical NIDS alerts using high-performance classifiers, thereby assisting in the automation of alert handling tasks for SOC personnel.To address the lack of public data sets in the problem domain and foster further research, we publicly share the large labeled NIDS alert data set used in our experimental setup.

Prospects for Detecting Gaps in Globular Cluster Stellar Streams in External Galaxies with the Nancy Grace Roman Space Telescope

Stellar streams form through the tidal disruption of satellite galaxies or globular clusters orbiting a host galaxy. Globular cluster streams are exciting since they are thin (dynamically cold) and therefore sensitive to perturbations from low-mass subhalos. Since the subhalo mass function differs depending on the dark matter composition, these gaps can provide unique constraints on dark matter models. However, current samples are limited to the Milky Way. With its large field of view, deep imaging sensitivity, and high angular resolution, the upcoming Nancy Grace Roman Space Telescope (Roman) presents a unique opportunity to increase the number of observed streams and gaps significantly. This paper presents a first exploration of the prospects for detecting gaps in streams in M31 and other nearby galaxies with resolved stars. We simulate the formation of gaps in a Palomar 5–like stream and generate mock observations of these gaps with background stars in M31 and foreground Milky Way stellar fields. We assess Roman's ability to detect gaps out to 10 Mpc through visual inspection and with the gap-finding tool FindTheGap. We conclude that gaps of ≈1.5 kpc in streams that are created from subhalos of masses ≥5 × 106 M ⊙ are detectable within a 2–3 Mpc volume in exposure times of 1000 s to 1 hr. This volume contains ≈150 galaxies, including ≈eight galaxies with luminosities >109 L ⊙. Large samples of stream gaps in external galaxies will open up a new era of statistical analyses of gap characteristics in stellar streams and help constrain dark matter models.

Efficient Online Stream Clustering Based on Fast Peeling of Boundary Micro-Cluster.

A growing number of applications generate streaming data, making data stream mining a popular research topic. Classification-based streaming algorithms require pre-training on labeled data. Manually labeling a large number of samples in the data stream is impractical and cost-prohibitive. Stream clustering algorithms rely on unsupervised learning. They have been widely studied for their ability to effectively analyze high-speed data streams without prior knowledge. Stream clustering plays a key role in data stream mining. Currently, most data stream clustering algorithms adopt the online-offline framework. In the online stage, micro-clusters are maintained, and in the offline stage, they are clustered using an algorithm similar to density-based spatial clustering of applications with noise (DBSCAN). When data streams have clusters with varying densities and ambiguous boundaries, traditional data stream clustering algorithms may be less effective. To overcome the above limitations, this article proposes a fully online stream clustering algorithm called fast boundary peeling stream clustering (FBPStream). First, FBPStream defines a decay-based kernel density estimation (KDE). It can discover clusters with varying densities and identify the evolving trend of streams well. Then, FBPStream implements an efficient boundary micro-cluster peeling technique to identify the potential core micro-clusters. Finally, FBPStream employs a parallel clustering strategy to effectively cluster core and boundary micro-clusters. The proposed algorithm is compared with ten popular algorithms on 15 data streams. Experimental results show that FBPStream is competitive with the other ten popular algorithms.

Enhancing User Interest based on Stream Clustering and Memory Networks in Large-Scale Recommender Systems

Enhancing User Interest based on Stream Clustering and Memory Networks in Large-Scale Recommender Systems

Streaming Readout and Data-Stream Processing With ERSAP

With the exponential growth in the volume and complexity of data generated at high-energy physics and nuclear physics research facilities, there is an imperative demand for innovative strategies to process this data in real or near-real-time. Given the surge in the requirement for high-performance computing, it becomes pivotal to reassess the adaptability of current data processing architectures in integrating new technologies and managing streaming data. This paper introduces the ERSAP framework, a modern solution that synergizes flow-based programming with the reactive actor model, paving the way for distributed, reactive, and high performance in data stream processing applications. Additionally, we unveil a novel algorithm focused on time-based clustering and event identification in data streams. The efficacy of this approach is further exemplified through the data-stream processing outcomes obtained from the recent beam tests of the EIC prototype calorimeter at DESY.

Online Sparse Representation Clustering for Evolving Data Streams.

Data stream clustering can be performed to discover the patterns underlying continuously arriving sequences of data. A number of data stream clustering algorithms for finding clusters in arbitrary shapes and handling outliers, such as density-based clustering algorithms, have been proposed. However, these algorithms are often limited in their ability to construct and merge microclusters by measuring the Euclidean distances between high-dimensional data objects, e.g., transferring valuable knowledge from historical landmark windows to the current landmark window, and exploiting evolving subspace structures adaptively. We propose an online sparse representation clustering (OSRC) method to learn an affinity matrix for evaluating the relationships among high-dimensional data objects in evolving data streams. We first introduce a low-dimensional projection (LDP) into sparse representation to adaptively reduce the potential negative influence associated with the noise and redundancy contained in high-dimensional data. Then, we take advantage of the l2,1 -norm optimization technique to choose the appropriate number of representative data objects and form a specific dictionary for sparse representation. The specific dictionary is integrated into sparse representation to adaptively exploit the evolving subspace structures of the high-dimensional data objects. Moreover, the data object representatives from the current landmark window can transfer valuable knowledge to the next landmark window. The experimental results based on a synthetic dataset and six benchmark datasets validate the effectiveness of the proposed method compared to that of state-of-the-art methods for data stream clustering.

A Comparative Analysis of Data Stream Clustering Algorithms

This study compares the performance of stream clustering algorithms (DenStream, CluStream, ClusTree) on Massive Online Analysis (MOA) using synthetic and real-world datasets. The algorithms are compared in the presence on noise level [0%, 10%, 30%] on the synthetic data. DenStream epsilon parameter was tune to 0.01 and 0.03 to improve its performance. We use the performance evaluation metrics CMM, F1-P, F1-R, Purity, Silhouette Coefficient, and Rand statistic. On synthetic data, our results show that ClusTree outperformed CluStream and DenStream on the almost all the metrics except in Purity and Silhouette were DenStream performs better at noise levels (10% and 30%). ClusTree outperform CluStream and DenStream on Forest Cover type dataset on metrics CMM, F1-P, F1-R, Silhouette Coefficient, and Rand statistic with 90%, 74%, 77% and 89% respectively. However, the tune DenStream epsilon parameter shows some improvements. On electricity data, DenStream outperform CluStream and ClusTree at epsilon parameter (0.03 and 0.05) on metrics F1-P, F1-R, and Purity. The investigation of DenStream epsilon parameter (0.03 and 0.05) on RandomBRF Generator with noise level [0%, 10%, 30%] shows that DenStream with epsilon 0.03 outperform other parameter adjustment.

A smart intelligent approach based on hybrid group search and pelican optimization algorithm for data stream clustering

A smart intelligent approach based on hybrid group search and pelican optimization algorithm for data stream clustering

Smoclust: synthetic minority oversampling based on stream clustering for evolving data streams

Many real-world data stream applications not only suffer from concept drift but also class imbalance. Yet, very few existing studies investigated this joint challenge. Data difficulty factors, which have been shown to be key challenges in class imbalanced data streams, are not taken into account by existing approaches when learning class imbalanced data streams. In this work, we propose a drift adaptable oversampling strategy to synthesise minority class examples based on stream clustering. The motivation is that stream clustering methods continuously update themselves to reflect the characteristics of the current underlying concept, including data difficulty factors. This nature can potentially be used to compress past information without caching data in the memory explicitly. Based on the compressed information, synthetic examples can be created within the region that recently generated new minority class examples. Experiments with artificial and real-world data streams show that the proposed approach can handle concept drift involving different minority class decomposition better than existing approaches, especially when the data stream is severely class imbalanced and presenting high proportions of safe and borderline minority class examples.

Temporal silhouette: validation of stream clustering robust to concept drift

Stream clustering is required in applications where data is generated continuously or periodically and must be processed considering its temporal nature. In the absence of a ground truth, internal validation is the only option to evaluate the quality of performances. Traditional internal validation is commonly used also in stream clustering, even in spite of the fact that it becomes inconsistent in the event of data evolution. Recent trends opt for incremental approaches, but these are closer to change detection rather than validation methods and limit themselves by imposing online validation on online analysis. In this work we study the impact of concept drift in the validation of stream clustering and propose the Temporal Silhouette index, therefore making internal validation conform to streaming data. We conduct tests with more than 200 datasets and contrast performances of four popular stream clustering algorithms with seven validation methods (three static internal, three incremental internal, one external) and the proposed index. Results show the suitability of the Temporal Silhouette index for stream clustering validation in the event of concept drift and different types of outliers. The demand for reliable unsupervised learning in applications that process data in streams is ever-increasing, and such reliability inevitably requires the use of validation. This fact highlights the significance of the novel approach proposed in this work.