Use Of Clustering Algorithms Research Articles

Comparison of K-Means and FCM Algorithms to Optimize Spatiotemporal Pore Pressure Prediction of Earth Dams

To identify the behavior and health monitoring of dams, it is necessary to correctly interpret the results of instrumentation in different phases of construction, impounding, and operation. Therefore, the approach based on spatiotemporal prediction is presented to improve the interpretation of pore pressure behavior of Eyvashan Earth Dam. In this research, using the results of other existing healthy piezometers, a spatiotemporal distribution model is proposed using panel data, which can be effective for predicting and reconstructing missing data. The optimal spatiotemporal clustering of pore pressure changes monitoring with K-Means and Fuzzy C-means (FCM) algorithms will enable the monitoring of points of the dam where instrumentations are not designed and installed or defective instrumentations. In predicting the pore pressure of dams, the input data is classified based on the pore pressure monitoring data, but with the use of clustering algorithms, the classification after the cluster analysis steps will lead to the proper resolution of the pore pressure clustering. According to the validation results of each of the clustering algorithms, the FCM clustering algorithm has more suitable results than the K-Means algorithm in determining the pore pressure clusters. In general, FCM clustering and K-Means algorithms are suitable and efficient tools in the field of more accurate monitoring of earth dams, and by using the proposed method, the detection of unusual areas of pore pressure and the related safety diagnosis is facilitated.

A pipeline for processing hyperspectral images, with a case of melanin-containing barley grains as an example.

Analysis of hyperspectral images is of great interest in plant studies. Nowadays, this analysis is used more and more widely, so the development of hyperspectral image processing methods is an urgent task. This paper presents a hyperspectral image processing pipeline that includes: preprocessing, basic statistical analysis, visualization of a multichannel hyperspectral image, and solving classification and clustering problems using machine learning methods. The current version of the package implements the following methods: construction of a confidence interval of an arbitrary level for the difference of sample averages; verification of the similarity of intensity distributions of spectral lines for two sets of hyperspectral images on the basis of the Mann-Whitney U-criterion and Pearson's criterion of agreement; visualization in two-dimensional space using dimensionality reduction methods PCA, ISOMAP and UMAP; classification using linear or ridge regression, random forest and catboost; clustering of samples using the EM-algorithm. The software pipeline is implemented in Python using the Pandas, NumPy, OpenCV, SciPy, Sklearn, Umap, CatBoost and Plotly libraries. The source code is available at: https://github.com/igor2704/Hyperspectral_images. The pipeline was applied to identify melanin pigment in the shell of barley grains based on hyperspectral data. Visualization based on PCA, UMAP and ISOMAP methods, as well as the use of clustering algorithms, showed that a linear separation of grain samples with and without pigmentation could be performed with high accuracy based on hyperspectral data. The analysis revealed statistically significant differences in the distribution of median intensities for samples of images of grains with and without pigmentation. Thus, it was demonstrated that hyperspectral images can be used to determine the presence or absence of melanin in barley grains with great accuracy. The flexible and convenient tool created in this work will significantly increase the efficiency of hyperspectral image analysis.

Identification of an algorithm for the analysis and study of urban road network trajectories

The object of this study is a clustering algorithm using various technologies. This paper compares clustering algorithms that are more commonly used to analyze urban road network trajectories, the growth curve model, with the elbow method and the x-means algorithm. Experiments were conducted with various volumes of big data to determine calculation time, accuracy, and ways to increase calculation time. These results can be used to manage traffic jams in congested areas and city streets. Considering the widespread use of clustering algorithms for solving various problems, this study proposes to introduce GCM, SPGK methods for monitoring and analyzing the state of congestion on city roads. The work was carried out in the following steps: research and selection of methods based on efficiency and time, implementation of parallel computing technologies to improve computation speed, demonstration of the selected method based on collected data from a real city with visualization of the results. The growth curve model algorithm has been proven to be almost 5 times more effective than the elbow method and the x-means algorithm. The time allocated for data processing has been calculated. An increase in the volume of processed data showed an almost stable execution time t = 3 s for the GCM algorithm for data with a volume of up to almost 2,000 units. The effectiveness of SPGK-means was shown for different values of the number of points. Models of the Chengdu transport network obtained using a clustering algorithm with maximum grid density of neighborhoods are presented. There are some deviations between the grid and the road network due to the large grid size. This error is explained by an error of up to one between the points and the real grid. The results obtained clearly show how optimization of congested roads can be influenced. They provide information to obtain data on available routes, which allows you to analyze the road network individually and as a whole

Redefining the boundaries of Chinese cities—Analysis based on multisource geographical big data

In the process of rapid urbanization in China, the development of Chinese cities is progressing rapidly and urgently needs a new method of city boundary division that can reflect the true boundaries of cities. This study uses national road intersection and points of interest (POIs) data from 2016, among other multisource big data, to define the true boundaries of Chinese cities in that year. The results show the following: first, the use of clustering algorithms resulted in 6789 preliminary natural cities in 2016. Second, the combination of POI data and the elimination of POIs with a value of zero or below a threshold resulted in 4905 natural cities in China in 2016. Third, the 4905 natural cities were divided into four groups, resulting in 1981 sprawling cities, 842 vibrant and nonexpanding cities, 1048 expanding and nonvibrant cities, and 1034 vibrant and expanding cities. Fourth, the number and distribution of “ghost cities” nationwide were determined, with 612 natural cities identified as “ghost cities.” Fifth, various big datasets were used to verify the scientificity and rationality of the classification in this study.

An Investigation on the Use of Clustering Algorithms for Data Preprocessing in Breast Cancer Diagnosis

Classification algorithms are commonly used as a decision support system for diagnosing many diseases, such as breast cancer. The accuracy of classification algorithms can be affected negatively if the data contains outliers and/or noisy data. For this reason, outlier detection methods are frequently used in this field. In this study, we propose and compare various models that use clustering algorithms to detect outliers in the data preprocessing stage of classification to investigate their effects on classification accuracy. Clustering algorithms such as DBSCAN, HDBSCAN, OPTICS, FuzzyCMeans, and MCMSTClustering (MCMST) were used separately in the data preprocessing stage of the k Nearest Neighbor (kNN) classification algorithm for outlier elimination, and then the results were compared. According to the obtained results, MCMST algorithm was more successful in outlier elimination. The classification accuracy of the kNN + MCMST model was 0.9834, which was the best one, while the accuracy of kNN algorithm without using any data preprocessing was 0.9719.

Comparative analysis of clustering algorithm and improved algorithm application in shipping congestion

In recent years, due to the lack of reasonable planning of ship operation routes, shipping congestion accidents have been increasing, seriously restricting port development and channel operation, posing a great threat to ship navigation safety and greatly restricting the development prospects of the shipping industry. This paper aims to compare and analyze the application of the clustering algorithm and its improved algorithm in solving port and waterway congestion problems. By analyzing the literature on the use of clustering algorithms to solve shipping congestion problems, and comparing the advantages and disadvantages of multiple clustering algorithms in solving problems, this paper concludes that Partition-based Methods are more suitable for identifying port congestion and the improved Fuzzy DBSCAN algorithm for identifying channel congestion. The research in this paper will help to select clustering algorithms when solving shipping congestion problems in the future.

NeuralFlood: An AI-Driven Flood Susceptibility Index

Flood events have the potential to impact every aspect of life, economic loss and casualties can quickly be coupled with damages to agricultural land, infrastructure, and water quality. Creating flood susceptibility maps is an effective manner that equips communities with valuable information to help them prepare for and cope with the impacts of potential floods. Flood indexing and forecasting are nonetheless complex because multiple external parameters influence flooding. Accordingly, this study explores the potential of utilizing artificial intelligence (AI) techniques, including clustering and neural networks, to develop a flooding susceptibility index (namely, NeuralFlood) that considers multiple factors that are not generally considered otherwise. By comparing four different sub-indices, we aim to create a comprehensive index that captures unique characteristics not found in existing methods. The use of clustering algorithms, model tuning, and multiple neural layers produced insightful outcomes for county-level data. Overall, the four sub-indices’ models yielded accurate results for lower classes (accuracy of 0.87), but higher classes had reduced true positive rates (overall average accuracy of 0.68 for all classes). Our findings aid decision-makers in effectively allocating resources and identifying high-risk areas for mitigation.

NanoFeatures: a cross-platform application to characterize nanoparticles from super-resolution microscopy images.

Super-resolution microscopy and Single-Molecule Localization Microscopy (SMLM) are powerful tools to characterize synthetic nanomaterials used for many applications such as drug delivery. In the last decade, imaging techniques like STORM, PALM, and PAINT have been used to study nanoparticle size, structure, and composition. While imaging has progressed significantly, image analysis has often not advanced accordingly and many studies remain limited to qualitative and semi-quantitative analyses. Therefore, it is imperative to have a robust and accurate method to analyze SMLM images of nanoparticles and extract quantitative features from them. Here, we introduce nanoFeatures, a cross-platform Matlab-based app for the automatic and quantitative analysis of super-resolution images. nanoFeatures makes use of clustering algorithms to identify nanoparticles from the raw data (localization list) and extract quantitative information about size, shape, and molecular abundance at the single-particle and single-molecule levels. Moreover, it applies a series of quality controls, increasing data quality and avoiding artifacts. nanoFeatures, thanks to its intuitive interface, is also accessible to non-experts and will facilitate analysis of super-resolution microscopy for materials scientists and nanotechnologies. This easy accessibility to expansive feature characterization at the single particle level will bring us one step closer to understanding the relationship between nanostructure features and their efficiency (https://github.com/n4nlab/nanoFeatures).

NeuralFlood: an AI-driven flood susceptibility index

Flood events have the potential to impact every aspect of life, economic loss and casualties can quickly be coupled with damages to agricultural land, infrastructure, and water quality. Creating flood susceptibility maps is an effective manner that equips communities with valuable information to help them prepare for and cope with the impacts of potential floods. Flood indexing and forecasting are nonetheless complex because multiple external parameters influence flooding. Accordingly, this study explores the potential of utilizing artificial intelligence (AI) techniques, including clustering and neural networks, to develop a flooding susceptibility index (namely, NeuralFlood) that considers multiple factors that are not generally considered otherwise. By comparing four different sub-indices, we aim to create a comprehensive index that captures unique characteristics not found in existing methods. The use of clustering algorithms, model tuning, and multiple neural layers produced insightful outcomes for county-level data. Overall, the four sub-indices' models yielded accurate results for lower classes (accuracy of 0.87), but higher classes had reduced true positive rates (overall average accuracy of 0.68 for all classes). Our findings aid decision-makers in effectively allocating resources and identifying high-risk areas for mitigation.

Non-intrusive load monitoring: Comparative analysis of transient state clustering methods

Non-intrusive load monitoring is one of the key tools in demand-side management (DSM). Recent advancements in the computational power of processors have accentuated the role of machine learning algorithms e.g., clustering, as a key function in the NILM solutions applied on power grids. In event-based NILM methods, the algorithm detects the transient states (load events) and clusters them based on the similarity of different features of the transient state. In this study, the performances of eight clustering algorithms are comprehensively investigated and the impact of choosing different input signals, e.g., P, Q, and I, on transient states clustering is analyzed. Various input signals from the BLUED dataset are fed to the clustering algorithms. By comparing the evaluation metrics including shape-based and ground-truth-based metrics, it is observed that the OPTICS algorithm fed by dual-stream input streams outperformed the rest of the investigated clustering algorithms and input sets. OPTICS algorithm groups load events based on their density in multi-dimensional space, using a dynamic radius. The OPTICS algorithm, as the best-performing transient state clustering algorithm for the low-frequency NILM purpose, is then tested with the downsampled input data in a wide frequency range, to observe the impact of the data-sampling frequency on the results, which simplifies the use of clustering algorithms in future studies.

Data Mining Oriented Automatic Scientific Documents Summarization

The scientific research process usually begins with an examination of the advanced, which may include voluminous publications. Summarizing scientific articles can assist researchers in their research by speeding up the research process. The summary of scientific articles differs from the abstract text in general due to its specific structure and the inclusion of cited sentences. Most of the important information in scientific articles is presented in tables, statistics, and algorithm pseudocode. These features, however, rarely appear in the standard text. Therefore, a number of methods that consider the value of the structure of a scientific article have been suggested that improve the standard of the produced summary. This paper makes use of clustering algorithms to handle CL- SciSumm 2020 and longsumm 2020 tasks for summarization of scientific documents. There are three well-known clustering algorithms that are employed to tackle CL- SciSumm 2020 and LongSumm 2020 tasks, and several sentences recording functions, with textual deduction, are used to retrieved phrases from each cluster to generate summary.

Using clustering algorithms and GPM data to identify spatial precipitation patterns over southeastern Brazil

Southeastern Brazil comprises an important geoeconomic and populous region in South America. Consequently, it is essential to analyze and understand the precipitation profiles in this region. Among different data sources and techniques available to perform such study, the use of clustering algorithms and information from the Global Precipitation Measurement (GPM) project rises as a convenient yet few exploited alternative. Precisely, this study employs the K-Means, the Hierarchical Ward, and the Self-Organizing Maps methods to cluster the annual and seasonal precipitation data from GPM project recorded from 2001 to 2019. The adopted methods are compared in terms of quantitative measures and the number of clusters defined through a well-established rule. The results demonstrate that the annual and seasonal periods are organized according to different number of clusters. Moreover, the results allow: identify the presence of a spatially heterogeneous distribution in the study area; to conclude that the K-Means algorithm is a suitable clustering method in the context of this investigation when compared to Ward’s Hierarchical and Self-Organizing Maps methods in terms of the Calinski-Harabasz and Davies-Bouldin measures; and that the spatial precipitation distribution over Southeastern Brazil is represented by 10 clusters in annual and summer periods, 11 clusters in autumn and spring and 9 clusters in winter period.

Rigour and reproducibility in perinatal and paediatric epidemiologic research using big data.

Rigour and reproducibility in perinatal and paediatric epidemiologic research using big data.

Use of Clustering Algorithms for Sensor Placement and Activity Recognition in Smart Homes

This work presents a novel method for motion sensor placement within smart homes. Using recordings from 3D depth cameras within six real homes, clusters are created with the resident’s tracked location. The resulting clusters identify the possible position of a sensor and its field of view. By using a sequence of clusters as input to a Recurrent Neural Network, we evaluate our method on the task of activity recognition and prediction. These results are compared to using sensor events as input sequence, from motion sensors that were installed empirically in the same homes. Different clustering methods are investigated and all outperform the installed motion sensors, achieving a significant increase of prediction accuracy and F1-score.

Machine Learning-Based False Positive Software Vulnerability Analysis

Measurements and fault data from an older software version were used to build the fault prediction model for the new release. When past fault data isn't available, it's a problem. The software industry's assessment of programme module failure rates without fault labels is a difficult task. Unsupervised learning can be used to build a software fault prediction model when module defect labels are not available. These techniques can help identify programme modules that are more prone to errors. One method is to make use of clustering algorithms. Software module failures can be predicted using unsupervised techniques such as clustering when fault labels are not available. Machine learning clustering-based software failure prediction is our approach to solving this complex problem.

Equivalent Modelling of Hybrid RES Plant for Power System Transient Stability Studies

The paper proposes methodology for development of dynamic equivalent model of hybrid renewable energy source (HRES) plant suitable for reliable assessment of the overall transient stability of the realistic, large power systems. Transient stability index is used for system stability assessment, and consequently for the evaluation of the model performance. The historical plant production data and transmission network short-circuit fault statistics, together with probabilistically modelled uncertainties associated with system operation, are used to generate a set of realistic dynamic responses of the plant over the period of one year. The use of clustering algorithms enabled unsupervised grouping/clustering of responses and development of the low order equivalent models in the form of a transfer function with voltage as an input and real and reactive power as outputs. The methodology is illustrated on a number of case studies using two HRES plant configurations. It is demonstrated that a very few low order dynamic models can be used to model the HRES plant during the year for sufficiently accurate transient stability assessment of the power system. The possibility of deriving these equivalent models based on forecasted HRES plant composition makes them attractive for practical application in power system dynamic security studies.

Network Theory and Switching Behaviors: A User Guide for Analyzing Electronic Records Databases

As part of studies that employ health electronic records databases, this paper advocates the employment of graph theory for investigating drug-switching behaviors. Unlike the shared approach in this field (comparing groups that have switched with control groups), network theory can provide information about actual switching behavior patterns. After a brief and simple introduction to fundamental concepts of network theory, here we present (i) a Python script to obtain an adjacency matrix from a records database and (ii) an illustrative example of the application of network theory basic concepts to investigate drug-switching behaviors. Further potentialities of network theory (weighted matrices and the use of clustering algorithms), along with the generalization of these methods to other kinds of switching behaviors beyond drug switching, are discussed.

Looking for the -scape in the sound: Discriminating soundscapes categories in the Sonoran Desert using indices and clustering

Soundscapes are increasingly used as innovative entry doors in environmental studies. Facing huge libraries of sound files which cannot be processed manually, acoustic indices provide an overview to the information contained in them, as well as to allow for automatic processing. Studies dealing with such indices have, however, focused more on specific topics or indices than on the overall characteristics of the soundscapes they were analyzing. The aim of this paper is to propose a holistic approach to soundscapes. Our hypothesis is that sufficient number and variety of indices can help frame the characteristics of sound environments and that the use of clustering algorithms allows us to group them in families and study the distribution of those across space and time, revealing a geography that will not necessarily coincide with the obvious landscape/visual geography. To demonstrate this point, we have run indices analysis and classification on a soundscapes database recorded in the Sonoran Desert region (Southeastern Arizona, USA). The results show that sound indices reveal temporal variations and patterns of soundscapes and point out to sometimes surprising similarities between otherwise different environments. As sound indices capture a wealth of information which characterizes the environment at a given place and time, they could be used as proxies to continuous monitoring without having to store extreme amounts of data.

Time Efficient Unmanned Aircraft Systems Deployment in Disaster Scenarios Using Clustering Methods and a Set Cover Approach

Unmanned aircraft, which are more commonly known as drones, are nowadays extensively used in an ever increasing set of applications. In a wider system, the aircraft are usually associated to additional elements such as ground-based controllers. Furthermore, when these components form a network of elements that can communicate, the system is said to form an Unmanned Aircraft System (UAS). This system is particularly effective when the aircraft within are organized into swarms with sets of objectives to accomplish. The extensive use of swarms into UASs is more and more exploited nowadays due to the decreasing cost of those aircraft. In the present work we are interested in a particular application of UASs, namely their deployment in disaster scenarios for communications services provision to targets on the ground. These ground targets, however, are not part of the UASs and should not be confused with ground-based controllers. The present work does not only focus on coverage for ground targets but also on a guaranteed minimum number of covers for each target, which is called the redundancy requirement. The research work also ensures that the deployed UAS forms a unique connected component so that a steady stream of communication is kept with the targets to cover. Research work similar to the present perform the initial deployment of their aircraft in a different manner, either randomly, based on a predetermined grid formation, or using other elaborated methods. This work proposes a new solution based on the use of clustering algorithms, combined to a design of the problem formulated as a set cover optimization model. The clustering phase is used to discretize the search space and ease the optimization phase by locating regions of interest, and then a further procedure is applied, only when needed, to reconnect scattered connected components and guarantee connectivity in the networks. This way of doing it has achieved a deployment of UASs with maximum coverage for all targets, a guaranteed minimum number of covers for each of them, and results in a competitive computation time. The latter also allowed for more scalability by extending the tests to very large input instances.

Applying Data Mining to Analysis of Mismatch Between Education and Labor Market in Indonesia

One of the labor problems in Indonesia is the occurrence of mismatch (incompatibility between education and job). The incompatibility of education and job (labor market) can be categorized into several criteria such as vertical mismatch, horizontal mismatch, overqualification, under qualification, over skilling, under skilling, credentialism and economic skills. In several countries, analysis of mismatch between education and labor market can be used with several methods such as the Maximum Likelihood equation, the Mincer-type income equation, and the use of clustering algorithms in data mining. Indonesia can adopt a method of measuring the mismatch between education and labor market which suitable with the condition in Indonesia. The use of the K-Means algorithm or the hierarchical clustering algorithm will be able to see how much mismatch is occurring in Indonesia. Clustering is carried out referring to the Republic of Indonesia’s Presidential Regulation No. 8/2012 and the Indonesian Classification of Work (KBJI).