Data Clustering Research Articles

Large-Scale Gene Expression Data Clustering through Incremental Ensemble Approach

Abstract DNA microarray technology monitors gene activity in real-time in living organisms. It creates a large amount of data that helps scientists learn about how genes work. Clustering this data helps understand gene interactions and uncover important biological processes. However, the traditional clustering techniques have difficulties due to the enormous dimensionality of gene expression data and the intricacy of biological networks. Although ensemble clustering is a viable strategy, such high-dimensional data may not lend itself well to traditional approaches. This study introduces a novel technique for gene expression data clustering called Incremental Ensemble Clustering for Gene expression data (IECG). There are two steps in the IECG. A technique for grouping gene expression data into windows is presented in the first step, producing a tree of clusters. This procedure is carried out again for succeeding windows that have distinct feature sets. The base clusterings of two consecutive windows are ensembled using a new goal function to form a new clustering solution. By repeating this step by step method for further windows, reliable patterns that are beneficial for medical applications can be extracted. The results from both biological and non-biological data demonstrate that the proposed algorithm outperformed the state-of-the-art algorithms. Additionally, the running time of the proposed algorithm has been examined.

A Comprehensive Evaluation Method for Environmental Pollution in Tourist Attractions Based on Improved Principle Component Analysis

In order to address the problem of the reliability of current environmental pollution assessment methods, a comprehensive assessment method of environmental pollution in tourist scenic spots based on improved principle component analysis was proposed. First air, water, and soil pollution for data tourist attractions in the top 5 metropolitan cities of India are collected. Then, the original pollution data is cleaned, the data clustering method is used to classify the pollution data, and the comprehensive evaluation index system of environmental pollution in tourist attractions is established. Finally, the comprehensive evaluation method is implemented to calculate the evaluation values of various indicators, and the comprehensive assessment results of pollution at tourist attractions are obtained. The experimental results show that the correlation of the evaluation index is always controlled above 0.90, the convergence rate of the method is fast, the optimal solution can be approached within 150 iterations, and the application effect is good.

Rational partitioning of spectral feature space for effective clustering of massive spectral image data

We have successfully proposed and demonstrated a clustering method that overcomes the “needle-in-a-haystack problem” (finding minuscule important regions from massive spectral image data sets). The needle-in-a-haystack problem is of central importance in the characterization of materials since in bulk materials, the properties of a very tiny region often dominate the entire function. To solve this problem, we propose that rational partitioning of the spectral feature space in which spectra are distributed, or defining of the decision boundaries for clustering, can be performed by focusing on the discrimination limit defined by the measurement noise and partitioning the space at intervals of this limit. We verified the proposed method, applied it to actual measurement data, and succeeded in detecting tiny (~ 0.5%) important regions that were difficult for human researchers and other machine learning methods to detect in discovering unknown phases. The ability to detect these crucial regions helps in understanding materials and designing more functional materials.

Improving Performance of Cluster Heads Selection in DEC Protocol Using K-Means Algorithm for WSN.

Wireless sensor networks (WSN) have found more and more applications in remote control and monitoring systems. Energy management in the network is crucial because all nodes in the WSN are energy constrained. Therefore, the design and implementation of WSN protocols that reduce energy depletion in the network is still an open scientific problem. In this paper, we propose a new clustering protocol that combines DEC (deterministic energy-efficient clustering) protocol with K-means clustering, called DEC-KM (deterministic energy-efficient clustering protocol with K-means). DEC is a very energy-efficient clustering protocol that outperforms its predecessors, such as LEACH and SEP. K-means ensures more effective clustering and shorter data transmission distances within the network. The shorter distances improve the network's lifetime and stability and reduce power consumption. Additional heuristic rules in DEC-KM ensure improved cluster head selection, taking into account node energy level and position and minimising the risk of premature cluster head exhaustion. The simulation results for the DEC-KM protocol using MATLAB show that cluster heads have shorter distances to nodes in cluster areas than for the original DEC protocol. The proposed protocol ensures reduced energy consumption, outperforms the standard DEC, and extends the stability period and lifetime of the network.

Comparative Analysis of K-Means and Divisive Clustering Techniques on Balanced Mental Health Data

Effective Clustering of mental health data can provide significant insights into patterns and relationships that are critical for understanding mental health conditions. This study investigated various clustering techniques applied to balanced mental health data to avoid biases associated with an imbalanced data. Clustering of the balanced mental health data was done with respect to the area of Residence feature. Firstly, Random undersampling and SMOTE techniques were incorporated to the imbalanced data set as balancing techniques so as to improve model performance. Random Undersampling Technique turned out to be the most ideal balancing technique with its accuracy, recall, precision and F-score values as 1. After balancing the data, two clustering techniques were applied to the Random Undersampled balanced data. The two techniques were namely: K-means and Divisive techniques. In order to select which of the two clustering techniques is ideal, two test statistics namely Internal Validation and Stability Validation were applied. Results showed that K-means clustering technique indicated slightly lower Average Propotion of None-overlap, Average Distance between Means and Figure Of Merit values given as 0.12, 0.41 and 0.9972 as compared to Divisive clustering technique which were 0.14, 0.42 and 0.9999. The conclusion was that K-means clustering has a better performance. This study's findings will help guide future researchers dealing with mental health data analysis on ways to improve model performance for better and more reliable predictions.

Evaluating GreenFeed and respiration chambers for daily and intraday measurements of enteric gaseous exchange in dairy cows housed in tie-stalls

The objective of this study was to evaluate the GreenFeed (GF) and respiration chambers (RC) for daily and intraday measurements of the enteric gaseous exchange, as well as the metabolic heat production, lying behavior, and feed intake (FI) rate of dairy cows at these 2 respective housing conditions [tie-stall barn (TS) vs. RC] during the summer periods. Sixteen multiparous lactating dairy cows were recruited and arranged in a randomized complete block design with a baseline period established for each cow. Cows were given a basal diet (CON) for a baseline period of 7 d and were then fed a 3-nitrooxypropanol (3-NOP)-containing feed for the subsequent 26 d as experimental period. During both the baseline and the last 7 d of treatment period, gaseous exchanges of each animal were measured in the TS using GF for 8 6-hourly staggered measurements over 3 d, immediately followed by the measurement in RC for 2 d. Corresponding DMI, milk yield, and behavior parameters (e.g., lying behavior and FI rate) in TS and RC were recorded. The correlation coefficients of CH4 and H2 using raw data were 0.84 and 0.85, respectively. For all gases, correlation coefficients between GF and RC on individual cow level decreased when the marginal fixed effects (e.g., inhibitor and breed) were corrected by a mixed model. There were no differences in daily CH4 production or intensity between GF and RC (442 vs. 443 g CH4/d or 16.6 vs. 16.2 g CH4 /kg MY). However, greater CH4 yield was measured by GF than RC (19.0 vs. 17.8 g CH4/kg DMI), driven by a lower DMI (23.3 vs. 24.6 kg/d) when cows were housed in TS sampled by GF compared with cows being housed and sampled in RC. The correlations for CO2 production and O2 consumption were moderate and expected due to the variation associated with the mild heat stress condition during GF measurements in the TS (Thermal humidity index (THI) 56 vs. 68), as indicated by the reduced lying time (-2.1 h/d). At the intraday level, there was an interaction between techniques and hour-of-day for CH4 production, as indicated by the discrepancies in post-prandial CH4 emissions between techniques. In summary, this set of results showed that there were strong positive correlations for CH4 and H2 emissions between GF and RC based on individual cow data. However, such relationship should be interpreted with caution, given the data clustering resulting from the use of inhibitor 3-NOP. On treatment level, these 2 techniques detected similar inhibitor effect on the estimated daily CH4 emissions. The intraday patterns of CH4 and H2 production captured by GF provided a close approximation for those measured by RC. Nevertheless, potential underestimation may occur, especially following fresh feed delivery. For measuring CO2 production and O2 consumption, the GF captured similar intraday variations to those in the RC. However, the estimated daily production and consumption were not directly comparable, which was expected due to the variable thermal conditions during the summer. Further evaluations under the same weather conditions are warranted.

Exploring other clustering methods and the role of Shannon Entropy in an unsupervised setting

In the ever-expanding landscape of digital technologies, the exponential growth of data in information science and health informatics presents both challenges and opportunities, demanding innovative approaches to data curation. This study focuses on evaluating various feasible clustering methods within the Data Washing Machine (DWM), a novel tool designed to streamline unsupervised data curation processes. The DWM integrates Shannon Entropy into its clustering process, allowing for adaptive refinement of clustering strategies based on entropy levels observed within data clusters. Rigorous testing of the DWM prototype on various annotated test samples revealed promising outcomes, particularly in scenarios with high-quality data. However, challenges arose when dealing with poor data quality, emphasizing the importance of data quality assessment and improvement for successful data curation. To enhance the DWM’s clustering capabilities, this study explored alternative unsupervised clustering methods, including spectral clustering, autoencoders, and density-based clustering like DBSCAN. The integration of these alternative methods aimed to augment the DWM’s ability to handle diverse data scenarios effectively. The findings demonstrated the practicability of constructing an unsupervised entity resolution engine with the DWM, highlighting the critical role of Shannon Entropy in enhancing unsupervised clustering methods for effective data curation. This study underscores the necessity of innovative clustering strategies and robust data quality assessments in navigating the complexities of modern data landscapes. This content is structured by the following sections: Introduction, Methodology, Results, Discussion, and Conclusion.

A Highly Accurate Adverse Drug Reactions (ADR) Detection from Medical Forum Comments Using Long Short-Term Memory Networks

Purpose: Adverse Drug Reactions (ADR) classification is useful in modern medical diagnostics and related applications. ADR is an example of how medical information is frequently accessible on social media platforms for healthcare, where people can share their experiences with treatments on desktop computers and mobile devices. Many researchers are interested in gathering valuable medical data from social media for the ADR system training and classification process. Materials and Methods: This research explores the effects of three aspects on recognizing ADR mentions in social media for the medical field and proposes a deep neural network of Long Short-Term Memory (LSTM) neural networks to do so. The comments are collected from various social media platforms to implement the ADR system with proper training and testing processes. The texts from the dataset are initially preprocessed by using a data filtering and clustering process to remove the input data's redundant information to increase the training process's quality. Characteristic features, such as semantic features and text statistics, are extracted from the input text using the American Standard Code for Information Interchange (ASCII) array. Further, the features are converted and fed to LSTM networks for training and validation. Results and Conclusion: This work is evaluated using two datasets, CODEC, and ADR Corpus datasets are used to evaluate the performance of the proposed ADR technique via multiple angles. Via extensive experiments, this work achieved 99.79 accuracy, 98.37 sensitivity, 97.63 specificity, 99.72 precision, 98.39 recall, 97.62 F1-score for the CODEC dataset, 98.16 for accuracy, 99.19 for sensitivity, 98.49 for specificity, 99.49 for precision, 96.72 for recall, and 93.16 for F1-score for ADR corpus, respectively.

Community detection using Jaya optimization algorithm based on deep learning methods and KNN graph-based clustering

Purpose In this paper, community identification has been considered as the most critical task of social network analysis. The purpose of this paper is to organize the nodes of a given network graph into distinct clusters or known communities. These clusters will therefore form the different communities available within the social network graph. Design/methodology/approach To date, numerous methods have been developed to detect communities in social networks through graph clustering techniques. The k-means algorithm stands out as one of the most well-known graph clustering algorithms, celebrated for its straightforward implementation and rapid processing. However, it has a serious drawback because it is insensitive to initial conditions and always settles on local optima rather than finding the global optimum. More recently, clustering algorithms that use a reciprocal KNN (k-nearest neighbors) graph have been used for data clustering. It skillfully overcomes many major shortcomings of k-means algorithms, especially about the selection of the initial centers of clusters. However, it does face its own challenge: sensitivity to the choice of the neighborhood size parameter k, which is crucial for selecting the nearest neighbors during the clustering process. In this design, the Jaya optimization method is used to select the K parameter in the KNN method. Findings The experiment on real-world network data results show that the proposed approach significantly improves the accuracy of methods in community detection in social networks. On the other hand, it seems to offer some potential for discovering a more refined hierarchy in social networks and thus becomes a useful tool in the analysis of social networks. Originality/value This paper introduces an enhancement to the KNN graph-based clustering method by proposing a local average vector method for selecting the optimal neighborhood size parameter k. Furthermore, it presents an improved Jaya algorithm with KNN graph-based clustering for more effective community detection in social network graphs.

Mind the information gap: How sampling and clustering impact the predictability of reach‐scale channel types in California (USA)

AbstractClustering and machine learning‐based predictions are increasingly used for environmental data analysis and management. In fluvial geomorphology, examples include predicting channel types throughout a river network and segmenting river networks into a series of channel types, or groups of channel forms. However, when relevant information is unevenly distributed throughout a river network, the discrepancy between data‐rich and data‐poor locations creates an information gap. Combining clustering and predictions addresses this information gap, but challenges and limitations remain poorly documented. This is especially true when considering that predictions are often achieved with two approaches that are meaningfully different in terms of information processing: decision trees (e.g., RF: random forest) and deep learning (e.g., DNNs: deep neural networks). This presents challenges for downstream management decisions and when comparing clusters and predictions within or across study areas. To address this, we investigate the performance of RF and DNN with respect to the information gap between clustering data and prediction data. We use nine regional examples of clustering and predicting river channel types, stemming from a single clustering methodology applied in California, USA. Our results show that prediction performance decreases when the information gap between field‐measured data and geospatial predictors increases. Furthermore, RF outperforms DNN, and their difference in performance decreases when the information gap between field‐measured and geospatial data decreases. This suggests that mismatched scales between field‐derived channel types and geospatial predictors hinder sequential information processing in DNN. Finally, our results highlight a sampling trade‐off between uniformly capturing geomorphic variability and ensuring robust generalisation.

NsDCC: dual-level contrastive clustering with nonuniform sampling for scRNA-seq data analysis.

Dimensionality reduction and clustering are crucial tasks in single-cell RNA sequencing (scRNA-seq) data analysis, treated independently in the current process, hindering their mutual benefits. The latest methods jointly optimize these tasks through deep clustering. However, contrastive learning, with powerful representation capability, can bridge the gap that common deep clustering methods face, which requires pre-defined cluster centers. Therefore, a dual-level contrastive clustering method with nonuniform sampling (nsDCC) is proposed for scRNA-seq data analysis. Dual-level contrastive clustering, which combines instance-level contrast and cluster-level contrast, jointly optimizes dimensionality reduction and clustering. Multi-positive contrastive learning and unit matrix constraint are introduced in instance- and cluster-level contrast, respectively. Furthermore, the attention mechanism is introduced to capture inter-cellular information, which is beneficial for clustering. The nsDCC focuses on important samples at category boundaries and in minority categories by the proposed nearest boundary sparsest density weight assignment algorithm, making it capable of capturing comprehensive characteristics against imbalanced datasets. Experimental results show that nsDCC outperforms the six other state-of-the-art methods on both real and simulated scRNA-seq data, validating its performance on dimensionality reduction and clustering of scRNA-seq data, especially for imbalanced data. Simulation experiments demonstrate that nsDCC is insensitive to "dropout events" in scRNA-seq. Finally, cluster differential expressed gene analysis confirms the meaningfulness of results from nsDCC. In summary, nsDCC is a new way of analyzing and understanding scRNA-seq data.

ScDFN: enhancing single-cell RNA-seq clustering with deep fusion networks.

Single-cell ribonucleic acid sequencing (scRNA-seq) technology can be used to perform high-resolution analysis of the transcriptomes of individual cells. Therefore, its application has gained popularity for accurately analyzing the ever-increasing content of heterogeneous single-cell datasets. Central to interpreting scRNA-seq data is the clustering of cells to decipher transcriptomic diversity and infer cell behavior patterns. However, its complexity necessitates the application of advanced methodologies capable of resolving the inherent heterogeneity and limited gene expression characteristics of single-cell data. Herein, we introduce a novel deep learning-based algorithm for single-cell clustering, designated scDFN, which can significantly enhance the clustering of scRNA-seq data through a fusion network strategy. The scDFN algorithm applies a dual mechanism involving an autoencoder to extract attribute information and an improved graph autoencoder to capture topological nuances, integrated via a cross-network information fusion mechanism complemented by a triple self-supervision strategy. This fusion is optimized through a holistic consideration of four distinct loss functions. A comparative analysis with five leading scRNA-seq clustering methodologies across multiple datasets revealed the superiority of scDFN, as determined by better the Normalized Mutual Information (NMI) and the Adjusted Rand Index (ARI) metrics. Additionally, scDFN demonstrated robust multi-cluster dataset performance and exceptional resilience to batch effects. Ablation studies highlighted the key roles of the autoencoder and the improved graph autoencoder components, along with the critical contribution of the four joint loss functions to the overall efficacy of the algorithm. Through these advancements, scDFN set a new benchmark in single-cell clustering and can be used as an effective tool for the nuanced analysis of single-cell transcriptomics.

IPFMC: an iterative pathway fusion approach for enhanced multi-omics clustering in cancer research

Abstract Using multi-omics data for clustering (cancer subtyping) is crucial for precision medicine research. Despite numerous methods having been proposed, current approaches either do not perform satisfactorily or lack biological interpretability, limiting the practical application of these methods. Based on the biological hypothesis that patients with the same subtype may exhibit similar dysregulated pathways, we developed an Iterative Pathway Fusion approach for enhanced Multi-omics Clustering (IPFMC), a novel multi-omics clustering method involving two data fusion stages. In the first stage, omics data are partitioned at each layer using pathway information, with crucial pathways iteratively selected to represent samples. Ultimately, the representation information from multiple pathways is integrated. In the second stage, similarity network fusion was applied to integrate the representation information from multiple omics. Comparative experiments with nine cancer datasets from The Cancer Genome Atlas (TCGA), involving systematic comparisons with 10 representative methods, reveal that IPFMC outperforms these methods. Additionally, the biological pathways and genes identified by our approach hold biological significance, affirming not only its excellent clustering performance but also its biological interpretability.

Clustering scRNA-seq data with the cross-view collaborative information fusion strategy.

Single-cell RNA sequencing (scRNA-seq) technology has revolutionized biological research by enabling high-throughput, cellular-resolution gene expression profiling. A critical step in scRNA-seq data analysis is cell clustering, which supports downstream analyses. However, the high-dimensional and sparse nature of scRNA-seq data poses significant challenges to existing clustering methods. Furthermore, integrating gene expression information with potential cell structure data remains largely unexplored. Here, we present scCFIB, a novel information bottleneck (IB)-based clustering algorithm that leverages the power of IB for efficient processing of high-dimensional sparse data and incorporates a cross-view fusion strategy to achieve robust cell clustering. scCFIB constructs a multi-feature space by establishing two distinct views from the original features. We then formulate the cell clustering problem as a target loss function within the IB framework, employing a collaborative information fusion strategy. To further optimize scCFIB's performance, we introduce a novel sequential optimization approach through an iterative process. Benchmarking against established methods on diverse scRNA-seq datasets demonstrates that scCFIB achieves superior performance in scRNA-seq data clustering tasks. Availability: the source code is publicly available on GitHub: https://github.com/weixiaojiao/scCFIB.

Auto-weighted Graph Reconstruction for efficient ensemble clustering

Ensemble clustering (EC) is an important field in data clustering, which aims to combine multiple base clustering results of a given dataset into a single consensus result. Previous studies have shown that one of the effective methods is to convert the base clustering results into a graph partitioning problem. Existing spectral ensemble clustering methods typically obtain a spectral embedding matrix for the graph and then perform k-means, separating the graph construction and clustering tasks into two distinct stages. To address these challenges, we propose a novel and efficient ensemble clustering method named Auto-weighted Graph Reconstruction for efficient ensemble clustering. This method integrates weighted base clusters, consensus distance graph construction, and clustering into a unified framework. A consensus distance graph is constructed, from which a stable and discrete label matrix is derived using spectral clustering. An effective optimization algorithm is then employed to address the resulting problem. Finally, the experimental results on eight real-world datasets verify the effectiveness and superiority of the proposed method.

Uncovering student profiles. An explainable cluster analysis approach to PISA 2022

Educational data mining (EDM) applied to the wealth of data generated from international large-scale assessments (ILSAs) shows potential for identifying successful educational initiatives. Despite limited research on clustering methods in ILSAs, leveraging these methods to uncover student profiles can help decision-making in designing tailored programs. This study aims to identify and characterize 15-year-old student profiles using PISA 2022 data and reveal insights into the relationship between these profiles and factors such as ICT availability and use, gender, academic performance, and educational expectations. We analyzed PISA 2022 Spanish student data (n = 30,800) with a selection of 74 contextual variables, applying an end-to-end explainable cluster analysis methodology that integrates different machine learning (ML) and explainable artificial intelligence (XAI) techniques. This methodology covered data pre-processing, dimensionality reduction, clustering, and classification to ensure data quality and result explainability. We obtained 16 derived variables, 7 student clusters, and an optimal XGBoost classifier with a global accuracy of 0.8643. Using local and global SHAP values, we interpreted clusters, finding that socio-economic status and ICT availability and use at home are the most important factors differentiating student profiles. Our findings suggest the need to emphasize (i) proper ICT accessibility and use, as well as student support networks to improve academic performance, (ii) gender-specific well-being programs, and (iii) the encouragement of educational expectations tailored to students’ gender and their exposure to higher education. These results pave the way for personalized academic policies and programs through ML-based tools for uncovering student profiles.

Comprehensive geophysical, geotechnical, and geochemical assessments of an offshore landfill in Singapore

Comprehensive site investigation techniques, including Electrical Resistivity Tomography (ERT), Induced Polarization (IP), Multichannel Analysis of Surface Waves (MASW), and Microtremor Array Method (MAM), were integrated with geotechnical and geochemical tests of retrieved waste samples from Singapore’s operational offshore landfill. The properties of landfill wastes vary widely, including shear-wave velocities 127–248 m/s, densities 1.2–2.1 Mg/m3, resistivity 3.0–25.3 Ω∙m, and chargeability 48–82 mV/V. The natural clay layer underneath was clearly delineated and effectively mitigated leachate leakage. K-means clustering of the geophysical data facilitates precise mapping of waste distribution and quantities of recoverable metals based on quantitative criteria. This study illustrates a thorough case study adopting the new site investigation and characterization paradigm for an offshore landfill, which provides insights into leachate leakage detection and evaluations of landfill mining and resource recovery.

Developmental Trajectories of Electric Vehicle Research in a Circular Economy: Main Path Analysis

This study explored the development history and future trends of academic research on electric vehicles (EVs) in a circular economy. We collected 4127 articles on circular economy and EVs from the Web of Science database, and main path analysis indicated that academic research in the field of EVs in a circular economy has covered the following topics in chronological order: EVs as a power resource; vehicle-to-grid (V2G) technology; renewable energy and energy storage grids; smart grid and charging station optimization; and sustainable development of energy, water, and environmental systems. Through cluster analysis and data mining, we identified the following main research topics in the aforementioned field: recycling and reuse of EV batteries, charging stations and energy management, V2G systems and renewable energy, power frequency control systems, dynamic economic emissions, and energy management. Finally, data mining and statistical analysis revealed the following emerging research topics in this field from 2020 to 2023: microgrids, deep learning, loop supply chain, blockchain, and automatic generation control. Various achievements have been attained in research on EVs in a circular economy; however, challenges related to aspects such as sustainable battery recycling charging infrastructure and renewable energy integration remain.

Adaptive Neighbors Graph Learning for Large-Scale Data Clustering using Vector Quantization and Self-Regularization

In traditional adaptive neighbors graph learning (ANGL)-based clustering, the time complexity is more than O(n2), where n is the number of data points, which is not scalable for large-scale data problems in real applications. Subsequently, ANGL adds a balance regularization to its objective function to avoid the sparse over-fitting problem in the learned similarity graph matrix. Still, the regularization may leads to many weak connections between data points in different clusters. To address these problems, we propose a new fast clustering method, namely, Adaptive Neighbors Graph Learning for Large-Scale Data Clustering using Vector Quantization and Self-Regularization (ANGL-LDC), to perform vector quantization (VQ) on original data and feed the obtained VQ data as the input in the n×n similarity graph matrix learning. Hence, the n×n similarity graph matrix learning problem is simplified to weighted m×m(m≪n) graph learning problem, where m is the number of distinct points and weight is the duplicate times of distinct points in VQ data. Consequently, the time complexity of ANGL-LDC is much lower than that of ANGL. At the same time, we propose a new ANGL objective function with a graph connection self-regularization mechanism, where the ANGL-LDC objective function will get an infinity value if the value of one graph connection is equal to 1. Therefore, ANGL-LDC naturally avoids obtaining the sparse over-fitting problem since we need to minimize the value of ANGL-LDC’s objective function. Experimental results on synthetic and real-world datasets demonstrate the scalability and effectiveness of ANGL-LDC.

Diagnosis of high‐speed railway ballastless track arching based on unsupervised learning framework

AbstractVehicle‐mounted detection methods have been widely applied in the maintenance of high‐speed railways (HSRs), providing feasibility for diagnosing ballastless track arching. However, applying detection data faces several key limitations: (1) The threshold mostly requires manual setting, making recognition accuracy highly subjective; (2) the extensive workload of manual inspections makes it challenging to label detection data, hindering the application of supervised learning approaches. To address these problems, this paper utilizes the longitudinal level irregularity data obtained from vehicle‐mounted detection, employing the concept of unsupervised learning for dimensionality reduction, combined with clustering algorithms and minimal label fine‐tuning, to design two frameworks: the fully unsupervised framework (FUF) and the few‐shot fine‐tuned framework (FFF). Experiments on dynamic detection data from a Chinese HSR line were conducted, comparing the performance of data dimensionality reduction, clustering, and classification under different strategy combinations. The results show that the improved variational autoencoder significantly enhances the performance of the encoder in dimensionality reduction, facilitating better feature extraction; the FUF achieves effective clustering outcomes without any labeled samples and its adjusted rand index score exceeded 0.8, showcasing its robustness and applicability in scenarios with no prior annotations; the FFF requires only a small number of labeled samples (labeling ratio of 5%) and achieves excellent performance, with metrics such as accuracy exceeding 0.85, thus greatly reducing the reliance on labeled data. This study offers a novel method for solving engineering issues with limited labeled data, providing an efficient solution for identifying track arching defects and advancing railway infrastructure monitoring.