Unsupervised Ensemble Learning Research Articles

Harnessing Unsupervised Ensemble Learning for Biomedical Applications: A Review of Methods and Advances

Advancements in data availability and computational techniques, including machine learning, have transformed the field of bioinformatics, enabling the robust analysis of complex, high-dimensional, and heterogeneous biomedical data. This paper explores how diverse bioinformatics tasks, including differential expression analysis, network inference, and somatic mutation calling, can be reframed as binary classification tasks, thereby providing a unifying framework for their analysis. Traditional single-method approaches often fail to generalize across datasets due to differences in data distributions, noise levels, and underlying biological contexts. Ensemble learning, particularly unsupervised ensemble approaches, emerges as a compelling solution by integrating predictions from multiple algorithms to leverage their strengths and mitigate weaknesses. This review focuses on the principles and recent advancements in ensemble learning, with a particular emphasis on unsupervised ensemble methods. These approaches demonstrate their ability to address critical challenges in bioinformatics, such as the lack of labeled data and the integration of predictions from algorithms operating on different scales. Overall, this paper highlights the transformative potential of ensemble learning in advancing predictive accuracy, robustness, and interpretability across diverse bioinformatics applications.

Unsupervised Ensemble Learning for Efficient Integration of Pre-trained Polygenic Risk Scores.

The growing availability of pre-trained polygenic risk score (PRS) models has enabled their integration into real-world applications, reducing the need for extensive data labeling, training, and calibration. However, selecting the most suitable PRS model for a specific target population remains challenging, due to issues such as limited transferability, data het-erogeneity, and the scarcity of observed phenotype in real-world settings. Ensemble learning offers a promising avenue to enhance the predictive accuracy of genetic risk assessments, but most existing methods often rely on observed phenotype data or additional genome-wide association studies (GWAS) from the target population to optimize ensemble weights, limiting their utility in real-time implementation. Here, we present the UN supervised en Semble PRS ( UNSemblePRS ), an unsupervised ensemble learning framework, that combines pre-trained PRS models without requiring phenotype data or summaries from the target population. Unlike traditional supervised approaches, UNSemblePRS aggregates models based on prediction concordance across a curated subset of candidate PRS models. We evaluated UNSemblePRS using both continuous and binary traits in the All of Us database, demonstrating its scalability and robust performance across diverse populations. These results underscore UNSemblePRS as an accessible tool for integrating PRS models into real-world contexts, offering broad applicability as the availability of PRS models continues to expand.

Lung image quality assessment and diagnosis using generative autoencoders in unsupervised ensemble learning

The lung is a critical organ for blood gas exchange. Early lung disease detection is often hindered by subtle symptoms. The diagnosis typically requires expert analysis of lung image scans, where both conventional methods and advanced machine learning (ML) and deep learning (DL) techniques are employed for scan segmentation and disease detection. However, it is a challenge to develop reliable computational models, due to the scarcity of high-quality data. With a major emphasis on lung disease classification and segmentation performance, this work presents a novel data quality assessment technique specifically designed for lung image datasets. The proposed pipeline combines ensemble learning, unsupervised learning, and generative autoencoders with attention mechanisms (GAME). By including attentional mechanisms in the generative autoencoders, we improve the tool’s ability to locate and prioritise areas of interest in lung images and extract the right features, which increases the accuracy of our algorithms. The pipeline automates quality checks and reduces the need for high-quality data, while reducing human oversight. Because it’s crucial to validate the robustness of the algorithms in our tool, we tested it using lung scans from the NIH-ChestXray and CheXpert datasets, and obtained IOU scores of 0.88 and 0.86, F1 scores of 0.95 and 0.95, and accuracy scores of 0.96 and 0.95, respectively, which shows they can be used as a classification tool as well as a segmentation tool. Given the challenging conditions in which the early diagnosis of lung disease is made, this is a significant step forward in the development of self-sustaining and accurate disease detection systems. The proposed model is made available to the public and the same is accessible via https://github.com/harshivchandra/LungDataQualityAssessment.

Ranking and combining latent structured predictive scores without labeled data

Combining multiple predictors obtained from distributed data sources to an accurate meta-learner is promising to achieve enhanced performance in lots of prediction problems. As the accuracy of each predictor is usually unknown, integrating the predictors to achieve a better performance is challenging. Conventional ensemble learning methods assess the accuracy of predictors based on extensive labeled data. In practical applications, however, the acquisition of such labeled data can prove to be an arduous task. Furthermore, the predictors under consideration may exhibit high degrees of correlation, particularly when similar data sources or machine learning algorithms were employed during their model training. In response to these challenges, this article introduces a novel Structured Unsupervised Ensemble Learning (SUEL) model to exploit the dependency between a set of predictors with continuous predictive scores, rank the predictors without labeled data and combine them to an ensembled score with weights. Two novel correlation-based decomposition algorithms are further proposed to estimate the SUEL model, constrained quadratic optimization (SUEL.CQO) and matrix-factorization-based (SUEL.MF) approaches. The efficacy of the proposed methods is rigorously assessed through both simulation studies and real-world application of risk genes discovery. The results compellingly demonstrate that the proposed methods can efficiently integrate the dependent predictors to an ensemble model without the need of ground-truth data.

Development of real time ECG monitoring and unsupervised learning classification framework for cardiovascular diagnosis

In this work, a novel meta-heuristic-based feature ranking and classification approach is developed on the real-time ECG data. Initially, the data is captured using AD8232 biopotential sensor and transmitted to Amazon Web Service (AWS) Internet of Things (IoT) core through ESP8266 gateway via Message Queuing and Telemetry Transport (MQTT) protocol. An improved inter quartile range (IIQR) based filtering is applied on each feature measure to find outliers in the data. A probability estimator-based feature ranking method is used to find the key features. With unsupervised clustering the essential key segments are calculated and used as class labels to train the classifier. Finally, the prediction rate of the model on the segmented classes is improved with the use of ensemble learning. The real-time ECG data of three different individuals are used to test the unsupervised ensemble learning model (Support Vector Machine (SVM) + Linear Regression (LR) + ORF (Optimized Random Forest) + (K-Nearest Neighbor (KNN) + Gradient Boost (XGBoost)) and obtains an overall accuracy of 99.63% (Person 1), 99.75% (Person 2) and 99.78% (Person 3) which significantly outperforms the base ensemble classifiers such as, (LR + SVM + XGBoost) with 94.2% accuracy, (LR + KNN + XGBoost) with 95.8% accuracy and (Random Forest (RF) + KNN + XGBoost) with 95.67% accuracy.

Ensemble deep learning of embeddings for clustering multimodal single-cell omics data.

Recent advances in multimodal single-cell omics technologies enable multiple modalities of molecular attributes, such as gene expression, chromatin accessibility, and protein abundance, to be profiled simultaneously at a global level in individual cells. While the increasing availability of multiple data modalities is expected to provide a more accurate clustering and characterization of cells, the development of computational methods that are capable of extracting information embedded across data modalities is still in its infancy. We propose SnapCCESS for clustering cells by integrating data modalities in multimodal single-cell omics data using an unsupervised ensemble deep learning framework. By creating snapshots of embeddings of multimodality using variational autoencoders, SnapCCESS can be coupled with various clustering algorithms for generating consensus clustering of cells. We applied SnapCCESS with several clustering algorithms to various datasets generated from popular multimodal single-cell omics technologies. Our results demonstrate that SnapCCESS is effective and more efficient than conventional ensemble deep learning-based clustering methods and outperforms other state-of-the-art multimodal embedding generation methods in integrating data modalities for clustering cells. The improved clustering of cells from SnapCCESS will pave the way for more accurate characterization of cell identity and types, an essential step for various downstream analyses of multimodal single-cell omics data. SnapCCESS is implemented as a Python package and is freely available from https://github.com/PYangLab/SnapCCESS under the open-source license of GPL-3. The data used in this study are publicly available (see section 'Data availability').

Unsupervised Ensemble Learning Improves Discriminability of Stochastic Neighbor Embedding

The purpose of feature learning is to obtain effective representation of the raw data and then improve the performance of machine learning algorithms such as clustering or classification. Some of the existing feature learning algorithms use discriminant information in the data to improve the representation of data features, but the discrimination of the data feature representation is not enough. In order to further enhance the discrimination, discriminant feature learning based on t-distribution stochastic neighbor embedding guided by pairwise constraints (pcDTSNE) is proposed in this paper. pcDTSNE introduces pairwise constraints by clustering ensemble and uses these pairwise constraints to impose penalties on the objective function, which makes sample points in the mapping space present stronger discrimination. In order to verify the feature learning performance of pcDTSNE, extensive experiments are carried out on several public data sets. The experimental results show that the expression ability of data representation generated by pcDTSNE is further improved.

Examining unsupervised ensemble learning using spectroscopy data of organic compounds.

One solution to the challenge of choosing an appropriate clustering algorithm is to combine different clusterings into a single consensus clusteringresult, known as cluster ensemble (CE). This ensemble learning strategy can provide more robust and stable solutions across different domains and datasets. Unfortunately, not all clusterings in the ensemble contribute to the final data partition. Cluster ensemble selection (CES) aims at selecting a subset from a large library of clustering solutions to form a smaller cluster ensemble that performs as well as or better than the set of all available clustering solutions. In this paper, we investigate four CES methods for the categorization of structurally distinct organic compounds using high-dimensional IR and Raman spectroscopy data. Single quality selection (SQI) forms a subset of the ensemble by selecting the highest quality ensemble members. The Single Quality Selection (SQI) method is used with various quality indices to select subsets by including the highest quality ensemble members. The Bagging method, usually applied in supervised learning,ranksensemble members by calculating the normalized mutual information (NMI) between ensemble members and consensus solutions generated from a randomly sampled subset of the full ensemble. The hierarchical cluster and select method (HCAS-SQI) uses the diversity matrix of ensemble members to select a diverse set of ensemble members with the highest quality. Furthermore, a combining strategy can be used to combine subsets selected using multiple quality indices (HCAS-MQI) for the refinement of clustering solutions in the ensemble. The IR + Raman hybrid ensemble library is created by merging two complementary "views" of the organic compounds. This inherently more diverse library gives the best full ensemble consensus results. Overall, the Bagging method is recommended because it provides the most robust results that are better than or comparable to the full ensemble consensus solutions.

Unsupervised ensemble based deep learning approach for attack detection in IoT network

SummaryThe Internet of Things (IoT) has altered living by controlling devices/things over the Internet. IoT has specified many smart solutions for daily problems, transforming cyber‐physical systems (CPS) and other classical fields into smart regions. Most of the edge devices that make up the Internet of Things have very minimal processing power. To bring down the IoT network, attackers can utilize these devices to conduct a variety of network attacks. In addition, as more and more IoT devices are added, the potential for new and unknown threats grows exponentially. For this reason, an intelligent security framework for IoT networks must be developed that can identify such threats. In this paper, we have developed an unsupervised ensemble learning model that is able to detect new or unknown attacks in an IoT network from an unlabeled dataset. The system‐generated labeled dataset is used to train a deep learning model to detect IoT network attacks. Additionally, the research presents a feature selection mechanism for identifying the most relevant aspects in the dataset for detecting attacks. The study shows that the suggested model is able to identify the unlabeled IoT network datasets and DBN (Deep Belief Network) outperform the other models with a detection accuracy of 97.5% and a false alarm rate of 2.3% when trained using labeled dataset supplied by the proposed approach.

Markov clustering ensemble

Clustering ensemble is an unsupervised ensemble learning method that is very important in machine learning, since it integrates multiple weak base clustering results to produce a strong consistency result. This paper proposes the Markov clustering ensemble (MCE) model to solve the weak stability and robustness of soft clustering ensemble. First, the base clustering algorithms are regarded as new features of the original datasets. Then, the results of the base clustering algorithms are the values of these features, which can break through the framework of consensus cluster ensemble. Second, as the base clustering results are discrete data, the maximum information coefficient is applied to measure their similarity. Accordingly, a graph-based cluster ensemble model can be constructed with row vectors as vertices and the similarity between row vectors as edges. Then, the Markov process can be applied to infer the graph-based cluster ensemble model; therefore, the MCE algorithm is designed according to the inference. To test the performance of the MCE algorithm, clustering algorithms and clustering ensemble algorithms are used to conduct comparative experiments on ten datasets. The experimental results show that the MCE algorithm outperforms the other algorithms in terms of accuracy and purity.

Unsupervised ensemble learning for genome sequencing

Unsupervised ensemble learning refers to methods devised for a particular task that combine data provided by decision learners taking into account their reliability, which is usually inferred from the data. Here, the variant calling step of the next generation sequencing technologies is formulated as an unsupervised ensemble classification problem. A variant calling algorithm based on the expectation-maximization algorithm is further proposed that estimates the maximum-a-posteriori decision among a number of classes larger than the number of different labels provided by the learners. Experimental results with real human DNA sequencing data show that the proposed algorithm is competitive compared to state-of-the-art variant callers as GATK, HTSLIB, and Platypus.

A reinforcement ensemble deep transfer learning network for rolling bearing fault diagnosis with Multi-source domains

Fault diagnosis with transfer learning has achieved great attention. However, existing methods mostly focused on single-source-single-target sceneries. In some cases, there may exist multiple source domains. Therefore, a reinforcement ensemble deep transfer learning network (REDTLN) is proposed for fault diagnosis with multi-source domains. Firstly, various new kernel maximum mean discrepancies (kMMDs) are used to construct multiple deep transfer learning networks (DTLNs) for single-source-single-target domain adaptation. The differences of kernel functions and source domains can help the DTLNs learn diverse transferable features. Secondly, a new unified metric is designed based on kMMD and diversity measures for unsupervised ensemble learning. Finally, using the unified metric as the reward, a reinforcement learning method is firstly explored to generate an effective combination rule for multi-domain-multi-model reinforcement ensemble. The proposed method is verified with experiment datasets, and the results empirically show its effectiveness and superiority compared with other methods.

Segmentation of turbulent computational fluid dynamics simulations with unsupervised ensemble learning

Computer vision and machine learning tools offer an exciting new way for automatically analyzing and categorizing information from complex computer simulations. Here we design an ensemble machine learning framework that can independently and robustly categorize and dissect simulation data output contents of turbulent flow patterns into distinct structure catalogs. The segmentation is performed using an unsupervised clustering algorithm, which segments physical structures by grouping together similar pixels in simulation images. The accuracy and robustness of the resulting segment region boundaries are enhanced by combining information from multiple simultaneously-evaluated clustering operations. The stacking of object segmentation evaluations is performed using image mask combination operations. This statistically-combined ensemble (SCE) of different cluster masks allows us to construct cluster reliability metrics for each pixel and for the associated segments without any prior user input. By comparing the similarity of different cluster occurrences in the ensemble, we can also assess the optimal number of clusters needed to describe the data. Furthermore, by relying on ensemble-averaged spatial segment region boundaries, the SCE method enables reconstruction of more accurate and robust region of interest (ROI) boundaries for the different image data clusters. We apply the SCE algorithm to 2-dimensional simulation data snapshots of magnetically-dominated fully-kinetic turbulent plasma flows where accurate ROI boundaries are needed for geometrical measurements of intermittent flow structures known as current sheets.

340-P: Nocturnal Hypoglycemia Prediction in Hospitalized Patients with Type 1 Diabetes Using Combined Supervised and Unsupervised Ensemble Learning

The Aim: To assess the efficacy of combined supervised and unsupervised ensemble machine learning based on continuous glucose monitoring (CGM) data for short-term prediction of nocturnal hypoglycemia (NH) in hospitalized patients with type 1 diabetes (T1D). Materials and Methods: We analyzed CGM records with a minimal duration of 72 hours obtained from 406 adult patients admitted to tertiary referral hospital. An episode of NH was defined as ≥3 consequent glucose values of <3.9 mmol/L recorded between 0 and 6 am. Depending on predictor sequence length (30 min - 1 h) and horizon (H) of forecasting (5-30 min), we obtained 209-256 and near 4∙104 records with and without NH respectively. Twenty nine metrics, including mean, minimal and maximum glucose, range, gradient, linear, quadratic and cubic trend coefficients, dynamic time wrapping similarity-based indices, and a number of glucose variability indices, were evaluated. The ensemble clustering was used for extraction of representative samples. Random Forest ensemble model was applied for NH prediction. The obtained quality metrics were averaged over 100 random partitions of the dataset on training and testing subsets. Results: The quality metrics did not depend significantly on predictor sequence length; we used 30 min as a baseline. Depending on H, values of sensitivity and specificity varied from 98% and 97% (H=5 min) to 85% and 87% (H=30 min). The false alarm ratio values were very high (>90%) due to the unbalanced nature of data. Minimal glucose, low blood glucose index, and increment in the penultimate value of sequence were the most important predictors assessed with Random Forest algorithm. Conclusions: The results demonstrate the usefulness of ensemble machine learning models based on CGM time series data for prediction of NH events in hospitalized patients with T1D. Additional studies are needed for improving false alarm ratio metrics of forecasting quality. Disclosure V. Berikov: None. J. F. Semenova: None. V. Klimontov: Advisory Panel; Self; Boehringer Ingelheim International GmbH, Sanofi, Research Support; Self; Boehringer Ingelheim International GmbH, Speaker’s Bureau; Self; AstraZeneca, Medtronic. Funding Russian Science Foundation (20-15-00057)

A reduced variance unsupervised ensemble learning algorithm based on modern portfolio theory

Unsupervised ensemble learning or consensus clustering has gained popularity due to its ability to combine multiple clustering solutions into a single solution that is robust and often performs better than the individual ones. There have been several approaches to consensus clustering including voting and weighted voting algorithmic schemes. Although there have been several algorithms for adjusting the weights of a consensus clustering all of them are tuned based on some performance characteristic associated with clustering accuracy. In this paper, we propose a method for incorporating weights by taking into consideration the intra algorithmic variability i.e. algorithms that provide solutions with very different performance upon multiple runs. The methodology is inspired by modern portfolio theory and more specifically from Markowitz model for asset allocation where one is trying to identify the most efficient portfolio through the solution of a convex optimization problem. Here, efficiency is defined as the minimum amount of risk for an expected return. We apply this method to different datasets and compare with respect to performance and robustness. The proposed scheme appears to achieve competitive average performance with very low variability.

R/PY-SUMMA: An R/Python Package for Unsupervised Ensemble Learning for Binary Classification Problems in Bioinformatics.

The increasing availability of complex data in biology and medicine has promoted the use of machine learning in classification tasks to address important problems in translational and fundamental science. Two important obstacles, however, may limit the unraveling of the full potential of machine learning in these fields: the lack of generalization of the resulting models and the limited number of labeled data sets in some applications. To address these important problems, we developed an unsupervised ensemble algorithm called strategy for unsupervised multiple method aggregation (SUMMA). By virtue of being an ensemble method, SUMMA is more robust to generalization than the predictions it combines. By virtue of being unsupervised, SUMMA does not require labeled data. SUMMA receives as input predictions from a diversity of models and estimates their classification performance even when labeled data are unavailable. It then uses these performance estimates to combine these different predictions into an ensemble model. SUMMA can be applied to a variety of binary classification problems in bioinformatics including but not limited to gene network inference, cancer diagnostics, drug response prediction, somatic mutation, and differential expression calling. In this application note, we introduce the R/PY-SUMMA packages, available in R or Python, that implement the SUMMA algorithm.

Ensemble-Based Algorithm for Synchrophasor Data Anomaly Detection

With the advent of phasor measurement units (PMUs), high resolution synchronized phasor measurements enables real time system monitoring and control. PMUs transmit data to local controllers in substations and phasor data concentrators for system wide monitoring and control application in the control center. They provide real-time phasor data for critical power system applications such as remedial action schemes, oscillation detection, and state estimation. The quality of phasor data from PMUs is critical for smart grid applications. Several methods are developed to detect anomalies in time series data, tailored for PMU data analysis. Nevertheless, applying stand alone methods (with fixed parameters) takes great tuning effort, and does not always achieve high accuracy. In this paper, we adopt an unsupervised ensemble learning approach to develop fast, scalable bad data/event detection for PMU data. The ensemble method invokes a set of base detectors to generate anomaly scores of the PMU data, and makes decisions by aggregating the scores from each detector. We develop two algorithms: 1) a learning algorithm that trains the ensemble model and 2) an online algorithm that infers the anomaly scores with the ensemble model over PMU data streams. The proposed method provides flag for data anomalies and triggers further analysis to differentiate between events and bad data. Using both simulated and real-world PMU data, we show that our ensemble model can be efficiently trained, can achieve high accuracy in detecting diversified errors/events, and outperforms its counterparts that use single standalone detection method.

Density-based unsupervised ensemble learning methods for time series forecasting of aggregated or clustered electricity consumption

This paper presents a comparison of the impact of various unsupervised ensemble learning methods on electricity load forecasting. The electricity load from consumers is simply aggregated or optimally clustered to more predictable groups by cluster analysis. The clustering approach consists of efficient preprocessing of data obtained from smart meters by a model-based representation and the K-means method. We have implemented two types of unsupervised ensemble learning methods to investigate the performance of forecasting on clustered or simply aggregated load: bootstrap aggregating based and the newly proposed density-clustering based. Three new bootstrapping methods for time series analysis methods were newly proposed in order to handle the noisy behaviour of time series. The smart meter datasets used in our experiments come from Australia, London, and Ireland, where data from residential consumers were available. The achieved results suggest that for extremely fluctuating and noisy time series the forecasting accuracy improvement through the bagging can be a challenging task. However, our experimental evaluation shows that in most of the cases the density-based unsupervised ensemble learning methods are significantly improving forecasting accuracy of aggregated or clustered electricity load.

Linear discriminant analysis guided by unsupervised ensemble learning

The high dimensionality and sparsity of data often increase the complexity of clustering; these factors occur simultaneously in unsupervised learning. Clustering and linear discriminant analysis (LDA) are methods to reduce the dimensionality and sparsity of data. In this study, the similarity of clustering and LDA are investigated based on their objective functions. Subsequently, their objective functions are integrated, and an LDA guided by an unsupervised ensemble learning (LDA–UEL) model is proposed. To create the proposed model, fuzziness F is designed to measure the confidence of unsupervised learning and the inference of the proposed model is illustrated. Furthermore, a corresponding algorithm for the inference is designed. Finally, extensive experiments are designed, and the results thus obtained demonstrate the effectiveness and high performance of the LDA–UEL model.

Sentiment analysis: An automatic contextual analysis and ensemble clustering approach and comparison

Product reviews are one of the most important resources to determine public sentiment. The existing literature on review sentiment analysis mostly utilizes supervised models, which usually suffer from domain-dependency and require expensive manual labelling effort to provide training data. This article addresses these issues by describing a completely automatic and unsupervised approach to sentiment analysis. The method consists of two phases, which are contextual analysis and unsupervised ensemble learning. In the implementation of both phases, a sentiment lexicon, SentiWordNet, is deployed. Using effective contextual procedures and modifying the base learning component (the k-means algorithm) results in developing a successful approach to sentiment analysis which can overcome the domain-dependency and the labelling cost problems. The results show that the proposed nonrandom initialization of k-means yields a significant improvement compared to other algorithms. In terms of accuracy and performance, the proposed method is effective compared to supervised and unsupervised approaches. We also introduce new sentiment analysis problems relating to Australian airlines and home builders which could be potential benchmark problems in the sentiment analysis field. Our experiments on datasets from different domains show that contextual analysis and the ensemble phases improve the clustering performance in term of accuracy, stability and generalizability.