Conventional Ensemble Methods Research Articles

A Structured Bipartite Graph Learning method for ensemble clustering

Given a set of base clustering results, conventional bipartite graph-based ensemble clustering methods typically require computing a sample-cluster similarity matrix from each base clustering result. These matrices are then either concatenated or averaged to form a bipartite weight matrix, which is used to create a bipartite graph. Graph-based partition techniques are subsequently applied to this graph to obtain the final clustering result. However, these methods often suffer from unreliable base clustering results, making it challenging to identify a clear cluster structure due to the variations in cluster structures across the base results. In this paper, we propose a novel Structured Bipartite Graph Learning (SBGL) method. Our approach begins by computing a sample-cluster similarity matrix from each base clustering result and constructing a base bipartite graph from each of these matrices. We assume these base bipartite graphs contain a set of latent clusters and project them into a set of sample-latent-cluster bipartite graphs. These new graphs are then ensembled into a bipartite graph with a distinct cluster structure, from which the final set of clusters is derived. Our method allows for different numbers of clusters across base clusterings, leading to improved performance. Experimental results on both synthetic and real-world datasets demonstrate the superior performance of our new method.

Ranking and Combining Latent Structured Predictive Scores without Labeled Data

Combining multiple predictors obtained from distributed data sources to an accurate meta-learner is promising to achieve enhanced performance in lots of prediction problems. As the accuracy of each predictor is usually unknown, integrating the predictors to achieve better performance is challenging. Conventional ensemble learning methods assess the accuracy of predictors based on extensive labeled data. In practical applications, however, the acquisition of such labeled data can prove to be an arduous task. Furthermore, the predictors under consideration may exhibit high degrees of correlation, particularly when similar data sources or machine learning algorithms were employed during their model training. In response to these challenges, this paper introduces a novel structured unsupervised ensemble learning model (SUEL) to exploit the dependency between a set of predictors with continuous predictive scores, rank the predictors without labeled data and combine them to an ensembled score with weights. Two novel correlation-based decomposition algorithms are further proposed to estimate the SUEL model, constrained quadratic optimization (SUEL.CQO) and matrix-factorization-based (SUEL.MF) approaches. The efficacy of the proposed methods is rigorously assessed through both simulation studies and real-world application of risk genes discovery. The results compellingly demonstrate that the proposed methods can efficiently integrate the dependent predictors to an ensemble model without the need of ground truth data.

Active Clustering Ensemble With Self-Paced Learning.

A clustering ensemble provides an elegant framework to learn a consensus result from multiple prespecified clustering partitions. Though conventional clustering ensemble methods achieve promising performance in various applications, we observe that they may usually be misled by some unreliable instances due to the absence of labels. To tackle this issue, we propose a novel active clustering ensemble method, which selects the uncertain or unreliable data for querying the annotations in the process of the ensemble. To fulfill this idea, we seamlessly integrate the active clustering ensemble method into a self-paced learning framework, leading to a novel self-paced active clustering ensemble (SPACE) method. The proposed SPACE can jointly select unreliable data to label via automatically evaluating their difficulty and applying easy data to ensemble the clusterings. In this way, these two tasks can be boosted by each other, with the aim to achieve better clustering performance. The experimental results on benchmark datasets demonstrate the significant effectiveness of our method. The codes of this article are released in https://Doctor-Nobody.github.io/codes/space.zip.

Arctic tern-optimized weighted feature regression system for predicting bridge scour depth

This paper presents a pioneering artificial intelligence (AI) solution – the Arctic Tern-Optimized Weighted Feature Least Squares Support Vector Regression (ATO-WFLSSVR) system to aid civil engineers in accurately predicting scour depth at bridges. This prediction system amalgamates the strengths of hybrid models by uniting a metaheuristic optimization algorithm with weighted features and least squares support vector regression (WFLSSVR). The metaheuristic algorithm concurrently optimizes all hyperparameters of constituent WFLSSVR models, resulting in a highly effective system. Validation involves a comprehensive assessment using two case studies, which include datasets of scour depths across various complexities and pier foundation types. Comparative analyses against single AI models, conventional ensemble models, hybrid techniques, and empirical methods demonstrate that ATO-WFLSSVR's reliability outperforms others in performance evaluation metrics. Specifically, for the field dataset, ATO-WFLSSVR achieves MAPE and R values of 20.92% and 0.9435, respectively, and for scour depth data at complex pier foundations, it records MAPE and R values of 6.49% and 0.9384, respectively. The automated predictive analytics underscore the robustness, efficiency, and stability of ATO-WFLSSVR compared to existing methods. This study's notable contributions include the development of an innovative optimization algorithm named Arctic Terns Optimizer (ATO), proficiency in solving high-dimensional optimization problems, and the creation of a user-friendly graphical interface system, a promising tool for civil engineers to estimate scour depth at bridges. Further testing and evaluation of ATO-WFLSSVR across diverse datasets encompassing more complex scenarios are recommended. The data and source code for this study are currently accessible at https://www.researchgate.net/profile/Jui-Sheng-Chou/publications.

Bagging Algorithm Based on Possibilistic Data Interpolation for Brain-Computer Interface

Recently, brain-computer interfaces (BCIs) and brain-machine interfaces have garnered the attention of researchers. Based on connections with external devices, external computers and machines can be controlled by brain signals measured via near-infrared spectroscopy (NIRS) or electroencephalograph devices. Herein, we propose a novel bagging algorithm that generates interpolation data around misclassified data using a possibilistic function, to be applied to BCIs. In contrast to AdaBoost, which is a conventional ensemble learning method that increases the weight of misclassified data to incorporate them with high probability to the next datasets, we generate interpolation data using a membership function centered on misclassified data and incorporate them into the next datasets simultaneously. The interpolated data are known as virtual data herein. By adding the virtual data to the training data, the volume of the training data becomes sufficient for adjusting the discriminate boundary more accurately. Because the membership function is defined as a possibility distribution, this method is named the bagging algorithm based on the possibility distribution. Herein, we formulate a bagging-type ensemble learning based on the possibility distribution and discuss the usefulness of the proposed method for solving simple calculations using NIRS data.

Stacking and ridge regression-based spectral ensemble preprocessing method and its application in near-infrared spectral analysis

Spectral preprocessing techniques can, to a certain extent, eliminate irrelevant information, such as current noise and stray light from spectral data, thereby enhancing the performance of prediction models. However, current preprocessing techniques mostly attempt to find the best single preprocessing method or their combination, overlooking the complementary information among different preprocessing methods. These preprocessing techniques fail to maximize the utilization of useful information in spectral data and restrict the performance of prediction models. This study proposed a spectral ensemble preprocessing method based on the rapidly developing ensemble learning methods in recent years and the ridge regression (RR) model, named stacking preprocessing ridge regression (SPRR), to address the aforementioned issues. Different from conventional ensemble learning methods, the proposed SPRR method applied multiple different preprocessing techniques to the original spectral data, generating multiple preprocessed datasets. These datasets were then individually inputted into RR base models for training. Ultimately, RR still served as the meta-model, integrating the output results of each RR base model through stacking. This approach not only produced diversity in base models but also achieved higher accuracy and lower computational complexity by using a single type of base model. On the apple spectral dataset collected by our team, correlation analysis showed significant complementary information among the data produced by different preprocessing techniques. This provided robust theoretical support for the proposed SPRR method. By introducing the currently popular averaging ensemble preprocessing method in a comparative experiment, the results of applying the proposed SPRR method to six datasets (apple, meat, wheat, olive oil, tablet, and corn) demonstrated that compared to the single preprocessing method and averaging ensemble preprocessing method, SPRR yielded the best accuracy and reliability for all six datasets. Furthermore, under the same conditions of the training and test datasets, the proposed SPRR method demonstrated better performance than the four commonly used ensemble preprocessing methods.

Ensemble-in-One: Ensemble Learning within Random Gated Networks for Enhanced Adversarial Robustness

Adversarial attacks have threatened modern deep learning systems by crafting adversarial examples with small perturbations to fool the convolutional neural networks (CNNs). To alleviate that, ensemble training methods are proposed to facilitate better adversarial robustness by diversifying the vulnerabilities among the sub-models, simultaneously maintaining comparable natural accuracy as standard training. Previous practices also demonstrate that enlarging the ensemble can improve the robustness. However, conventional ensemble methods are with poor scalability, owing to the rapidly increasing complexity when containing more sub-models in the ensemble. Moreover, it is usually infeasible to train or deploy an ensemble with substantial sub-models, owing to the tight hardware resource budget and latency requirement. In this work, we propose Ensemble-in-One (EIO), a simple but effective method to efficiently enlarge the ensemble with a random gated network (RGN). EIO augments a candidate model by replacing the parametrized layers with multi-path random gated blocks (RGBs) to construct an RGN. The scalability is significantly boosted because the number of paths exponentially increases with the RGN depth. Then by learning from the vulnerabilities of numerous other paths within the RGN, every path obtains better adversarial robustness. Our experiments demonstrate that EIO consistently outperforms previous ensemble training methods with smaller computational overheads, simultaneously achieving better accuracy-robustness trade-offs than adversarial training methods under black-box transfer attacks. Code is available at https://github.com/cai-y13/Ensemble-in-One.git

Ensemble deep learning of embeddings for clustering multimodal single-cell omics data.

Recent advances in multimodal single-cell omics technologies enable multiple modalities of molecular attributes, such as gene expression, chromatin accessibility, and protein abundance, to be profiled simultaneously at a global level in individual cells. While the increasing availability of multiple data modalities is expected to provide a more accurate clustering and characterization of cells, the development of computational methods that are capable of extracting information embedded across data modalities is still in its infancy. We propose SnapCCESS for clustering cells by integrating data modalities in multimodal single-cell omics data using an unsupervised ensemble deep learning framework. By creating snapshots of embeddings of multimodality using variational autoencoders, SnapCCESS can be coupled with various clustering algorithms for generating consensus clustering of cells. We applied SnapCCESS with several clustering algorithms to various datasets generated from popular multimodal single-cell omics technologies. Our results demonstrate that SnapCCESS is effective and more efficient than conventional ensemble deep learning-based clustering methods and outperforms other state-of-the-art multimodal embedding generation methods in integrating data modalities for clustering cells. The improved clustering of cells from SnapCCESS will pave the way for more accurate characterization of cell identity and types, an essential step for various downstream analyses of multimodal single-cell omics data. SnapCCESS is implemented as a Python package and is freely available from https://github.com/PYangLab/SnapCCESS under the open-source license of GPL-3. The data used in this study are publicly available (see section 'Data availability').

Easy Ensemble: Simple Deep Ensemble Learning for Sensor-Based Human Activity Recognition

Sensor-based human activity recognition (HAR) is a paramount technology in the Internet of Things services. HAR using representation learning, which automatically learns a feature representation from raw data, is the mainstream method because it is difficult to interpret relevant information from raw sensor data to design meaningful features. Ensemble learning is a robust approach to improve generalization performance; however, deep ensemble learning requires various procedures, such as data partitioning and training multiple models, which are time-consuming and computationally expensive. In this study, we propose an easy ensemble (EE) for HAR, which enables the easy implementation of deep ensemble learning in a single model. In addition, we propose various techniques (input variationer, stepwise ensemble, and channel shuffle) for the EE. Experiments on a benchmark data set for HAR demonstrated the effectiveness of EE and various techniques and their characteristics compared with conventional ensemble learning methods.

An efficient stacking based NSGA-II approach for predicting type 2 diabetes

<span lang="EN-US">Diabetes has been acknowledged as a well-known risk factor for renal and cardiovascular disorders, cardiac stroke and leads to a lot of morbidity in the society. Reducing the disease prevalence in the community will provide substantial benefits to the community and lessen the burden on the public health care system. So far, to detect the disease innumerable data mining approaches have been used. These days, incorporation of machine learning is conducive for the construction of a faster, accurate and reliable model. Several methods based on ensemble classifiers are being used by researchers for the prediction of diabetes. The proposed framework of prediction of diabetes mellitus employs an approach called stacking based ensemble using non-dominated sorting genetic algorithm (NSGA-II) scheme. The primary objective of the work is to develop a more accurate prediction model that reduces the lead time i.e., the time between the onset of diabetes and clinical diagnosis. Proposed NSGA-II stacking approach has been compared with Boosting, Bagging, Random Forest and Random Subspace method. The performance of Stacking approach has eclipsed the other conventional ensemble methods. It has been noted that k-nearest neighbors (KNN) gives a better performance over decision tree as a stacking combiner.</span>

Understanding the nanoscale interactions of surface plasmon-mediated semiconductor surfaces with water and light for renewable energy harvesting and conversion

Photocatalysts integrated with surface plasmon metals serve as an interesting platform for fully understanding how energy harvesting and storage are managed through physical and chemical enhancement processes. Nanoelectrode with a single plasmon metal nanoparticle (NP) provides unique features capable of resolving complex enhancement processes which otherwise cannot be achievable through conventional ensemble averaging methods. Preliminary theoretical results are presented here to illustrate both electrochemical and optical field enhancement characteristics of a single NP when interacting with a semiconductor surface. Potential methodologies to construct single plasmon nanoelectrodes and challenges are discussed in this opinion.

A novel feature susceptibility approach for a PEMFC control system based on an improved XGBoost-Boruta algorithm

Data-driven modelling methods are being developed in the quest to achieve more accurate performance prediction of protons exchange membrane fuel cell (PEMFC) systems in response to their complicated physicochemical phenomena. However, there is little research in this field detailing the pre-processing and selection of balance of plants (BOP) features for the input layer of system performance prediction at different current densities. Furthermore, most of the previous research applies neural networks based on simulation data rather than real-time bench or vehicle operation datasets which leads to low robustness and unreliable practical results. This paper details the application of a novel algorithm denoted XGBoost-Boruta, which utilises the combination of an ensemble learning approach and a wrapping approach, to improve the robustness of feature selection and to increase the accuracy and robustness of PEMFC system performance prediction. By introduction of the Z score and shadow features to eliminate the randomness of conventional ensemble learning methods, seven key controllable BOP variables of the hydrogen anode, air cathode and cooling subsystems are selected as the original input variables to determine their dependency on the stack voltage. Two case studies are presented for verification and validation of the proposed algorithm based on the real-time dataset of bench experimental data and data obtained from heavy truck operation at current densities ranging from 100 to 1500 mA/cm2. The feature selection strategy, based on the proposed XGBoost-Boruta algorithm, largely decreases the RMSE by 23.8% and 14.1% and the R2 increases by 0.06 and 0.04 of both the bench experimental and the heavy truck validation datasets respectively.

Evolutionary bagging for ensemble learning

Ensemble learning has gained success in machine learning with major advantages over other learning methods. Bagging is a prominent ensemble learning method that creates subgroups of data, known as bags, that are trained by individual machine learning methods such as decision trees. Random forest is a prominent example of bagging with additional features in the learning process. Evolutionary algorithms have been prominent for optimisation problems and also been used for machine learning. Evolutionary algorithms are gradient-free methods that work with a population of candidate solutions that maintain diversity for creating new solutions. In conventional bagged ensemble learning, the bags are created once and the content, in terms of the training examples, are fixed over the learning process. In our paper, we propose evolutionary bagged ensemble learning, where we utilise evolutionary algorithms to evolve the content of the bags in order to iteratively enhance the ensemble by providing diversity in the bags. The results show that our evolutionary ensemble bagging method outperforms conventional ensemble methods (bagging and random forests) for several benchmark datasets under certain constraints. We find that evolutionary bagging can inherently sustain a diverse set of bags without reduction in performance accuracy.

Acute coronary syndrome risk prediction by ensemble‐MLPs

Acute coronary syndrome (ACS) is a serious cardiovascular disease. The ACS risk prediction model is of great significance during the hospitalisation of ACS patients. However, traditional machine learning methods are not effective in predicting risk events in the ACS treatment process because of sample imbalance and noise. In this letter, the multilayer perceptron (MLP) and ensemble method are combined and then ensemble-MLPs are proposed, which has made two innovations: 1) increase the diversity of the base MLP classifier at the data level, structure level, and parameter level and 2) improve the ensemble performance by proposing a new ensemble method using f1-score weighted average. Experiments have shown that the proposed method outperforms conventional ensemble MLP method and other traditional machine learning methods on the task of predicting risk events in the ACS treatment process.

A novel bio-inspired hybrid multi-filter wrapper gene selection method with ensemble classifier for microarray data.

Microarray technology is known as one of the most important tools for collecting DNA expression data. This technology allows researchers to investigate and examine types of diseases and their origins. However, microarray data are often associated with a small sample size, a significant number of genes, imbalanced data, etc., making classification models inefficient. Thus, a new hybrid solution based on a multi-filter and adaptive chaotic multi-objective forest optimization algorithm (AC-MOFOA) is presented to solve the gene selection problem and construct the Ensemble Classifier. In the proposed solution, a multi-filter model (i.e., ensemble filter) is proposed as preprocessing step to reduce the dataset's dimensions, using a combination of five filter methods to remove redundant and irrelevant genes. Accordingly, the results of the five filter methods are combined using a voting-based function. Additionally, the results of the proposed multi-filter indicate that it has good capability in reducing the gene subset size and selecting relevant genes. Then, an AC-MOFOA based on the concepts of non-dominated sorting, crowding distance, chaos theory, and adaptive operators is presented. AC-MOFOA as a wrapper method aimed at reducing dataset dimensions, optimizing KELM, and increasing the accuracy of the classification, simultaneously. Next, in this method, an ensemble classifier model is presented using AC-MOFOA results to classify microarray data. The performance of the proposed algorithm was evaluated on nine public microarray datasets, and its results were compared in terms of the number of selected genes, classification efficiency, execution time, time complexity, hypervolume indicator, and spacing metric with five hybrid multi-objective methods, and three hybrid single-objective methods. According to the results, the proposed hybrid method could increase the accuracy of the KELM in most datasets by reducing the dataset's dimensions and achieve similar or superior performance compared to other multi-objective methods. Furthermore, the proposed Ensemble Classifier model could provide better classification accuracy and generalizability in the seven of nine microarray datasets compared to conventional ensemble methods. Moreover, the comparison results of the Ensemble Classifier model with three state-of-the-art ensemble generation methods indicate its competitive performance in which the proposed ensemble model achieved better results in the five of nine datasets.Supplementary InformationThe online version contains supplementary material available at 10.1007/s00521-021-06459-9.

Single Molecule Fluorescence Energy Transfer Study of Ribosome Protein Synthesis

Single molecule fluorescence energy transfer is a method that tracks the tRNA dynamics during ribosomal protein synthesis. By tracking individual ribosomes, inhomogeneous populations are identified, which shed light on mechanisms. This method can be used to track biological conformational changes in general to reveal dynamic-function relationships in many other complexed biosystems. Single molecule methods can observe non-rate limiting steps and low-populated key intermediates, which are not accessible by conventional ensemble methods due to the average effect.

Ensemble Method of Convolutional Neural Networks with Directed Acyclic Graph Using Dermoscopic Images: Melanoma Detection Application.

The early detection of melanoma is the most efficient way to reduce its mortality rate. Dermatologists achieve this task with the help of dermoscopy, a non-invasive tool allowing the visualization of patterns of skin lesions. Computer-aided diagnosis (CAD) systems developed on dermoscopic images are needed to assist dermatologists. These systems rely mainly on multiclass classification approaches. However, the multiclass classification of skin lesions by an automated system remains a challenging task. Decomposing a multiclass problem into a binary problem can reduce the complexity of the initial problem and increase the overall performance. This paper proposes a CAD system to classify dermoscopic images into three diagnosis classes: melanoma, nevi, and seborrheic keratosis. We introduce a novel ensemble scheme of convolutional neural networks (CNNs), inspired by decomposition and ensemble methods, to improve the performance of the CAD system. Unlike conventional ensemble methods, we use a directed acyclic graph to aggregate binary CNNs for the melanoma detection task. On the ISIC 2018 public dataset, our method achieves the best balanced accuracy (76.6%) among multiclass CNNs, an ensemble of multiclass CNNs with classical aggregation methods, and other related works. Our results reveal that the directed acyclic graph is a meaningful approach to develop a reliable and robust automated diagnosis system for the multiclass classification of dermoscopic images.

Tri-level Robust Clustering Ensemble with Multiple Graph Learning

Clustering ensemble generates a consensus clustering result by integrating multiple weak base clustering results. Although it often provides more robust results compared with single clustering methods, it still suffers from the robustness problem if it does not treat the unreliability of base results carefully. Conventional clustering ensemble methods often use all data for ensemble, while ignoring the noises or outliers on the data. Although some robust clustering ensemble methods are proposed, which extract the noises on the data, they still characterize the robustness in a single level, and thus they cannot comprehensively handle the complicated robustness problem. In this paper, to address this problem, we propose a novel Tri-level Robust Clustering Ensemble (TRCE) method by transforming the clustering ensemble problem to a multiple graph learning problem. Just as its name implies, the proposed method tackles robustness problem in three levels: base clustering level, graph level and instance level. By considering the robustness problem in a more comprehensive way, the proposed TRCE can achieve a more robust consensus clustering result. Experimental results on benchmark datasets also demonstrate it. Our method often outperforms other state-of-the-art clustering ensemble methods. Even compared with the robust ensemble methods, ours also performs better.

Scanning Electrochemical Cell Microscope Study of Individual H2 Gas Bubble Nucleation on Platinum: Effect of Surfactants

Adsorption of gas bubbles on the electrode surface is one of important issues responsible for the reduced efficiency of water electrolysis. Understanding the bubble phenomena can benefit the management of gas bubbles. In this study, we carried out a single-entity electrochemical study of individual gas bubbles on the Pt electrode using scanning electrochemical cell microscopy. Different from conventional ensemble methods, the bubble nucleation behaviour and the effect of surfactants (perfluorooctanoic acid (PFOA), hexadecyl trimethyl ammonium bromide (CTAB) and siloxane defoamer) on the electrode surface were investigated at microscale resolution. Statistical analysis of individual nucleation event showed the stochastic characteristic and the addition of surfactant could significantly improve the stability of gas bubbles especially those formed in small nanopipets. A correlation between the critical supersaturation (or Faradic current) for bubble nucleation and surface tension with was observed, which was consistent with classic nucleation theory. However, chronoamperometry study showed that the water electrolysis efficiency after surfactant addition was not significantly improved. This study provided unique insights into understanding of nucleation event at single-entity.

Metaheuristics‐optimized ensemble system for predicting mechanical strength of reinforced concrete materials

This paper develops a novel artificial intelligence (AI)-based approach, called the metaheuristics-optimized ensemble system (MOES), to assist civil engineers significantly in achieving accurate estimations of the mechanical strength of reinforced concrete (RC) materials. MOES integrates the advantages of hybrid and ensemble models by combining a metaheuristic optimization algorithm and efficient AI models. The metaheuristic algorithm finds the optimal hyperparameters of individual AI techniques and simultaneously adjusts their weights to yield the best optimized-weight-ensemble model. Particularly, the developed MOES was established by integrating the forensic-based investigation optimization algorithm, the radial basis function neural network, and the least squares support vector regression. Four case studies of predicting structural mechanics of RC beams were performed to evaluate the performance of MOES and compare it to those of other single AI models, conventional ensemble models, hybrid models, and empirical methods. The analytical results of cross-validation reveal that MOES was the most reliable approach, achieving the best values of all performance evaluation indexes. The automated predictive analytics revealed the robustness, efficiency, and stability of MOES. Thus, the proposed approach is a highly promising tool for predicting the structural mechanics of RC beams. The success of MOES in estimating the mechanical strength of RC beams has redefined the way of optimizing an ensemble AI model, which is the primary contribution of this research to the relevant body of knowledge.