Ensemble Training Research Articles

‘The Gauntlet’; enacting social transformation through the facilitation of community in ensemble actor training

One of the fundamental aspects of actor training is the creation of the ensemble as a ‘community’. In an intersectional feminist actor training, this community is built for the purpose of social transformation; defined by the dissolution of structures designed to suppress; racism, sexism, heterosexist ‘norms’ and other oppressions that marginalise and harm those that self-define in ways outside of a white, heterosexual, male ‘norm’. Social transformation seems a lofty, generalised goal based on theoretical constructs, and one that is difficult to measure. How can feminist pedagogues facilitate community creation with actors in training towards social transformation, and how can this be measured? In this article, I will interrogate community through the lens of bell hooks and Audre Lorde's similar definitions of it; as a tacit sense and an entity intentionally built that acknowledges similarity and difference for the purpose of social transformation (2003, xv, 2017a, 14, 18). I will argue for the Gauntlet as a foundational practice for creating community in a feminist pedagogic framework, and as positively impacting rehearsal and performance for ensemble in general. I will detail my facilitation of the Gauntlet in a recent workshop, making reference to the ways in which myself and other practitioners have used and adapted it. I will discuss the further potential of this exercise as a tool to facilitate recognition, connection, and communication as well as for conflict transformation and adaptations for daily use as a shared language to affirm community.

Ensembles of Locally Independent Prediction Models

Ensembles depend on diversity for improved performance. Many ensemble training methods, therefore, attempt to optimize for diversity, which they almost always define in terms of differences in training set predictions. In this paper, however, we demonstrate the diversity of predictions on the training set does not necessarily imply diversity under mild covariate shift, which can harm generalization in practical settings. To address this issue, we introduce a new diversity metric and associated method of training ensembles of models that extrapolate differently on local patches of the data manifold. Across a variety of synthetic and real-world tasks, we find that our method improves generalization and diversity in qualitatively novel ways, especially under data limits and covariate shift.

End-to-end training of CNN ensembles for person re-identification

We propose an end-to-end ensemble method for person re-identification (ReID) to address the problem of overfitting in discriminative models. These models are known to converge easily, but they are biased to the training data in general and may produce a high model variance, which is known as overfitting. The ReID task is more prone to this problem due to the large discrepancy between training and test distributions. To address this problem, our proposed ensemble learning framework produces several diverse and accurate base learners in a single DenseNet. Since most of the costly dense blocks are shared, our method is computationally efficient, which makes it favorable compared to the conventional ensemble models. Experiments on several benchmark datasets demonstrate that our method achieves state-of-the-art results. Noticeable performance improvements, especially on relatively small datasets, indicate that the proposed method deals with the overfitting problem effectively.

A deep learning based medical image segmentation technique in Internet-of-Medical-Things domain

Medical Image Segmentation is the process of automatic or semi-automatic detection of boundaries within a 2D or 3D image in Internet-of-Medical-Things (IoMT) domain. The main difficulty of medical image segmentation is the high variability in medical images. For example, CT images contain a large amount of noise, and complex boundaries. In this paper, we propose an adaptive fully dense(AFD) neural network for CT image segmentation. By adding the horizontal connections in UNet structure, it can extract various features from all layers adaptively. And it use ensemble training for the output to extract more edge information in the multiple rounds training. We have validated our method on two data sets, a natural scene image data set and a liver cancer CT image data set. The experimental results demonstrate that it performs better than state-of-the-art segmentation methods. And our method yields superior segmentation results for CT images with complex boundaries.

Training ensembles of faceted classification models for quantitative stock trading

Forecasting the stock markets is among the most popular research challenges in finance. Several quantitative trading systems based on supervised machine learning approaches have been presented in literature. Recently proposed solutions train classification models on historical stock-related datasets. Training data include a variety of features related to different facets (e.g., stock price trends, exchange volumes, price volatility, news and public mood). To increase the accuracy of the predictions, multiple models are often combined together using ensemble methods. However, understanding which models should be combined together and how to effectively handle features related to different facets within different models are still open research questions. In this paper we investigate the use of ensemble methods to combine faceted classification models for supporting stock trading. To this aim, separate classification models are trained on each subset of features belonging to the same facet. They produce trading signals tailored to a specific facet. Signals are then combined together and filtered to generate a unified, multi-faceted recommendation. The experimental validation, performed on different markets and in different conditions, shows that, in many cases, some of the faceted models perform as good as or better than models trained on a mix of different features. An ensemble of the faceted recommendations makes the generated trading signals more profitable yet robust to draw-down periods.

KalmanTune: A Kalman Filter Based Tuning Method to Make Boosted Ensembles Robust to Class-Label Noise

Boosting has been shown to be a very effective approach to training ensemble classification models. Although they perform very well, boosting algorithms are sensitive to class-label noise (where training data instances are mislabelled). As the level of class-label noise in the training dataset increases, the generalisation performance of ensembles trained using boosting decreases. This paper introduces KalmanTune, a tuning process that can be applied to ensemble models after they have been trained using a boosting algorithm that reduces the impact of class-label noise. KalmanTune frames the tuning of a trained ensemble model as a static state estimation problem that can be addressed using a Kalman filter. This approach exploits the sensor fusion capability of the Kalman filter to reduce the impact of class-label noise on the trained ensemble. This paper describes KalmanTune and an evaluation experiment performed using 34 multi-class datasets with 5 levels of synthetically induced class-label noise that demonstrates that applying KalmanTune after training can improve the performance of ensemble models trained using boosting, especially when training data contains noisy class-labels.

КОМПЬЮТЕРНЫЕ И ИНТЕРНЕТ-ТЕХНОЛОГИИ В ОБУЧЕНИИ ВОКАЛЬНОМУ АНСАМБЛЮ: ВЫЗОВ ВРЕМЕНИ

The relevance of the article lies in the urgent need to introduce modern computer and Internet technologies in the creative educational process. The considered methods and approaches in vocal ensemble training with use of such technologies are designed primarily for the initial stage of work on music. The use of innovative teaching tools indicated in the study allows to make a variety of classes, to conduct them remotely, as well as to optimize the time spent on the classroom load, the share of which is concentrated on artistic work with this approach. Introduction of MOODLE distance learning platform and IGTV Instagram technologies into creative educational process allows to record offline classes of long duration. Yandex, Mail, and Google cloud services provide disk space for saving tasks and visual materials that can be used by an unlimited number of students at the same time. Dorico or Sibelius music editors help to eliminate rhythmic problems while working on a piece of music, which can correctly sound the musical text, Cubase and Reaper programs indicate inaccuracies of intonation in voice recording. Video conferences in social networks Vkontakte, Facebook, WhatsApp, Android, IOs, Scype, Zoom, etc. are also useful, thanks to which it is possible to conduct some types of online classes with the participation of a teacher, concertmaster and students. It is important that these technologies are available to healthy students as well as students with disabilities.

Cellular automata as convolutional neural networks

Deep-learning techniques have recently demonstrated broad success in predicting complex dynamical systems ranging from turbulence to human speech, motivating broader questions about how neural networks encode and represent dynamical rules. We explore this problem in the context of cellular automata (CA), simple dynamical systems that are intrinsically discrete and thus difficult to analyze using standard tools from dynamical systems theory. We show that any CA may readily be represented using a convolutional neural network with a network-in-network architecture. This motivates the development of a general convolutional multilayer perceptron architecture, which we find can learn the dynamical rules for arbitrary CA when given videos of the CA as training data. In the limit of large network widths, we find that training dynamics are nearly identical across replicates, and that common patterns emerge in the structure of networks trained on different CA rulesets. We train ensembles of networks on randomly sampled CA, and we probe how the trained networks internally represent the CA rules using an information-theoretic technique based on distributions of layer activation patterns. We find that CA with simpler rule tables produce trained networks with hierarchical structure and layer specialization, while more complex CA produce shallower representations-illustrating how the underlying complexity of the CA's rules influences the specificity of these internal representations. Our results suggest how the entropy of a physical process can affect its representation when learned by neural networks.

MEAL: Multi-Model Ensemble via Adversarial Learning

Often the best performing deep neural models are ensembles of multiple base-level networks. Unfortunately, the space required to store these many networks, and the time required to execute them at test-time, prohibits their use in applications where test sets are large (e.g., ImageNet). In this paper, we present a method for compressing large, complex trained ensembles into a single network, where knowledge from a variety of trained deep neural networks (DNNs) is distilled and transferred to a single DNN. In order to distill diverse knowledge from different trained (teacher) models, we propose to use adversarial-based learning strategy where we define a block-wise training loss to guide and optimize the predefined student network to recover the knowledge in teacher models, and to promote the discriminator network to distinguish teacher vs. student features simultaneously. The proposed ensemble method (MEAL) of transferring distilled knowledge with adversarial learning exhibits three important advantages: (1) the student network that learns the distilled knowledge with discriminators is optimized better than the original model; (2) fast inference is realized by a single forward pass, while the performance is even better than traditional ensembles from multi-original models; (3) the student network can learn the distilled knowledge from a teacher model that has arbitrary structures. Extensive experiments on CIFAR-10/100, SVHN and ImageNet datasets demonstrate the effectiveness of our MEAL method. On ImageNet, our ResNet-50 based MEAL achieves top-1/5 21.79%/5.99% val error, which outperforms the original model by 2.06%/1.14%.

The methods of duo output neural network ensemble for prediction of coronary heart disease

The occurrence of Coronary heart disease (CHD) is hard to predict yet , but the assessment of CHD risk for the next ten years is possible . The p rediction of coronary heart disease can be modelled using multi-layer perceptron neural network (MLP-ANN). Prediction model with MLP-ANN ha s either positive or negative CHD output , which is a binary classification. A prediction model with binary classification requires determin ation of threshold value before the classification process which increases the uncertainty in the classification process. Another weakness of the MLP-ANN model is the presence of overfitting. This study propose s a prediction model for coronary heart disease using the duo output artificial neural network ensemble (DOANNE) method to overcome the problems of overfitting and uncertainty of classification in MLP-ANN. This research method wa s divided into several stages, namely data acquisition, pre-processing, modelling into DOANNE, neural network ensemble training with Levenberg-Marquard (LM) algorithm, system performance testing, and evaluation. The results of the study showed that the use of DOANNE-LM method was able to provide a significant improvement from the MLP-ANN method, indicated by the results of statistical tests with p-value <0.05.

The methods of duo output neural network ensemble for prediction of coronary heart disease

The occurrence of Coronary heart disease (CHD) is hard to predict yet , but the assessment of CHD risk for the next ten years is possible . The p rediction of coronary heart disease can be modelled using multi-layer perceptron neural network (MLP-ANN). Prediction model with MLP-ANN ha s either positive or negative CHD output , which is a binary classification. A prediction model with binary classification requires determin ation of threshold value before the classification process which increases the uncertainty in the classification process. Another weakness of the MLP-ANN model is the presence of overfitting. This study propose s a prediction model for coronary heart disease using the duo output artificial neural network ensemble (DOANNE) method to overcome the problems of overfitting and uncertainty of classification in MLP-ANN. This research method wa s divided into several stages, namely data acquisition, pre-processing, modelling into DOANNE, neural network ensemble training with Levenberg-Marquard (LM) algorithm, system performance testing, and evaluation. The results of the study showed that the use of DOANNE-LM method was able to provide a significant improvement from the MLP-ANN method, indicated by the results of statistical tests with p-value <0.05.

A Multivariate Global Spatiotemporal Stochastic Generator for Climate Ensembles

In order to understand and quantify the uncertainties in projections and physics of a climate model, a collection of climate simulations (an ensemble) is typically used. Given the high-dimensionality of the input space of a climate model, as well as the complex, nonlinear relationships between the climate variables, a large ensemble is often required to accurately assess these uncertainties. If only a small number of climate variables are of interest at a specified spatial and temporal scale, the computational and storage expenses can be substantially reduced by training a statistical model on a small ensemble. The statistical model then acts as a stochastic generator (SG) able to simulate a large ensemble, given a small training ensemble. Previous work on SGs has focused on modeling and simulating individual climate variables (e.g., surface temperature, wind speed) independently. Here, we introduce a SG that jointly simulates three key climate variables. The model is based on a multistage spectral approach that allows for inference of more than 80 million data points for a nonstationary global model, by conducting inference in stages and leveraging large-scale parallelization across many processors. We demonstrate the feasibility of jointly simulating climate variables by training the SG on five ensemble members from a large ensemble project and assess the SG simulations by comparing them to the ensemble members not used in training.Supplementary materials accompanying this paper appear online.

Probabilistic Multimodel Ensemble Wake-Vortex Prediction Employing Bayesian Model Averaging

This paper investigates the capability of the multimodel ensemble approach to improve the deterministic forecast of wake-vortex behavior and to produce reliable probabilistic vortex habitation areas. Therefore, the deterministic two-phase wake vortex model D2P, the aircraft vortex spacing system prediction algorithm (APA) 3.2, APA 3.4, APA 3.8, and terminal area simulation system driven algorithms for wake prediction (TDP 2.1) wake-vortex models are exchanged within the framework of a NASA/DLR cooperation. These models are fused by the Bayesian model averaging approach, which is extended by temporally increasing uncertainties. In addition, combined confidence areas for the vertical and lateral vortex positions are derived from bivariate probability density distributions that are delivered by the ensemble and allow the computation of well-defined probability levels. For ensemble training and evaluation data collected at wake-vortex campaigns accomplished by NASA (at Memphis, Dallas, and Denver airports) and DLR (at Frankfurt, Munich, and Oberpfaffenhofen airports) are employed. Various training strategies are considered to obtain optimal prediction skill. The results demonstrate that a thoughtfully trained ensemble improves the deterministic prediction skill by up to 4.3% and is capable of predicting vortex habitation areas featuring reliable probability levels.

Ensemble of Random Binary Output Encoding for Adversarial Robustness

Despite the excellent classification performance, recent research has revealed that the Convolutional Neural Network (CNN) could be readily deceived by only the small adversarial perturbation. Its imperceptible to human eyes and transferability from one model to another actually threaten the security of a CNN-based system. In this paper, we propose to create multiple and independent random binary codes per input class and train ensemble of homogeneous CNN classifiers with these codes to improve the adversarial robustness of the networks. The proposed ensemble structure consists of replicas of the same learning architecture, but each network is trained with different random target outputs. The network model is simultaneously trained with their own unique binary codes, and optimized through a single and common objective function in an end-to-end manner. It is demonstrated with experimental results that assigning different encoded labels for each classifier in ensemble leverages the diversity and eventually improves the classification performance on adversarial attacks. We also conduct several performance analysis to understand how the different aspects can contribute to the robustness of the proposed algorithm. The proposed algorithm provides significantly improved classification accuracies as compared to the recent relevant studies, verified with various network architectures, datasets, and adversarial attacks.

Ensemble Learning for Power Systems TTC Prediction With Wind Farms

Being aware of the reliable margin of vital tie-lines, acting on the connection of power exporting area and power importing area, is significant to power systems. However, the high penetration of wind power causes fast variation of boundary limit parameters such as the available amount of power that can be transferred on the tie-lines, namely, total transfer capability (TTC), which may result in the inaccurate security assessment. Unfortunately, the traditional optimal power flow-based TTC model has computation burden for online applications. To address this problem, computational efficiency is improved via a data-driven TTC predictor based on an ensemble learning architecture in this paper. In the first stage, a daily profiles-based method including probabilistic sampling is proposed to simulate plenty of operation scenarios as data samples for ensemble training. Then, a hybrid feature selection approach, which is composed of the maximal information coefficient and nonparametric independence screening, is applied to determine the most correlative features to the objective variable. To enable the TTC predictor with high accuracy and generalization ability, a novel ensemble learning scheme for TTC predictor is constituted through clustering few adaptive hierarchical GA-based neural networks (AHGA-NNs predictor). At last, a modified New England test system is used to validate the proposed methodology. The results illustrate that combining with the appropriate feature selection, the presented ensemble learning has high performance on creating the accurate TTC predictor, which enables online secure margin monitoring for the vital tie-lines.

Problems of encoder-decoder frameworks for high-resolution remote sensing image segmentation: Structural stereotype and insufficient learning

This work explores the use of deep convolutional neural networks for high resolution remote sensing imagery segmentation. Encoder-decoder frameworks are popular in semantic image segmentation. However, encoder-decoder models face two main problems. The one is structural stereotype which is receptive fields imbalance rooted in this kind of frameworks. The other is insufficient learning that deeper neural networks tend to encounter the notorious problem of vanishing gradients. Structural stereotype leads to unfair learning and inhomogeneous reasoning. We are the first to reveal the problem and propose ensemble training and inference strategies to suppress the adverse consequences of structural stereotype as far as possible. To alleviate the problem of insufficient learning, we propose a novel residual architecture for encoder-decoder models. The proposed method yields state-of-the-art performances on the ISPRS 2D semantic labeling contest benchmark.

Machine Learning Methods for Rapid Prediction of Thermostabilizing Mutants of G‐protein‐coupled receptors in Detergents

G protein‐coupled receptors (GPCRs) are highly dynamic and often denature when extracted in detergents. Deriving thermostable mutants has been a successful strategy to stabilize GPCRs in detergents, but this process is experimentally tedious. We have developed a computational method that scans the entire receptor for alanine or leucine mutations and calculates the stability score for mutation of each position in the GPCR structural model. The method involves generating homology models of the receptors of varying accuracies and an ensemble of conformations by sampling the rigid body degrees of freedom of transmembrane helices. Then, an all‐atom force field function is used to calculate the enthalpy gain, known as the “stability score” upon mutation of every residue, in these receptor structures, to alanine. Inclusion of conformational sampling to account for structural perturbations improves the thermostability predictions compared to using a single structure. We have validated the method against experimentally measured thermostability data for single mutants of the β1‐adrenergic receptor (β1AR), adenosine A2A receptor (A2AR) and neurotensin receptor 1 (NTSR1). We will present some of our recent results on blind predictions of thermostabilizing mutations on NTSR1.Using the computed energy components in our dataset, we have recently tested several machine learning algorithms for rapid and reliable prediction of thermostable and non‐thermostable mutants. We used Matlab to build a binary classification model (thermostable vs. non‐thermostable) trained on data from an ensemble of three proteins (25% of dataset used for holdout cross‐validation), and tested against a fourth protein omitted from the training set. Initial dataset was heavily imbalanced to the “non‐thermostable” class. This was balanced by random replication of the undersampled “thermostable” class. Models trained by decision tree algorithms performed best in our study compared to k‐nearest neighbor (k‐NN) algorithms. Our precision and recall statistics for the best performing ensemble dataset were as follows: Boosted Trees – 62% precision, 34% recall, Simple Trees – 30% precision, 90% recall, Complex Trees – 36% precision, 73% recall, Medium k‐NN – 28% precision, 34% recall.These results are promising for predicting true thermostable mutants, and we are working to improve our methods for reducing the number of false positive predictions. This requires testing different training algorithms such as Support Vector Machines and other k‐NN and Decision Tree models, and potentially ensemble training models combining multiples of these algorithms. Our method is the first step toward a computational method for rapid prediction of thermostable mutants of GPCRs. These tools are critical for reliable and quick prediction of thermostabilizing mutations of GPCRs for drug discovery.Support or Funding InformationFunding for this work was from NIH‐RO1GM097261 to N.V.This abstract is from the Experimental Biology 2018 Meeting. There is no full text article associated with this abstract published in The FASEB Journal.

El Sistema-inspired ensemble music training is associated with changes in children’s neurocognitive functional integration: preliminary ERP evidence

ABSTRACTChildren (aged 9–12) training in an El Sistema-inspired program (OrKidstra) and a matched comparison group participated in an auditory Go/No-Go task while event-related potentials (ERPs) were recorded. Entire-sweep waveform patterns correlated with known ERP peaks associated with executive and other cognitive functions and indicated that the spread of neural activity in the initial 250 ms of executive attention processing (pre-P300) showed higher level of topographical overlap in OrKidstra children. In these children, late potentials (post-P300) concurrent with response control were more widely distributed and temporally coordinated. Intensive ensemble music training, we suggest, may be associated with neuroplastic changes facilitating integration of neural information.

Enhancing Virtual Screening Performance of Protein Kinases with Molecular Dynamics Simulations.

In silico virtual screening (VS) is a powerful hit identification technique used in drug discovery projects that aims to effectively distinguish true actives from inactive or decoy molecules. To better capture the dynamic behavior of protein drug targets, compound databases may be screened against an ensemble of protein conformations, which may be experimentally determined or generated computationally, i.e. via molecular dynamics (MD) simulations. Several studies have shown that conformations generated by MD are useful in identifying novel hit compounds, in part because structural rearrangements sampled during MD can provide novel targetable areas. However, it remains difficult to predict a priori when an MD conformation will outperform a VS against the crystal structure alone. Here, we assess whether MD conformations result in improved VS performance for six protein kinases. MD conformations are selected using three different methods, and their VS performances are compared to the corresponding crystal structures. Additionally, these conformations are used to train ensembles, and their VS performance is compared to the individual MD conformations and the corresponding crystal structures using receiver operating characteristic curve (ROC) metrics. We show that performing MD results in at least one conformation that offers better VS performance than the crystal structure, and that, while it is possible to train ensembles to outperform the crystal structure alone, the extent of this enhancement is target dependent. Lastly, we show that the optimal structural selection method is also target dependent and recommend optimizing virtual screens on a kinase-by-kinase basis to improve the likelihood of success.

Training Ensemble Perception

Training Ensemble Perception