Neural Network Ensemble Research Articles

AggreProt: a web server for predicting and engineering aggregation prone regions in proteins.

Recombinant proteins play pivotal roles in numerous applications including industrial biocatalysts or therapeutics. Despite the recent progress in computational protein structure prediction, protein solubility and reduced aggregation propensity remain challenging attributes to design. Identification of aggregation-prone regions is essential for understanding misfolding diseases or designing efficient protein-based technologies, and as such has a great socio-economic impact. Here, we introduce AggreProt, a user-friendly webserver that automatically exploits an ensemble of deep neural networks to predict aggregation-prone regions (APRs) in protein sequences. Trained on experimentally evaluated hexapeptides, AggreProt compares to or outperforms state-of-the-art algorithms on two independent benchmark datasets. The server provides per-residue aggregation profiles along with information on solvent accessibility and transmembrane propensity within an intuitive interface with interactive sequence and structure viewers for comprehensive analysis. We demonstrate AggreProt efficacy in predicting differential aggregation behaviours in proteins on several use cases, which emphasize its potential for guiding protein engineering strategies towards decreased aggregation propensity and improved solubility. The webserver is freely available and accessible at https://loschmidt.chemi.muni.cz/aggreprot/.

Reduced-order modeling of unsteady fluid flow using neural network ensembles

The use of deep learning has become increasingly popular in reduced-order models (ROMs) to obtain low-dimensional representations of full-order models. Convolutional autoencoders (CAEs) are often used to this end as they are adept at handling data that are spatially distributed, including solutions to partial differential equations. When applied to unsteady physics problems, ROMs also require a model for time-series prediction of the low-dimensional latent variables. Long short-term memory (LSTM) networks, a type of recurrent neural network useful for modeling sequential data, are frequently employed in data-driven ROMs for autoregressive time-series prediction. When making predictions at unseen design points over long time horizons, error propagation is a frequently encountered issue, where errors made early on can compound over time and lead to large inaccuracies. In this work, we propose using bagging, a commonly used ensemble learning technique, to develop a fully data-driven ROM framework referred to as the CAE-eLSTM ROM that uses CAEs for spatial reconstruction of the full-order model and LSTM ensembles for time-series prediction. When applied to two unsteady fluid dynamics problems, our results show that the presented framework effectively reduces error propagation and leads to more accurate time-series prediction of latent variables at unseen points.

Learning from virtual experiments to assist users of Small Angle Neutron Scattering in model selection

In this work, we combine the advantages of virtual Small Angle Neutron Scattering (SANS) experiments carried out by Monte Carlo simulations with the recent advances in computer vision to generate a tool that can assist SANS users in small angle scattering model selection. We generate a dataset of almost 260.000 SANS virtual experiments of the SANS beamline KWS-1 at FRM-II, Germany, intended for Machine Learning purposes. Then, we train a recommendation system based on an ensemble of Convolutional Neural Networks to predict the SANS model from the two-dimensional scattering pattern measured at the position-sensitive detector of the beamline. The results show that the CNNs can learn the model prediction task, and that this recommendation system has a high accuracy in the classification task on 46 different SANS models. We also test the network with real data and explore the outcome. Finally, we discuss the reach of counting with the set of virtual experimental data presented here, and of such a recommendation system in the SANS user data analysis procedure.

Contributions of core, mantle and climatological processes to Earth’s polar motion

Earth’s spin axis slowly moves relative to the crust over time. A 120-year-long record of this polar motion from astronomical and more modern geodetic measurements displays interannual and multidecadal fluctuations of 20 to 40 milliarcseconds superimposed on a secular trend of about 3 milliarcseconds per year. Earth’s polar motion is thought to be driven by various surface and interior processes, but how these processes operate and interact to produce the observed signal remains enigmatic. Here we show that predictions made by an ensemble of physics-informed neural networks trained on measurements to capture geophysical processes can explain the main features of the observed polar motion. We find that glacial isostatic adjustment and mantle convection primarily account for the secular trend. Mass redistribution on the Earth’s surface—for example, ice melting and global changes in water storage—yields a relatively weak trend but explains about 90% of the interannual and multidecadal variations. We also find that core processes contribute to both the secular trend and fluctuations in polar motion, either due to variations in torque at the core–mantle boundary or dynamical feedback of the core in response to surface mass changes. Our findings provide constraints on core–mantle interactions for which observations are rare and global ice mass balance over the past century and suggest feedback operating between climate-related surface processes and core dynamics.

Uncertainty Quantification of a Machine Learning Subgrid‐Scale Parameterization for Atmospheric Gravity Waves

AbstractSubgrid‐scale processes, such as atmospheric gravity waves (GWs), play a pivotal role in shaping the Earth's climate but cannot be explicitly resolved in climate models due to limitations on resolution. Instead, subgrid‐scale parameterizations are used to capture their effects. Recently, machine learning (ML) has emerged as a promising approach to learn parameterizations. In this study, we explore uncertainties associated with a ML parameterization for atmospheric GWs. Focusing on the uncertainties in the training process (parametric uncertainty), we use an ensemble of neural networks to emulate an existing GW parameterization. We estimate both offline uncertainties in raw NN output and online uncertainties in climate model output, after the neural networks are coupled. We find that online parametric uncertainty contributes a significant source of uncertainty in climate model output that must be considered when introducing NN parameterizations. This uncertainty quantification provides valuable insights into the reliability and robustness of ML‐based GW parameterizations, thus advancing our understanding of their potential applications in climate modeling.

Symbolic Interpretation of Trained Neural Network Ensembles

Symbolic Interpretation of Trained Neural Network Ensembles

Evolutionary Ensemble Learning for EEG-Based Cross-Subject Emotion Recognition.

Electroencephalogram (EEG) has been widely utilized in emotion recognition due to its high temporal resolution and reliability. However, the individual differences and non-stationary characteristics of EEG, along with the complexity and variability of emotions, pose challenges in generalizing emotion recognition models across subjects. In this paper, an end-to-end framework is proposed to improve the performance of cross-subject emotion recognition. A novel evolutionary programming (EP)-based optimization strategy with neural network (NN) as the base classifier termed NN ensemble with EP (EPNNE) is designed for cross-subject emotion recognition. The effectiveness of the proposed method is evaluated on the publicly available DEAP, FACED, SEED, and SEED-IV datasets. Numerical results demonstrate that the proposed method is superior to state-of-the-art cross-subject emotion recognition methods. The proposed end-to-end framework for cross-subject emotion recognition aids biomedical researchers in effectively assessing individual emotional states, thereby enabling efficient treatment and interventions.

Increasing the Accuracy of Determining the Cardiothoracic Ratio with the Help of an Ensemble of Neural Networks

Increasing the Accuracy of Determining the Cardiothoracic Ratio with the Help of an Ensemble of Neural Networks

Efficient neural network-aided seismic life-cycle cost optimization of steel moment frames

In this paper, a novel and efficient neural network-based methodology is proposed to achieve seismic total cost optimization of steel moment-resisting frames in a timely manner. The computational burden of an optimization process based on performing nonlinear time-history analysis is prohibitively high. To address this crucial issue, a new and efficient neural network model is proposed in this paper to accurately predict the nonlinear time history response of steel frames during the optimization process. In the proposed neural network model, an ensemble of parallel neural networks is used to provide excellent prediction accuracy. In addition, a new repairability constraint is proposed to check the seismic damage level of structures during the optimization process with the aid of the proposed neural network model. Moreover, an efficient metaheuristic algorithm is used to achieve the optimization task. Two numerical examples are illustrated to demonstrate the efficiency of the proposed methodology. The results show that the proposed neural network model outperforms the existing standard models in terms of prediction accuracy. Furthermore, it is shown that by using the proposed methodology, the optimal seismic total cost of steel frames increases by less than 2.5%, yet their seismic collapse capacity increases by at least 30%.

A complementary Diagnostic Tool for Diabetic Peripheral Neuropathy Through Muscle Ultrasound and Machine Learning Algorithms

Diabetic peripheral neuropathy represents one of the common long-terms complications that effect about fifty percentage?of diabetes patients. The habitual diagnosis tool based on nerve conduction study that examine the nerve damage and classify the patient status into normal and diabetic peripheral neuropathy with degree of severity without considering the effect on skeletal muscle and take on patient data. A complementary diagnostic tool proposed, in this study integrates the patient’s data including body mass index, age and duration of diabetic, average blood glucose levels, nerve conduction study that involves amplitude and latency of peroneal and tibial nerves and muscle ultrasound alongside the machine learning algorithms to facilitate the clinicians for a precise diagnosis. A group of control and diabetic patients utilized to gather the data with calculating the muscle thickness and statistical properties from the gray-level ultrasound images of six skeletal muscles. Support vector machine, naïve bayes, ensemble of bagged tree and artificial neural network supervised machine learning algorithms categorize each class with a high classification accuracy, 98.1% for tibialis anterior with naïve bayes algorithm. The outcomes of this study show a promising complementary diagnostic tool that will help the clinicians to perform an exact diagnosis and disclose the side effect on both nerves and muscles of diabetic patients.

Hybrid brain tumor classification of histopathology hyperspectral images by linear unmixing and an ensemble of deep neural networks.

Hyperspectral imaging has demonstrated its potential to provide correlated spatial and spectral information of a sample by a non-contact and non-invasive technology. In the medical field, especially in histopathology, HSI has been applied for the classification and identification of diseased tissue and for the characterization of its morphological properties. In this work, we propose a hybrid scheme to classify non-tumor and tumor histological brain samples by hyperspectral imaging. The proposed approach is based on the identification of characteristic components in a hyperspectral image by linear unmixing, as a features engineering step, and the subsequent classification by a deep learning approach. For this last step, an ensemble of deep neural networks is evaluated by a cross-validation scheme on an augmented dataset and a transfer learning scheme. The proposed method can classify histological brain samples with an average accuracy of 88%, and reduced variability, computational cost, and inference times, which presents an advantage over methods in the state-of-the-art. Hence, the work demonstrates the potential of hybrid classification methodologies to achieve robust and reliable results by combining linear unmixing for features extraction and deep learning forclassification.

High-resolution mapping of tree species and associated uncertainty by combining aerial remote sensing data and convolutional neural networks ensemble

Mapping tree species diversity is essential for monitoring and managing forest ecosystems. Automating ecoforestry mapping using remote sensing images remains an important challenge due to the tremendous variability in forest covers and the conditions under which images used for classification are acquired. Deep learning algorithms have been increasingly used over the last few years to analyse remote sensing data for mapping tree species. However, most of these studies focus only on a small number of species or a limited area and avoid providing a spatially explicit representation of the uncertainty related to the predictions, rendering them unsuitable for operational use.In this study, we used an ensemble of convolutional neural networks to map forest species and land cover types across a 10,000 km2 area in Quebec, Canada, spanning mixed and boreal forests. We built a georeferenced label database to train and test nine models, which resulted from the combination of three training datasets and three-commonly used convolutional neural network architectures (VGG16, ResNet50v2, Densenet121). These models were trained on multiband aerial photographs and a canopy height model derived from airborne lidar point clouds and used to map the diversity and distribution of tree species and land cover types. The level of agreement among models was used to generate uncertainty maps. The performance of the super-ensemble using 1311 independent forest inventory plots and assessed the extent to which uncertainty maps could serve as a spatially explicit indicator of model performance.The super-ensemble achieved 90% global accuracy. Our results indicated that the performance of the super-ensemble surpassed that of all individual architectures while also showing a positive effect of the canopy height model on performance. The comparison of the super-ensemble map with the proportion of basal area measured in forest inventory plots confirmed the reliability of the model over the study area. Our results also indicated that mapping the inter-model agreement provides a reliable spatially explicit estimate of model performance. The robustness and reliability of the proposed approach support its use in an operational context while also providing a conceptual framework to evaluate the reliability of uncertainty maps.

Analysis of convolutional neural networks for fronto-temporal dementia biomarker discovery.

Frontotemporal lobe dementia (FTD) results from the degeneration of the frontal and temporal lobes. It can manifest in several different ways, leading to the definition of variants characterised by their distinctive symptomatologies. As these variants are detected based on their symptoms, it can be unclear if they represent different types of FTD or different symptomatological axes. The goal of this paper is to investigate this question with a constrained cohort of FTD patients in order to see if the heterogeneity within this cohort can be inferred from medical images rather than symptom severity measurements. An ensemble of convolutional neural networks (CNNs) is used to classify diffusion tensor images collected from two databases consisting of 72 patients with behavioural variant FTD and 120 healthy controls. FTD biomarkers were found using voxel-based analysis on the sensitivities of these CNNs. Sparse principal components analysis (sPCA) is then applied on the sensitivities arising from the patient cohort in order to identify the axes along which the patients express these biomarkers. Finally, this is correlated with their symptom severity measurements in order to interpret the clinical presentation of each axis. The CNNs result in sensitivities and specificities between 83 and 92%. As expected, our analysis determines that all the robust biomarkers arise from the frontal and temporal lobes. sPCA identified four axes in terms of biomarker expression which are correlated with symptom severity measurements. Our analysis confirms that behavioural variant FTD is not a singular type or spectrum of FTD, but rather that it has multiple symptomatological axes that relate to distinct regions of the frontal and temporal lobes. This analysis suggests that medical images can be used to understand the heterogeneity of FTD patients and the underlying anatomical changes that lead to their different clinical presentations.

ESEN: Efficient GPU sharing of Ensemble Neural Networks

Ensemble neural networks are widely applied in cloud-based inference services due to their remarkable performance, while the growing demand for low-latency services leads researchers to pay more attention to the execution efficiency of these models, especially the device utilization. It is highly desirable to fully utilize GPUs by multiplexing different inference tasks on the same GPU with advanced sharing technique, such as Multi-Process-Service (MPS). However, we find it struggling when applying MPS to Ensemble Neural Networks, which consist of multiple related sub-models. The critical challenge in this predicament revolves around the efficient allocation of resources within an ensemble, aiming to minimize job completion time.To tackle this challenge, we initially examine the interplay among individual neural networks within an ensemble, outlining a guideline for achieving the shortest job completion time. Subsequently, we establish a mathematical model to formalize the resource requirements of each neural network. We introduce a search-based allocation algorithm designed to swiftly identify optimal solutions. Finally, we introduce ESEN, comprising the search-based resource allocation algorithm and efficient model execution mechanisms within PyTorch. ESEN is augmented with customized execution mechanisms for user-friendly implementation. Experimental results demonstrate that proposed ESEN can attain an efficiency improvement up to 17.84% and a GPU utilization increase of 28.09% compared to the default strategy. With the optimization of GPU resource allocation, ESEN significantly improves the efficiency of ensemble models. It provides a low-latency and high-accuracy solution for online interactive services.

3D brain image based tumor classification using ensemble of reinforcement transfer-based belief neural networks

3D brain image based tumor classification using ensemble of reinforcement transfer-based belief neural networks

Advancing Arabic Handwriting Recognition with Convolutional and Recurrent Neural Network Ensembles

Advancing Arabic Handwriting Recognition with Convolutional and Recurrent Neural Network Ensembles

An Architecture as an Alternative to Gradient Boosted Decision Trees for Multiple Machine Learning Tasks

Deep networks-based models have achieved excellent performances in various applications for extracting discriminative feature representations by convolutional neural networks (CNN) or recurrent neural networks (RNN). However, CNN or RNN may not work when handling data without temporal/spatial structures. Therefore, finding a new technique to extract features instead of CNN or RNN is a necessity. Gradient Boosted Decision Trees (GBDT) can select the features with the largest information gain when building trees. In this paper, we propose an architecture based on the ensemble of decision trees and neural network (NN) for multiple machine learning tasks, e.g., classification, regression, and ranking. It can be regarded as an extension of the widely used deep-networks-based model, in which we use GBDT instead of CNN or RNN. This architecture consists of two main parts: (1) the decision forest layers, which focus on learning features from the input data, (2) the fully connected layers, which focus on distilling knowledge from the decision forest layers. Powered by these two parts, the proposed model could handle data without temporal/spatial structures. This model can be efficiently trained by stochastic gradient descent via back-propagation. The empirical evaluation results of different machine learning tasks demonstrate the the effectiveness of the proposed method.

A deep learning-based compression and classification technique for whole slide histopathology images

This paper presents an autoencoder-based neural network architecture to compress histopathological images while retaining the denser and more meaningful representation of the original images. Current research into improving compression algorithms is focused on methods allowing lower compression rates for Regions of Interest (ROI-based approaches). Neural networks are great at extracting meaningful semantic representations from images and, therefore can select the regions to be considered of interest for the compression process. In this work, we focus on the compression of whole slide histopathology images. The objective is to build an ensemble of neural networks that enables a compressive autoencoder in a supervised fashion to retain a denser and more meaningful representation of the input histology images. Our proposed system is a simple and novel method to supervise compressive neural networks. We test the compressed images using transfer learning-based classifiers and show that they provide promising accuracy and classification performance.

Advancements in Adolescent Idiopathic Scoliosis Diagnosis: A Focus on Non-Invasive Techniques

Abstract: Adolescent idiopathic scoliosis (AIS) poses a significant health concern, affecting the spinal curvature of adolescents during their growth period. Early diagnosis of AIS is critical for effective intervention and management. However, conventional diagnostic methods often involve repetitive exposure to X-rays, which can lead to radiation-related side effects. This research paper explores the significance of early diagnosis in AIS and emphasizes the utilization of non-invasive techniques, particularly video raster stereography (VRS), combined with deep ensemble neural networks (DNNs) to enhance diagnostic accuracy while minimizing the risks associated with radiation exposure. We delve into the potential of DNNs in analyzing VRS data and compare its efficacy with other machine learning alternatives

An ensemble spatial prediction method considering geospatial heterogeneity

Ensemble learning synthesizes the advantages of different models and has been widely applied in the field of spatial prediction. However, the nonlinear constraints of spatial heterogeneity on the model ensemble process make it difficult to adaptively determine the ensemble weights, greatly limiting the predictive ability of the ensemble learning model. This paper therefore proposes a novel geographical spatial heterogeneous ensemble learning method (GSH-EL). Firstly, the geographically weighted regression model, geographically optimal similarity model, and random forest model are used as three base learners to express local spatial heterogeneity, global feature correlation, and nonlinear relationship of geographic elements, respectively. Then, a spatially weighted ensemble neural network module (SWENN) of GSH-EL is proposed to express spatial heterogeneity by exploring the complex nonlinear relationship between the spatial proximity and ensemble weights. Finally, the outputs of the three base learners are combined with the spatial heterogeneous ensemble weights from SWENN to obtain the spatial prediction results. The proposed method is validated on the PM2.5 air quality and landslide dataset in China, both of which obtain more accurate prediction results than the existing ensemble learning strategies. The results confirm the need to accurately express spatial heterogeneity in the model ensemble process.