Use Of Ensemble Methods Research Articles

Proposed Methods for Preventing Overfitting in Machine Learning and Deep Learning

The article discusses various approaches and methods to combat overfitting, which is a key problem in the field of artificial intelligence. Overfitting occurs when the model over-adapts to the training dataset, losing the ability to generalize to new data. The main causes and signs of overfitting are also discussed, including excessive complexity of models and limited data. The focus is on methods such as regularization, dropout, and the use of ensemble methods that can significantly reduce the risk of overfitting. These approaches are evaluated using examples from various fields of neural network applications, providing the reader with a comprehensive understanding of the problem and its solution methods.

Probabilistic Record Linkage for Families (PRLF): A Discussion of the Development and Validation of this Open-Source Linkage Tool

Linking administrative records across programs can yield person-centered information, including client characteristics, public service trajectories, and outcomes and help to answer policy-related questions. Several solutions are available for undertaking record linkage, producing linkage keys for merging data sources for positively matched pairs of records. In this session, we will demonstrate a new application of the Python RecordLinkage package to family-based record linkages with machine learning algorithms for probability scoring, which we call probabilistic record linkage for families (PRLF). First, we will demonstrate the utility of PRLF with a simulation of administrative records and assess linkage accuracy with variations in match rates and data degradation. Second, we will compare generalized linear model estimates across three record linkage solutions (PRLF, ChoiceMaker, and Link Plus). Findings from the simulation study indicate linkage accuracy is largely influenced by degradation (e.g., missing data fields, erroneous or incomplete values) compared to the proportion of simulated matches between datasets. Results from the methods comparison using real world data indicate that all three solutions, when optimized, provide similar results for researchers. We discuss the strengths of our process, such as the use of ensemble methods, to improve match accuracy. We then will identify caveats of record linkage in the context of administrative data. The tool was developed in Python to allow for researchers to work with open-source software and adjust the basic workflow to fit their linkage needs. We will identify several partnerships where this collaboration has worked successfully and empower attendees with access to this useful tool.

River Ecomorphodynamic Models Exhibit Features of Nonlinear Dynamics and Chaos

AbstractModeling the nonlinear interactions between flow, sediment, and vegetation is essential for improving our understanding and prediction of river system dynamics. Using simple numerical models, we simulate the key flow‐sediment‐vegetation interaction where the disturbance is intrinsically generated by the presence of vegetation. In this case, biomass growth modifies the flow field, induces bed scour, and thus potentially causes vegetation uprooting when erosion exceeds root depth. Our results show that this nonlinear feedback produces deterministic chaos under a wide range of conditions, with complex aperiodic dynamics generated by a period‐doubling route to chaos. Moreover, our results suggest relatively small values of Lyapunov time, spanning 2–4 growth‐flood cycles, which significantly restrict the predictability of riverbed evolution. Although further spatial and temporal processes may add complexity to the system, these results call for the use of ensemble methods and associated uncertainty estimates in ecomorphodynamic models.

Enhancing Agricultural Decision-Making through Data Analysis: Predicting Crop Health Outcomes

This research employs advanced data analysis techniques to predict crop health outcomes during harvest seasons, with a focus on insect count, pesticide use, and soil type. The study encompasses two main components: feature correlation and predictive modeling. Feature engineering techniques are applied to capture variations in pesticide use and insect infestation, enhancing predictive capabilities. Ensemble methods, including Random Forest, XGBoost, and Decision Trees, are employed to forecast patterns of crop damage based on identified trends. Decision Trees exhibit robust capabilities, achieving an impressive accuracy rate of 90.03%. Random Forest excels with a robust accuracy of 90.35%, highlighting its classification abilities. XGBoost stands out with an accuracy rate of 86.51%. In contrast, Logistic Regression, Naive Bayes, and Convolutional Neural Networks face challenges, displaying lower accuracy. The evaluation further emphasizes the strength of ensemble methods and Decision Trees through precision, recall, and F1-Score metrics, providing a comprehensive understanding of relationships within pesticide damage. The framework of the study introduced in this paper can be seen as a major step forward with regard to agricultural decision-making. We present actionable strategies to enhance crop health while reducing damage through the integration of feature correlation, predictive modeling and precise evaluation metrics. The innovativeness is in the use of ensemble methods and Decision Trees that are implemented to promote informed decision-making among stakeholders through a sustainable approach to agriculture.

An open-source probabilistic record linkage process for records with family-level information: Simulation study and applied analysis.

Research with administrative records involves the challenge of limited information in any single data source to answer policy-related questions. Record linkage provides researchers with a tool to supplement administrative datasets with other information about the same people when identified in separate sources as matched pairs. Several solutions are available for undertaking record linkage, producing linkage keys for merging data sources for positively matched pairs of records. In the current manuscript, we demonstrate a new application of the Python RecordLinkage package to family-based record linkages with machine learning algorithms for probability scoring, which we call probabilistic record linkage for families (PRLF). First, a simulation of administrative records identifies PRLF accuracy with variations in match and data degradation percentages. Accuracy is largely influenced by degradation (e.g., missing data fields, mismatched values) compared to the percentage of simulated matches. Second, an application of data linkage is presented to compare regression model estimate performance across three record linkage solutions (PRLF, ChoiceMaker, and Link Plus). Our findings indicate that all three solutions, when optimized, provide similar results for researchers. Strengths of our process, such as the use of ensemble methods, to improve match accuracy are discussed. We then identify caveats of record linkage in the context of administrative data.

Introducing CNN-LSTM network adaptations to improve remaining useful life prediction of complex systems

AbstractPrognostics and Health Management (PHM) models aim to estimate remaining useful life (RUL) of complex systems, enabling lower maintenance costs and increased availability. A substantial body of work considers the development and testing of new models using the NASA C-MAPSS dataset as a benchmark. In recent work, the use of ensemble methods has been prevalent. This paper proposes two adaptations to one of the best-performing ensemble methods, namely the Convolutional Neural Network – Long Short-Term Memory (CNN-LSTM) network developed by Li et al. (IEEE Access, 2019, 7, pp 75464–75475)). The first adaptation (adaptable time window, or ATW) increases accuracy of RUL estimates, with performance surpassing that of the state of the art, whereas the second (sub-network learning) does not improve performance. The results give greater insight into further development of innovative methods for prognostics, with future work focusing on translating the ATW approach to real-life industrial datasets and leveraging findings towards practical uptake for industrial applications.

The dark energy survey 5-yr photometrically identified type Ia supernovae

ABSTRACT As part of the cosmology analysis using Type Ia Supernovae (SN Ia) in the Dark Energy Survey (DES), we present photometrically identified SN Ia samples using multiband light curves and host galaxy redshifts. For this analysis, we use the photometric classification framework SuperNNovatrained on realistic DES-like simulations. For reliable classification, we process the DES SN programme (DES-SN) data and introduce improvements to the classifier architecture, obtaining classification accuracies of more than 98 per cent on simulations. This is the first SN classification to make use of ensemble methods, resulting in more robust samples. Using photometry, host galaxy redshifts, and a classification probability requirement, we identify 1863 SNe Ia from which we select 1484 cosmology-grade SNe Ia spanning the redshift range of 0.07 < z < 1.14. We find good agreement between the light-curve properties of the photometrically selected sample and simulations. Additionally, we create similar SN Ia samples using two types of Bayesian Neural Network classifiers that provide uncertainties on the classification probabilities. We test the feasibility of using these uncertainties as indicators for out-of-distribution candidates and model confidence. Finally, we discuss the implications of photometric samples and classification methods for future surveys such as Vera C. Rubin Observatory Legacy Survey of Space and Time.

Ensemble Forecasting Methods in DCF Modelling of the Fair Value of Enterprises

The purpose of this study is to improve the methodology for assessing the fair value of enterprises based on the DCF model with the use of ensemble methods for predicting the selling price of their products. The study compares the fair values of three Ukrainian agricultural enterprises. The value of each company was calculated in several ways: using forecast data from statistical agencies, calculated forecast values, and real indicators. The methodology is based on the use of methods and models for forecasting time series. The study uses neural networks, as well as prediction methods such as linear regression, FB Prophet model, Holt-Winters exponential smoothing method, SARIMA, XGBoost, which are combined into a single ensemble with the use of the stacking method. The study employed such general scientific methods as analysis, synthesis, abstraction, and comparison. Also, methods of graphical and tabular presentation of materials were applied. Statistical modelling was used to determine the parameters of the model. The study found that the DCF model can potentially be improved by using ensemble methods when predicting metrics such as inflation, exchange rates, and the selling price of goods and services. These metrics have a significant impact on the final result of the model and therefore the slightest changes in these input data can lead to significant deviations in the result. It was demonstrated that the use of ensemble methods only at the selling price can increase the accuracy of the model by 5-15%. It is advisable to use the results of the study in the investment activities of companies, in mergers and acquisitions of companies, as well as in measuring the fair or investment value of companies.

Ensemble Methods for Jump-Diffusion Models of Power Prices

We propose a machine learning-based methodology which makes use of ensemble methods with the aims (i) of treating missing data in time series with irregular observation times and detecting anomalies in the observed time behavior; (ii) of defining suitable models of the system dynamics. We applied this methodology to US wholesale electricity price time series that are characterized by missing data, high and stochastic volatility, jumps and pronounced spikes. For missing data, we provide a repair approach based on the missForest algorithm, an imputation algorithm which is completely agnostic about the data distribution. To identify anomalies, i.e., turbulent movements of power prices in which jumps and spikes are observed, we took into account the no-gap reconstructed electricity price time series, and then we detected anomalous regions using the isolation forest algorithm, an anomaly detection method that isolates anomalies instead of profiling normal data points as in the most common techniques. After removing anomalies, the additional gaps will be newly filled by the missForest imputation algorithm. In this way, a complete and clean time series describing the stable dynamics of power prices can be obtained. The decoupling between the stable motion and the turbulent motion allows us to define suitable jump-diffusion models of power prices and to provide an estimation procedure that uses the full information contained in both the stable and the turbulent dynamics.

On the use of ensemble method for multi view textual data

ABSTRACT Nowadays, trends detection is an important task on social media to determine trends that are being discussed the most on a social platform. One of the main challenges of this task is the processing of unstructured textual data which has different representations. Therefore, multi view text clustering presents a useful solution for trends detection by integrating various representations called ‘views’ to provide a complementary description of the same content. In this context, we propose a new ensemble method for multi-view text clustering that exploits different representations of text in order to produce more accurate and high quality clustering. Extensive experiments on real-world text datasets were conducted to demonstrate its superiority by comparing with the existing methods. An application of the proposed method in trends detection from twitter is also illustrated.

Investigations on the Use of Ensemble Methods for Specification-Oriented Indirect Test of RF Circuits

In order to reduce the costs of industrial testing of analog and Radio Frequency (RF) integrated circuits, a widely studied solution is indirect testing. Indeed, indirect testing is based on learning-machine algorithms to train a regression model that links the space of low-cost indirect measurements to the space of performance parameters guaranteed by datasheets, thus relaxing the constraints on expensive test equipment. This article explores the potential benefit of using ensemble learning in this context. Unlike traditional learning models that use a single model to estimate targeted parameters, ensemble-learning models involve training several individual regression models and combining their outputs to improve the predictive power of the ensemble model. Different ensemble methods based on bagging, boosting or stacking are investigated and compared to classical individual models. Experiments are performed on three RF performances of a LNA for which we have production test data and model quality is discussed in terms of goodness-of-fit, accuracy and reliability. The influence of the training set size is also explored. Finally, the efficiency of classical and ensemble models is compared in the context of a two-tier test flow that permits to tradeoff test cost and test quality.

Ensemble Learning to Improve the Prediction of Fetal Macrosomia and Large-for-Gestational Age.

Background: The objective of this study was to investigate the use of ensemble methods to improve the prediction of fetal macrosomia and large for gestational age from prenatal ultrasound imaging measurements. Methods: We evaluated and compared the prediction accuracies of nonlinear and quadratic mixed-effects models coupled with 26 different empirical formulas for estimating fetal weights in predicting large fetuses at birth. The data for the investigation were taken from the Successive Small-for-Gestational-Age-Births study. Ensemble methods, a class of machine learning techniques, were used to improve the prediction accuracies by combining the individual models and empirical formulas. Results: The prediction accuracy of individual statistical models and empirical formulas varied considerably in predicting macrosomia but varied less in predicting large for gestational age. Two ensemble methods, voting and stacking, with model selection, can combine the strengths of individual models and formulas and can improve the prediction accuracy. Conclusions: Ensemble learning can improve the prediction of fetal macrosomia and large for gestational age and have the potential to assist obstetricians in clinical decisions.

Training ensembles of faceted classification models for quantitative stock trading

Forecasting the stock markets is among the most popular research challenges in finance. Several quantitative trading systems based on supervised machine learning approaches have been presented in literature. Recently proposed solutions train classification models on historical stock-related datasets. Training data include a variety of features related to different facets (e.g., stock price trends, exchange volumes, price volatility, news and public mood). To increase the accuracy of the predictions, multiple models are often combined together using ensemble methods. However, understanding which models should be combined together and how to effectively handle features related to different facets within different models are still open research questions. In this paper we investigate the use of ensemble methods to combine faceted classification models for supporting stock trading. To this aim, separate classification models are trained on each subset of features belonging to the same facet. They produce trading signals tailored to a specific facet. Signals are then combined together and filtered to generate a unified, multi-faceted recommendation. The experimental validation, performed on different markets and in different conditions, shows that, in many cases, some of the faceted models perform as good as or better than models trained on a mix of different features. An ensemble of the faceted recommendations makes the generated trading signals more profitable yet robust to draw-down periods.

An intelligent ensemble learning approach for energy efficient and interference aware dynamic virtual machine consolidation

Inefficient resource usage is one of the greatest causes of high energy consumption in cloud data centers. Virtual Machine (VM) consolidation is an effective method for improving energy related costs and environmental sustainability for modern data centers. While dynamic VM consolidation algorithms can improve energy efficiency, virtualisation technologies cannot guarantee performance isolation between co-located VMs resulting in interference issues. We address the problem by introducing an energy and interference aware VM consolidation algorithm which uses predictive modelling to classify workloads using their resource usage features to make more informed consolidation decisions. The use of ensemble methods plays a pivotal role for improving predictive performance for many different problems. Using recent workload data from Microsoft Azure we present a comparative analysis of several ensemble methods using state-of-the-art prediction models and propose an ensemble based VM consolidation algorithm. Our empirical results demonstrate how our approach improves energy efficiency by 34% while also reducing service violations by 77%.

Combination of loss functions for deep text classification

Ensemble methods have shown to improve the results of statistical classifiers by combining multiple single learners into a strong one. In this paper, we explore the use of ensemble methods at the level of the objective function of a deep neural network. We propose a novel objective function that is a linear combination of single losses and integrate the proposed objective function into a deep neural network. By doing so, the weights associated with the linear combination of losses are learned by back propagation during the training stage. We study the impact of such an ensemble loss function on the state-of-the-art convolutional neural networks for text classification. We show the effectiveness of our approach through comprehensive experiments on text classification. The experimental results demonstrate a significant improvement compared with the conventional state-of-the-art methods in the literature.

Applications of Python to Evaluate the Performance of Bagging Methods

The use of ensemble methods for obtaining scalable solutions on complex multi-dimensional datasets has increased manifold in the field of advanced machine learning and analytics owing to the ensemble method's capabilities of combining multiple base estimators to generate a more robust estimator than any single estimator with a given algorithm. Bagging and boosting are the two widely used ensemble methods. This paper presents a step-by-step approach to the applications of python in evaluating the performance of three bagging ensemble methods, namely, bagging, random forest, and extremely randomized trees for predicting the in-bus carbon dioxide concentrations. The bagging ensemble model evaluation results from this study were compared with the results obtained from a prior study that evaluated the performance of four boosting (gradient boosting machine, light gradient boosting machine, extreme gradient boosting, adaptive boosting) ensemble methods utilizing the same in-bus database. Among the seven ensemble methods, the random forest ensemble method provided better results on the basis of predictive model evaluation with operational performance measures. The readers may adopt the bagging ensemble methods (inclusive of the python coding) discussed in this article to successfully address their own data science problems. © 2018 American Institute of Chemical Engineers Environ Prog, 37: 1555–1559, 2018

Creation of a Robust and Generalizable Machine Learning Classifier for Patient Ventilator Asynchrony.

As healthcare increasingly digitizes, streaming waveform data is being made available from an variety of sources, but there still remains a paucity of performant clinical decision support systems. For example, in the intensive care unit (ICU) existing automated alarm systems typically rely on simple thresholding that result in frequent false positives. Recurrent false positive alerts create distrust of alarm mechanisms that can be directly detrimental to patient health. To improve patient care in the ICU, we need alert systems that are both pervasive, and accurate so as to be informative and trusted by providers. We aimed to develop a machine learning-based classifier to detect abnormal waveform events using the use case of mechanical ventilation waveform analysis, and the detection of harmful forms of ventilation delivery to patients. We specifically focused on detecting injurious subtypes of patient-ventilator asynchrony (PVA). Using a dataset of breaths recorded from 35 different patients, we used machine learning to create computational models to automatically detect, and classify two types of injurious PVA, double trigger asynchrony (DTA), breath stacking asynchrony (BSA). We examined the use of synthetic minority over-sampling technique (SMOTE) to overcome class imbalance problems, varied methods for feature selection, and use of ensemble methods to optimize the performance of our model. We created an ensemble classifier that is able to accurately detect DTA at a sensitivity/specificity of 0.960/0.975, BSA at sensitivity/specificity of 0.944/0.987, and non-PVA events at sensitivity/specificity of .967/.980. Our results suggest that it is possible to create a high-performing machine learning-based model for detecting PVA in mechanical ventilator waveform data in spite of both intra-patient, and inter-patient variability in waveform patterns, and the presence of clinical artifacts like cough and suction procedures. Our work highlights the importance of addressing class imbalance in clinical data sets, and the combined use of statistical methods and expert knowledge in feature selection.

Applications of python to evaluate the performance of bagging methods

The use of ensemble methods for obtaining scalable solutions on complex multi‐dimensional datasets has increased manifold in the field of advanced machine learning and analytics owing to the ensemble method's capabilities of combining multiple base estimators to generate a more robust estimator than any single estimator with a given algorithm. Bagging and boosting are the two widely used ensemble methods. This paper presents a step‐by‐step approach to the applications of python in evaluating the performance of three bagging ensemble methods, namely, bagging, random forest, and extremely randomized trees for predicting the in‐bus carbon dioxide concentrations. The bagging ensemble model evaluation results from this study were compared with the results obtained from a prior study that evaluated the performance of four boosting (gradient boosting machine, light gradient boosting machine, extreme gradient boosting, adaptive boosting) ensemble methods utilizing the same in‐bus database. Among the seven ensemble methods, the random forest ensemble method provided better results on the basis of predictive model evaluation with operational performance measures. The readers may adopt the bagging ensemble methods (inclusive of the python coding) discussed in this article to successfully address their own data science problems. © 2018 American Institute of Chemical Engineers Environ Prog, 37: 1555–1559, 2018

Open Data for Global Multimodal Land Use Classification: Outcome of the 2017 IEEE GRSS Data Fusion Contest

In this paper, we present the scientific outcomes of the 2017 Data Fusion Contest organized by the Image Analysis and Data Fusion Technical Committee of the IEEE Geoscience and Remote Sensing Society. The 2017 Contest was aimed at addressing the problem of local climate zones classification based on a multitemporal and multimodal dataset, including image (Landsat 8 and Sentinel-2) and vector data (from OpenStreetMap). The competition, based on separate geographical locations for the training and testing of the proposed solution, aimed at models that were accurate (assessed by accuracy metrics on an undisclosed reference for the test cities), general (assessed by spreading the test cities across the globe), and computationally feasible (assessed by having a test phase of limited time). The techniques proposed by the participants to the Contest spanned across a rather broad range of topics, and of mixed ideas and methodologies deriving from computer vision and machine learning but also deeply rooted in the specificities of remote sensing. In particular, rigorous atmospheric correction, the use of multidate images, and the use of ensemble methods fusing results obtained from different data sources/time instants made the difference.

Parameterizations for ensemble Kalman inversion

The use of ensemble methods to solve inverse problems is attractive because it is a derivative-free methodology which is also well-adapted to parallelization. In its basic iterative form the method produces an ensemble of solutions which lie in the linear span of the initial ensemble. Choice of the parameterization of the unknown field is thus a key component of the success of the method. We demonstrate how both geometric ideas and hierarchical ideas can be used to design effective parameterizations for a number of applied inverse problems arising in electrical impedance tomography, groundwater flow and source inversion. In particular we show how geometric ideas, including the level set method, can be used to reconstruct piecewise continuous fields, and we show how hierarchical methods can be used to learn key parameters in continuous fields, such as length-scales, resulting in improved reconstructions. Geometric and hierarchical ideas are combined in the level set method to find piecewise constant reconstructions with interfaces of unknown topology.