Ensemble Of Regression Trees Research Articles

Development of a recommendation engine for university study programme selection: A regression-based approach

This paper is the third in a series focused on bridging the gap between secondary and higher education. Our primary objective is to develop a robust theoretical framework for an innovative e-business model called the Undergraduate Study Programme Search System (USPSS). This system considers multiple criteria to reduce the likelihood of exam failure or the need for multiple retakes, while maximizing the chances of successful program completion. Testing of the proposed algorithm demonstrated that the Stochastic Gradient Boosted Regression Trees method outperforms the current method used in Lithuania for admitting applicants to 47 educational programs. Specifically, it is more accurate than the Probabilistic Neural Network for 25 programs, the Ensemble of Regression Trees for 24 programs, the Single Regression Tree for 18 programs, the Random Forest Regression for 16 programs, the Bayesian Additive Regression Trees for 13 programs, and the Regression by Discretization for 10 programs.

Prediction of Flotation Deinking Performance: A Comparative Analysis of Machine Learning Techniques

Flotation deinking is one of the most widely used techniques for the separation of ink particles from cellulose fibers during the process of paper recycling. It is a complex process influenced by a variety of factors, and is difficult to represent and usually results in models that are inconvenient to implement and/or interpret. In this paper, a comprehensive study of several machine learning methods for the prediction of flotation deinking performance is carried out, including support vector regression, regression tree ensembles (random forests and boosting) and Gaussian process regression. The prediction relies on the development of a limited dataset that assumes representative data samples obtained under a variety of laboratory conditions, including different reagents, pH values and flotation residence times. The results obtained in this paper confirm that the machine learning methods enable the accurate prediction of flotation deinking performance even when the dataset used for training the model is limited, thus enabling the determination of optimal conditions for the paper recycling process, with only minimal costs and effort. Considering the low complexity of the Gaussian process regression compared to the aforementioned ensemble models, it should be emphasized that the Gaussian process regression gave the best performance in estimating fiber recovery (R2 = 97.77%) and a reasonable performance in estimating the toner recovery (R2 = 86.31%).

Human-AI collaboration improves adults’ oral biomechanical functions: A multi-centre, self-controlled clinical trial

ObjectivesMaintenance of oral muscle functions is important for survival and communication. Utilizing Artificial Intelligence (AI) as a self-health-management material has shown promise. Here we developed a functional and AI-enabled smartphone e-Oral APP that provided real-time feedback features to let humans collaborate with AI, targeting to improve oral biomechanical and speech functions of adults. MethodsA total of 113 neurologically healthy Hong Kong citizens over 50 years old were recruited in this study from June to October 2022. A set of 12 exercises of the tongue, lips, and jaw were conducted with guidance provided in the self-developed e-Oral APP, in which visual-audio demonstrations and immediate feedback on facial movements by an open-source AI-pretrained 68-point Ensemble of Regression Tree (ERT) face landmark detector was given to the users. After 8-week exercises, assessments after 1-week and 4-week post-training were conducted. Primary outcomes were measured on oral functions, including occlusal force, masticatory efficiency, tongue strength, along with dry mouth condition, and oral health-related quality of life (OHQoL), while secondary outcomes were comprised of oral diadochokinesis (DDK) and swallowing ability. The usability of the APP was evaluated by a self-written questionnaire. ResultsAfter exercising a set of 12 exercises on the tongue, lips, and jaw for 8 weeks, 70 out of 113 participants (average age of 67.70±4.93 y.o., 52 female and 18 male) had a significant improvement in the occlusal force (p < 0.001), while masticatory efficiency (p = 0.002), tongue pressure (p < 0.001) and endurance (p = 0.004) were also improved. 80.3 % of respondents rated the APP with an overall rating of 3 out of 5 points, and 68.6 % would recommend the APP to others. ConclusionsAI-based APP can be an effective approach to help healthy adults improve their occlusal force, masticatory efficiency, tongue functions, and oral diadochokinesis after 8-week home oral exercises. Furthermore, these improvements can be sustained for at least four weeks. Clinical significanceThis is the first AI-assisted APP developed for oral muscle training. Our findings demonstrated that a self-administrated human-AI collaboration APP can improve clinically oral muscle biomechanics and functions for healthy adults. AI technology in smartphone gadgets provides a cost-effective, convenient, and reliable means for oral muscles training for adults.

Modeling joint health effects of temperature and air pollution mixtures using Bayesian regression tree ensembles

Modeling joint health effects of temperature and air pollution mixtures using Bayesian regression tree ensembles

Seeing beyond the symptoms: biomarkers and brain regions linked to cognitive decline in Alzheimer's disease.

Early Alzheimer's disease (AD) diagnosis remains challenging, necessitating specific biomarkers for timely detection. This study aimed to identify such biomarkers and explore their associations with cognitive decline. A cohort of 1759 individuals across cognitive aging stages, including healthy controls (HC), mild cognitive impairment (MCI), and AD, was examined. Utilizing nine biomarkers from structural MRI (sMRI), diffusion tensor imaging (DTI), and positron emission tomography (PET), predictions were made for Mini-Mental State Examination (MMSE), Clinical Dementia Rating Scale Sum of Boxes (CDRSB), and Alzheimer's Disease Assessment Scale-Cognitive Subscale (ADAS). Biomarkers included four sMRI (e.g., average thickness [ATH]), four DTI (e.g., mean diffusivity [MD]), and one PET Amyloid-β (Aβ) measure. Ensemble regression tree (ERT) technique with bagging and random forest approaches were applied in four groups (HC/MCI, HC/AD, MCI/AD, and HC/MCI/AD). Aβ emerged as a robust predictor of cognitive scores, particularly in late-stage AD. Volumetric measures, notably ATH, consistently correlated with cognitive scores across early and late disease stages. Additionally, ADAS demonstrated links to various neuroimaging biomarkers in all subject groups, highlighting its efficacy in monitoring brain changes throughout disease progression. ERT identified key brain regions associated with cognitive scores, such as the right transverse temporal region for Aβ, left and right entorhinal cortex, left inferior temporal gyrus, and left middle temporal gyrus for ATH, and the left uncinate fasciculus for MD. This study underscores the importance of an interdisciplinary approach in understanding AD mechanisms, offering potential contributions to early biomarker development.

A Novel DWT-ERT-based Fault Location for Distribution Network

Background: A new DWT-ERT-based fault location method is suggested in the IEEE test feeder. Objective: The fault location approach in the distribution network has been proposed in this pa-per that utilizes the discrete wavelet transform (DWT) and ensemble regression tree (ERT). Methods: The fault location methodology has been validated by simulations conducted on an IEEE 13 bus node test feeder. Results: The results show that the suggested solution has low compute burden and memory re-quirements, and is unaffected by system and fault situations. Conclusion: In this study, the fault location approach for the distribution system employing DWT and ERT has been proposed.

A Theoretical Study of the Representational Power of Weighted Randomised Univariate Regression Tree Ensembles

Univariate regression trees have representation problems for non-orthogonal regression functions. Ensembles of univariate regression trees have better representational power. In some cases, weighted ensembles have shown better performance than unweighted ensembles. In this paper, we study the properties of ensembles of regression trees by using regression classification models. We propose a theoretical framework to study the representational power of infinite-sized weighted ensembles, consisting of randomised finite-sized regression trees. We show for some datasets that the weighted ensembles may have better representational power than unweighted ensembles, but the performance is highly dependent on the weighting scheme and the properties of datasets. Our model cannot be used for all the datasets. However, for some datasets, we can accurately predict the experimental results of ensembles of regression trees.

Machine Learning-Assisted design of boron and nitrogen doped graphene nanosheets with tailored thermomechanical properties

Doping graphene with boron and/or nitrogen enhances its potential for various applications, such as electronics and energy devices. However, these modifications impact the material's properties, influencing its response to external forces, temperature, and heating. Investigating the connection between the structure of co-doped graphene nanosheets and their thermomechanical properties is crucial for accurate predictions and tailored nanosheet engineering. In the present study, molecular dynamics (MD) simulations were performed to compute the thermal and mechanical properties of 300 randomly generated boron/nitrogen co-doped graphene nanosheets with doping concentrations in the range of 0%-15%. Next, the obtained MD results were used to train, tune, and validate a wide spectrum of supervised machine learning models, including linear regression models, regression trees, support vector machines, Gaussian process regression models, kernel approximation models, ensembles of regression trees, and regression neural networks. Our aim is to find models that better estimate, without overfitting, the structure-properties relation of boron/nitrogen co-doped nanosheets. To this end, the models with the highest coefficient of determination (R2) and the lowest root mean square error (RMSE) were utilized. Finally, an appropriate implementation of guided random search algorithms, mainly Genetic Algorithms, was used to solve the inverse problem, i.e., find the nanosheet structure which exhibits the desired set of property values. To this end, thermal conductivity, Young’s modulus, failure stress, and failure strain were specified as target values. A detailed comparison between the employed models and their validation via MD simulations is also included. The achieved results provide sufficient evidence of the accuracy and viability of the applied procedure in both prediction of thermomechanical properties as well as identification of structures with desired properties. This study provides a roadmap that could help scientists and engineers develop nanosheets-based materials with tunable thermomechanical properties, ultimately leading to the creation of innovative and high-performance technologies.

A Review on the Recent Advancements and Artificial Intelligence in Tablet Technology.

Tablet formulation could be revolutionized by the integration of modern technology and established pharmaceutical sciences. The pharmaceutical sector can develop tablet formulations that are not only more efficient and stable but also patient-friendly by utilizing artificial intelligence (AI), machine learning (ML), and materials science. The primary objective of this review is to explore the advancements in tablet technology, focusing on the integration of modern technologies like artificial intelligence (AI), machine learning (ML), and materials science to enhance the efficiency, cost-effectiveness, and quality of tablet formulation processes. This review delves into the utilization of AI and ML techniques within pharmaceutical research and development. The review also discusses various ML methodologies employed, including artificial neural networks, an ensemble of regression trees, support vector machines, and multivariate data analysis techniques. Recent studies showcased in this review demonstrate the feasibility and effectiveness of ML approaches in pharmaceutical research. The application of AI and ML in pharmaceutical research has shown promising results, offering a potential avenue for significant improvements in the product development process. The integration of nanotechnology, AI, ML, and materials science with traditional pharmaceutical sciences presents a remarkable opportunity for enhancing tablet formulation processes. This review collectively underscores the transformative role that AI and ML can play in advancing pharmaceutical research and development, ultimately leading to more efficient, reliable and patient-centric tablet formulations.

Modeling injury severity of crashes involving golf carts: A case study of The Villages, Florida

Objective Crashes involving golf carts (GCs) have been on an increasing trend in recent years, particularly in the United States. This study focuses on analyzing GC crashes in the Florida community known as The Villages, one of the largest GC-oriented communities in the nation and worldwide. The objective was to evaluate the injury severity of crashes involving GCs in a retirement community where GCs are a common mode of transportation. Methods The ordinal logistic regression (OLR) and Decision Tree Ensemble (DTE) models were used to analyze the injury severity of 616 GC-related crashes. Models’ accuracy parameters were used to check their reliability. Results The analysis revealed that GC crash severity is influenced by various factors. Factors found to be significant by the OLR model in determining injury severity include ejection of one or more occupants from the GC, the extent of damage to the GC, GC speed prior to the crash, roadway characteristics (including divided roadways, traffic control devices, paved shoulders, and T-intersections), and roll-over incidents. The OLR model demonstrated an overall accuracy of approximately 71% in predicting injury severity. The DTE model performed better, with an overall accuracy of 78%. The OLR model’s findings were supported by the DTE model, which identified estimated GC speed, occupant(s) ejection from the GC, estimated GC vehicle damage, intersection type, and type of shoulder as the most important factors influencing GC crash severity. Conclusions Understanding these factors is vital for transportation agencies to develop effective strategies to reduce the severity of GC crashes, ensuring the safety of GC users. This study provides recommendations to transportation agencies on measures to improve the safety of GCs.

Utilizing a Machine Learning Approach to Define Racial Difference in Multi-Systemic Impact of Monoclonal Protein in Patient with Multiple Myeloma

Multiple myeloma (MM) is the second common hematologic malignancy characterized by the clonal proliferation of plasma cells. While the clinical outcomes of MM have been improved significantly, there remains an intriguing observation regarding the potential influence of race on disease biology and treatment response. In MM, the malignant plasma cell clone not only disrupts the local bone marrow microenvironment but also exerts a systemic impact on various physiological processes result in significant derangements in immune function, hematopoiesis, renal function, bone metabolism, and electrolyte homeostasis. Understanding the extent of this systemic impact and its potential variation between racial groups is crucial for optimizing patient care and treatment strategies. Here, we hypothesized that the systemic impact of the plasma cell clone does not differ significantly between African American and Caucasian patients. We employed a machine learning approach to capture the full spectrum of systemic effects comprehensively. Our novel model was trained to predict the magnitude of the plasma cell clone, as represented by serum monoclonal protein levels. By systematically comparing model performance with and without the inclusion of race as a variable, we sought to evaluate the potential influence of race on the systemic impact of the plasma cell clone. Method: A total of 171 patients with plasma cell dyscrasias including 1,472 observations were analyzed. where the upper limit of the observed M-spike was 3.5 gr/dL. Forty three clinical and lab variables as predictors of M-spike were fed into the machine learning model. Two lagged variables as the last two preceding M-spike values by the same subject were included. The random forest model was used, where regression forests are an ensemble of different regression trees and are used for nonlinear multiple regression. The goal of using a large number of trees was to train enough that each feature had a chance to appear in several models. The data was randomly split into a training set (80%) and a test set (20%), and a regression tree was built with the training set and then validated using the test set. Bootstrapping was used to generate a collection of data sets (n=500). Importance was measured by leaving a covariate out of models and comparing performance with its inclusion. Result: The training set used to develop the ML algorithm comprised 749 patient-based observations, which represented ~50% of the patient-based observations. The number of observations at each M-Spike value and the distribution of M-Spike values measured by SPEP are shown (Fig. 1). The residual distribution of the RF model indicated that nearly all M-spike values determined using the 43 variables distributed equally on either side of zero. The addition of race as variables did not significantly change the residual plots. The weighted value of each of the 43 independent variables was determined by individually removing a variable from the ML algorithm and measuring its effect on the mean squared error (MSE). As shown, removal of the first lagged M-spike, serum total protein, second-lagged M-spike, serum IgG, serum IgM, and serum IgA, had the greatest effects on the ML algorithm. The effect of race was then added to the ML algorithm. However, patient race had a low weighted value compared to the other variables included in the ML algorithm (Fig-2). M-spike values determined using the ML algorithm correlated highly with M-spike values determined using the laboratory measured SPEP values as indicated by the proximity of the Pearson and Spearman correlation coefficients to +1 . Using the 43 independent variables, the Pearson coefficient was 0.96 and the Spearman coefficient was 0.91, when M-spike values determined using the ML algorithm were compared to laboratory determined M-spike values. When race was added as a variable in the ML algorithm, the Pearson coefficient was 0.95 and the Spearman coefficient was 0.94 which is not significantly different. Conclusion: Here, we present the results of our machine learning analysis, which further supports our hypothesis and sheds light on the potential racial difference in the overall systemic impact of MM. Understanding the factors underlying racial disparities in MM will pave the way for more personalized and equitable treatment strategies, ultimately improving patient outcomes and overall disease management.

Tree canopy extent and height change in Europe, 2001–2021, quantified using Landsat data archive

European forests are among the most extensively studied ecosystems in the world, yet there are still debates about their recent dynamics. We modeled the changes in tree canopy height across Europe from 2001 to 2021 using the multidecadal spectral data from the Landsat archive and calibration data from Airborne Laser Scanning (ALS) and spaceborne Global Ecosystem Dynamics Investigation (GEDI) lidars. Annual tree canopy height was modeled using regression tree ensembles and integrated with annual tree canopy removal maps to produce harmonized tree height map time series. From these time series, we derived annual tree canopy extent maps using a ≥ 5 m tree height threshold. The root-mean-square error (RMSE) for both ALS-calibrated and GEDI-calibrated tree canopy height maps was ≤4 m. The user's and producer's accuracies estimated using reference sample data are ≥94% for the tree canopy extent maps and ≥ 80% for the annual tree canopy removal maps. Analyzing the map time series, we found that the European tree canopy extent area increased by nearly 1% overall during the past two decades, with the largest increase observed in Eastern Europe, Southern Europe, and the British Isles. However, after the year 2016, the tree canopy extent in Europe declined. Some regions reduced their tree canopy extent between 2001 and 2021, with the highest reduction observed in Fennoscandia (3.5% net decrease). The continental extent of tall tree canopy forests (≥ 15 m height) decreased by 3% from 2001 to 2021. The recent decline in tree canopy extent agrees with the FAO statistics on timber harvesting intensification and with the increasing extent and severity of natural disturbances. The observed decreasing tree canopy height indicates a reduction in forest carbon storage capacity in Europe.

The stratified K-folds cross-validation and class-balancing methods with high-performance ensemble classifiers for breast cancer classification

Breast cancer is one of the most common causes of death among women, and early diagnosis is vital for reducing the fatality rate. This study evaluates the most widely used machine-learning breast cancer prediction and diagnosis methods. We use synthetic minority over-sampling to handle imbalanced data in the breast cancer diagnosis dataset obtained from the Wisconsin Machine Learning Repository. We use a variety of machine learning algorithms, including Logistic Regression (LR), Support Vector Machine (SVM), K-Nearest Neighbours (KNN), Classification and Regression Tree (CART), Naive Bayes (NB), and well-known ensembles methods like Majority-Voting, eXtreme Gradient Boosting algorithm (XGBoost), and Random Forest (RF) for the breast cancer classification. The findings show that the Majority-Voting ensemble method, built on the top three classifiers (LR, SVM, and CART), outperforms all other individual classifiers and offers the highest accuracy of 99.3%.

Measurement Method of Interpupillary Distance and Pupil Height Based on Ensemble of Regression Trees and the BlendMask Algorithm

Measuring interpupilary distance and pupil height is a crucial step in the process of optometry. However, existing methods suffer from low accuracy, high cost, a lack of portability, and limited research on studying both parameters simultaneously. To overcome these challenges, we propose a method that combines ensemble regression trees (ERT) with the BlendMask algorithm to accurately measure both interpupillary distance and pupil height. First, we train an ERT-based face keypoint model to locate the pupils and calculate their center coordinates. Then, we develop an eyeglass dataset and train a BlendMask model to obtain the coordinates of the lowest point of the lenses. Finally, we calculate the numerical values of interpupillary distance and pupil height based on their respective definitions. The experimental results demonstrate that the proposed method can accurately measure interpupillary distance (IPD) and pupil height, and the calculated IPD and pupil height values are in good agreement with the measurements obtained by an auto-refractometer. By combining the advantages of the two models, our method overcomes the limitations of traditional methods with high measurement accuracy, low cost, and strong portability. Moreover, this method enables fast and automatic measurement, minimizing operation time, and reducing human errors. Therefore, it possesses broad prospects for application, particularly in the fields of eyeglass customization and vision inspection.

A predictive model for determining the nitrite concentration in the effluent of an anammox reactor using ensemble regression tree algorithm

Anaerobic ammonium oxidation (anammox) is a cost-effective biological nitrogen removal method for treating wastewater. Nitrite has strong negative effect on microbial activity of anammox bacteria, while the conventional equitment available for determining nitrite on-line is challenging due to high price. By knowing the concentration of nitrite in the effluent, its concentration in the reactor can be controlled accordingly. To investigate this, an ensemble regression tree algorithm was used to establish the predictive model proposed in the current work. Moreover, the Bayesian algorithm was adopted to systematically optimize various parameters of machine learning algorithms. The predicted concentrations of nitrite were in good agreement with the observed values, and the coefficient of determination (R2) and root mean squared error (RMSE) values reached 0.91 and 4.81, respectively. Furthermore, the model established by the ensemble regression tree algorithm was compared with models established by commonly used machine learning algorithms. Finally, the established models were applied to another anammox reactor, and the predicted results of ensemble regression tree model were found to be in good agreement with the experimental values with R2 and RMSE values of 0.84 and 6.34, respectively.

A High Performance and Robust FPGA Implementation of a Driver State Monitoring Application.

A high-performance Driver State Monitoring (DSM) application for the detection of driver drowsiness is presented in this paper. The popular Ensemble of Regression Trees (ERTs) machine learning method has been employed for the alignment of 68 facial landmarks. Open-source implementation of ERTs for facial shape alignment has been ported to different platforms and adapted for the acceleration of the frame processing speed using reconfigurable hardware. Reducing the frame processing latency saves time that can be used to apply frame-to-frame facial shape coherency rules. False face detection and false shape estimations can be ignored for higher robustness and accuracy in the operation of the DSM application without sacrificing the frame processing rate that can reach 65 frames per second. The sensitivity and precision in yawning recognition can reach 93% and 97%, respectively. The implementation of the employed DSM algorithm in reconfigurable hardware is challenging since the kernel arguments require large data transfers and the degree of data reuse in the computational kernel is low. Hence, unconventional hardware acceleration techniques have been employed that can also be useful for the acceleration of several other machine learning applications that require large data transfers to their kernels with low reusability.

HardGBM: A Framework for Accurate and Hardware-Efficient Gradient Boosting Machines

Gradient boosting machine (GBM) is a powerful and widely used type of ensemble machine learning methods, among which the most famous one is XGBoost. However, the cost of running large GBMs on hardware could become prohibitive given stringent resources. Ensemble reduction on boosting ensemble is intrinsically hard because member models are constructed in a sequential order, where the training targets for latter ones depend on the performance of the former ones. In this work, a GBM reduction framework is proposed for the first time to tackle the problem. For the first time, the framework supports automatic hardware implementation of regression tree ensembles. Experiments on 24 datasets from various applications demonstrate that our method reduces overall area utilization by 81.60% (80.64%) and power consumption by 21.15% (19.06%), while exceeding or successfully maintaining the performance level comparing with the original XGBoost (LightGBM) ensembles. In comparative experiments, to attain approximately the same accuracy level as our framework or XGBoost, deep learning-based solutions require >52.7× footprints, 6.0× power consumption, and 1.4× training time. Equipped with tunable parameters, the framework is expected to seek a Pareto optimal front considering hardware resource limitation, accuracy and stability, and computation (training) efficiency.

A novel LOF-based ensemble regression tree methodology

A novel LOF-based ensemble regression tree methodology

Expanded feature space-based gradient boosting ensemble learning for risk prediction of type 2 diabetes complications

As a metabolic disease, diabetes is a serious threat to human health, in which type 2 diabetes (T2D) constitutes about 90% of diabetic cases. As the disease progresses, T2D often comes with various complications such as kidney disease and retinopathy, which may cause huge threats to patients’ health. In recent years, complication prediction of T2D has gained wide research interest as early disease detection and management can effectively reduce the risk of death. In this paper, to break through the limitation of the available data and achieve higher overall prediction accuracy, an improved feature space-based gradient boosting regression tree ensemble (IFS-GBRTE) approach is proposed for the risk prediction of developing T2D complications. Specifically, the original feature space is expanded based on existing feature generation theories. Then the new feature space is refined by comparing the total contribution of each feature during the construction of a classification and regression tree (CART) under a cross-validation scheme. Finally, a gradient boosting ensemble algorithm using CART as base learners is utilized for model training. The prediction model is validated on the T2D data provided by the China National Clinical Medical Science Data Center. The experimental results show that the proposed IFS-GBRTE achieves 82.49% accuracy and realizes better generalization ability than compared single models and ensemble learning models, meanwhile it is helpful to obtain more effective predictive variables for improving the accuracy of risk prediction of T2D complications, which is of great significance for achieving the early prevention, screening and nursing of complications, and further reducing mortality and save medical resources.

Cardiovascular Disease Prediction Based on Soft Voting Ensemble Model

In recent years, cardiovascular disease has become a serious threat to the health and safety of people all over the world. Machine learning, deep learning and other artificial intelligence (AI) technologies used to assist medical diagnosis are becoming more and more popular. In order to improve the performance of cardiovascular disease prediction, this paper proposes an ALD soft voting ensemble model (ALD-SVE), which is composed of three individual learners, Attentional Factorization Machines (AFM) can make full use of the cross features of cardiovascular disease data and capture. The attention mechanism introduced in the AFM model gives different weights to cross features to enhance interpretation, and then uses the soft voting ensemble of Logistic Regression (LR) and Decision Tree (DT) to further enhance the stability and generalization of the model. Experiments show that the ALD-SVE model has excellent performance in cardiovascular disease data sets, and its AUC value reached 0.80306, better than the listed comparison model.