Tree Learning Research Articles

Change detection and adaptation in multi-target regression on data streams

AbstractAn essential characteristic of data streams is the possibility of occurrence of concept drift, i.e., change in the distribution of the data in the stream over time. The capability to detect and adapt to changes in data stream mining methods is thus a necessity. While methods for multi-target prediction on data streams have recently appeared, they have largely remained without such capability. In this paper, we propose novel methods for change detection and adaptation in the context of incremental online learning of decision trees for multi-target regression. One of the approaches we propose is ensemble based, while the other uses the Page–Hinckley test. We perform an extensive evaluation of the proposed methods on real-world and artificial data streams and show their effectiveness. We also demonstrate their utility on a case study from spacecraft operations, where cosmic events can cause change and demand an appropriate and timely positioning of the space craft.

Decision tree learning for predicting chronic postpartum depression in the Japan Environment and Children's Study

BackgroundMany studies have used machine learning techniques to construct predictive models of postpartum depression, but few such models are simple enough to use in community maternal health settings with pen and paper. Here, we use a decision tree to construct a prediction model for chronic postpartum depression. MethodsParticipants were 84,091 mothers. Chronic postpartum depression was identified as an Edinburgh Postnatal Depression Scale score of ≥9 at both 1 and 6 months postpartum. The training dataset included 84 diverse variables measured during pregnancy, including health status and biomarkers. In learning, the branching depth was constrained to 3, the number of branches per branch to 4, and the minimum number of n in a branch was 100. The training to validation data ratio was set to 7:3. ResultsA decision tree with 35 branches and an area under the receiver operating characteristic of 0.84 was created. Ten of 84 variables were extracted, and the most effective in classification was “feeling worthless.” At training (n = 58,635), the most and least prevalent branches were 73.2 % and 0.84 % (mean = 6.29 %), respectively; at validation (n = 25,456), they were 60.4 % and 0.72 % (mean = 6.52 %), respectively. LimitationsChronic postpartum depression was identified using self-administered questionnaires. ConclusionsThis study created a simple and relatively high-performing prediction model. Because the model can be easily understood and used without expertise in machine learning, it is expected to be useful in maternal health settings, including grassroots community health.

Multiple fairness criteria in decision tree learning

The use of algorithmic decision-making systems based on machine learning models has led to a need for fair (unbiased) and explainable classification outcomes. In particular, machine learning algorithms can encode biases, which might result in discriminatory decisions for certain groups such as gender, race, or age. Although a number of works on decision tree learning have been proposed to decrease the chance of discrimination, they usually focus on the use of a single fairness metric. In general, creating a model based on a single fairness metric is not a sufficient way to mitigate discrimination since bias can originate from various sources—e.g., the data itself or the optimization process. In this paper, we propose a novel decision tree learning process that utilizes multiple fairness metrics to address both group and individual discrimination. This is achieved by extending the attribute selection procedure to consider not only information gain but also gain in fairness. Computational experiments on fourteen different datasets with various sensitive features demonstrate that the proposed Fair-C4.5 models improve fairness without a loss in predictive accuracy when compared to the well-known C4.5 and the fairness-aware FFTree algorithms.

Multi-Elemental Analysis and Geographical Discrimination of Greek "Gigantes Elefantes" Beans Utilizing Inductively Coupled Plasma Mass Spectrometry and Machine Learning Models.

Greek giant beans, also known as "Gigantes Elefantes" (elephant beans, Phaseolus vulgaris L.,) are a traditional and highly cherished culinary delight in Greek cuisine, contributing significantly to the economic prosperity of local producers. However, the issue of food fraud associated with these products poses substantial risks to both consumer safety and economic stability. In the present study, multi-elemental analysis combined with decision tree learning algorithms were investigated for their potential to determine the multi-elemental profile and discriminate the origin of beans collected from the two geographical areas. Ensuring the authenticity of agricultural products is increasingly crucial in the global food industry, particularly in the fight against food fraud, which poses significant risks to consumer safety and economic stability. To ascertain this, an extensive multi-elemental analysis (Ag, Al, As, B, Ba, Be, Ca, Cd, Co, Cr, Cs, Cu, Fe, Ga, Ge, K, Li, Mg, Mn, Mo, Na, Nb, Ni, P, Pb, Rb, Re, Se, Sr, Ta, Ti, Tl, U, V, W, Zn, and Zr) was performed using Inductively Coupled Plasma Mass Spectrometry (ICP-MS). Bean samples originating from Kastoria and Prespes (products with Protected Geographical Indication (PGI) status) were studied, focusing on the determination of elemental profiles or fingerprints, which are directly related to the geographical origin of the growing area. In this study, we employed a decision tree algorithm to classify Greek "Gigantes Elefantes" beans based on their multi-elemental composition, achieving high performance metrics, including an accuracy of 92.86%, sensitivity of 87.50%, and specificity of 96.88%. These results demonstrate the model's effectiveness in accurately distinguishing beans from different geographical regions based on their elemental profiles. The trained model accomplished the discrimination of Greek "Gigantes Elefantes" beans from Kastoria and Prespes, with remarkable accuracy, based on their multi-elemental composition.

Analysis of Public Sentiment Towards Celebrity Endorsment On Social Media Using Support Vector Machine

Analysis of public sentiment towards celebrity endorsements on social media is very important to understand the public's response to promotional campaigns involving celebrities. In this study, we combine the VADER labeling method with the Support Vector Machine (SVM) method to analyze public sentiment toward celebrity endorsements on social media. Data is taken from various social media sources such as Twitter, Instagram, and Facebook. The data is pre-processed to ensure data accuracy and relevance and then labeled with the VADER method to determine the positive, negative, or neutral sentiment of the text. The labeled data is then extracted for features and used to train the SVM model. The trained SVM model is then validated using test data to measure its accuracy and performance. The results of the analysis provide useful insight into public sentiment towards celebrity endorsements on social media and can provide recommendations for stakeholders regarding this matter. Overall, combining the VADER labeling method with SVM in analyzing public sentiment towards celebrity endorsements on social media shows more accurate results and can provide practical benefits in marketing and promotional strategies. The results shown using the Support Vector Machine method with a ratio of 80:20 can provide average precision results of 77%, recall of 100%, f1-score of 87%, and accuracy of 76.92%. Twitter application user sentiment shows that 77% (338 data) of Twitter user reviews provide positive sentiment and 23% (119 data) provide negative sentiment reviews from a total of 517 data. Suggestions from researchers are that in future research they can add more data to make modeling easier to provide higher accuracy values. Using other classification and performance evaluation methods, such as Naive Bayes, Decision Tree, Fuzzy, or Deep Learning. Use other data processing tools, such as RapidMiner, Jupyter Notebook, RStudio, or others.

Spot-checking machine learning algorithms for tool wear monitoring in automatic drilling operations in CFRP/Ti6Al4V/Al stacks in the aircraft industry

In aircraft manufacturing, where diverse materials, including Carbon Fiber-Reinforced Plastics (CFRP), aluminum, and titanium alloys, are employed, the assembly process heavily relies on creating thousands of holes. These holes accommodate bolts and rivets, facilitating the secure interlocking of structural components within the aircraft fuselage. The proliferation of sensor systems in this domain has led to a substantial increase in data generation during the hole-making process, offering a compelling opportunity to optimize the production system. In this context, this article is dedicated to harnessing the data collected from the production system of a commercial aircraft to refine the assembly process, with a specific focus on reducing consumable costs. The primary approach involves developing a real-time Tool Wear Monitoring System by comparing the performance of Linear Regression, Lasso Regression, Ridge Regression, k-Nearest Neighbors, Support Vector Regression, Decision Tree, Random Forest, and Extreme Gradient Boosting Machine Learning models. Using a scale of the general drill condition as an outcome, the Gradient Boosting Regressor has shown outstanding results. Notably, the residuals consistently exhibited zero-centered errors in training and test sets. However, it suggests that further enhancements are needed to surpass human-level performance in predicting tool conditions because of the quality and quantity of available data.

Role of Artificial intelligence model in prediction of low back pain using T2 weighted MRI of Lumbar spine

Background Low back pain (LBP), the primary cause of disability, is the most common musculoskeletal disorder globally and the primary cause of disability. Magnetic resonance imaging (MRI) studies are inconclusive and less sensitive for identifying and classifying patients with LBP. Hence, this study aimed to investigate the role of artificial intelligence (AI) models in the prediction of LBP using T2 weighted MRI image of the lumbar spine. Methods This was a prospective case-control study. A total of 200 MRI patients (100 cases and controls each) referred for lumbar spine and whole spine screening were included. The scans were performed using 3.0 Tesla MRI (United Imaging Healthcare). T2 weighted images of the lumbar spine were segmented to extract radiomic features. Machine learning (ML) models, such as random forest, decision tree, logistic regression, K-nearest neighbors, adaboost, and deep learning methods (DL), such as ResNet and GoogleNet, were used, and performance measures were calculated. Results Our study showed that Random forest and AdaBoost are the most reliable ML models for predicting LBP. Random forest showed high performance with area under curve (AUC) values from 0.83 to 0.88 across all lumbar vertebrae and L2-L3, L3-L4, and L4-L5 intervertebral discs (IVDs), with AUCs of 0.88 the highest at L5-S1 IVD (0.92). Adaboost demonstrated high performance at the L2-L5 vertebrae with AUC values of 0.82 to 0.90, with the highest AUC (0.97) at the L5-S1 IVD. Among the DL models, GoogleNet outperformed the other models at 30 epochs with an accuracy of 0.85, followed by ResNet 18 (30 epochs) with an accuracy of 0.84. Conclusion The study demonstrated that ML and DL models can effectively predict LBP from MRI T2 weighted image of the lumbar spine. ML and DL models could also enhance the diagnostic accuracy of LBP, potentially leading to better patient management and outcomes.

A holistic approach to software fault prediction with dynamic classification

Software Fault Prediction is a critical domain in machine learning aimed at pre-emptively identifying and mitigating software faults. This study addresses challenges related to imbalanced datasets and feature selection, significantly enhancing the effectiveness of fault prediction models. We mitigate class imbalance in the Unified Dataset using the Random-Over Sampling technique, resulting in superior accuracy for minority-class predictions. Additionally, we employ the innovative Ant-Colony Optimization algorithm (ACO) for feature selection, extracting pertinent features to amplify model performance. Recognizing the limitations of individual machine learning models, we introduce the Dynamic Classifier, a ground-breaking ensemble that combines predictions from multiple algorithms, elevating fault prediction precision. Model parameters are fine-tuned using the Grid-Search Method, achieving an accuracy of 94.129% and superior overall performance compared to random forest, decision tree and other standard machine learning algorithms. The core contribution of this study lies in the comparative analysis, pitting our Dynamic Classifier against Standard Algorithms using diverse performance metrics. The results unequivocally establish the Dynamic Classifier as a frontrunner, highlighting its prowess in fault prediction. In conclusion, this research introduces a comprehensive and innovative approach to software fault prediction. It pioneers the resolution of class imbalance, employs cutting-edge feature selection, and introduces dynamic ensemble classifiers. The proposed methodology, showcasing a significant advancement in performance over existing methods, illuminates the path toward developing more accurate and efficient fault prediction models.

Decision tree learning with random forest models using agricultural and ecological field data incorporating multi-factor studies and covariate structure

Decision tree learning with random forest models using agricultural and ecological field data incorporating multi-factor studies and covariate structure

Numerical and machine learning modeling of GFRP confined concrete-steel hollow elliptical columns

This article investigates the behavior of hybrid FRP Concrete-Steel columns with an elliptical cross section. The investigation was carried out by gathering information through literature and conducting a parametric study, which resulted in 116 data points. Moreover, multiple machine learning predictive models were developed to accurately estimate the confined ultimate strain and the ultimate load of confined concrete at the rupture of FRP tube. Decision Tree (DT), Random Forest (RF), Adaptive Boosting (ADAB), Categorical Boosting (CATB), and eXtreme Gradient Boosting (XGB) machine learning techniques were utilized for the proposed models. Finally, these models were visually and quantitatively verified and evaluated. It was concluded that the CATB and XGB are standout models, offering high accuracy and strong generalization capabilities. The CATB model is slightly superior due to its consistently lower error rates during testing, indicating it is the best model for this dataset when considering both accuracy and robustness against overfitting.

Increasing trust in AI through privacy preservation and model explainability: Federated Learning of Fuzzy Regression Trees

Federated Learning (FL) lets multiple data owners collaborate in training a global model without any violation of data privacy, which is a crucial requirement for enhancing users’ trust in Artificial Intelligence (AI) systems. Despite the significant momentum recently gained by the FL paradigm, most of the existing approaches in the field neglect another key pillar for the trustworthiness of AI systems, namely explainability. In this paper, we propose a novel approach for FL of fuzzy regression trees (FRTs), which are generally acknowledged as highly interpretable by-design models. The proposed FL procedure is designed for the scenario of horizontally partitioned data and is based on the transmission of aggregated statistics from the clients to a central server for the tree induction procedure. It is shown that the proposed approach faithfully approximates the ideal case in which the tree induction algorithm is applied on the union of all local datasets, while still ensuring privacy preservation. Furthermore, the FL approach brings benefits, in terms of generalization capability, compared to the local learning setting in which each participant learns its own FRT based only on the private, local, dataset. The adoption of linear models in the leaf nodes ensures a competitive level of performance, as assessed by an extensive experimental campaign on benchmark datasets. The analysis of the results covers both the aspects of accuracy and interpretability of FRT. Finally, we discuss the application of the proposed federated FRT to the task of Quality of Experience forecasting in an automotive case-study.

Online Learning Based on Learning Analytics in Big Data for College English Language Teaching

This study addresses the challenges of enhancing the quality of education and improving the overall student experience in online English language teaching sessions. Current approaches often struggle with session initiation, real-time data processing, and personalized learning experiences. To tackle these issues, the study proposes a manifold learning data analytics model (MLDAM). This innovative method leverages classifier tree learning to distinguish between trivial and non-trivial aspects of student learning experiences and session data. MLDAM integrates multi-dimensional data extraction, classification learning, and impact evaluation to enhance the effectiveness of online English language teaching. The model analyzes data from 176,009 English phrases across 36 online teaching sessions, focusing on improving session accessibility, student comprehension, and suggestion accuracy. Using an iterative training process based on student performance and feedback, it continuously extracts and processes multiple types of data to refine teaching strategies. Results show notable improvements: a 14.74% increase in classification accuracy, an 8.73% increase in data extraction ratio, an 11.84% reduction in feature discard, a 10.57% decrease in initialization time, and a 13.17% reduction in classification time. These metrics demonstrate MLDAM’s ability to efficiently process and analyze session data, enabling real-time adjustments during lessons. The model optimizes data utilization, making learning more responsive and adaptable. It enhances the precision of impact evaluations, facilitating targeted course adjustments and personalized learning experiences. This research presents a comprehensive, data-driven methodology for improving teaching quality and student outcomes in virtual English classrooms.

SAT-based Decision Tree Learning for Large Data Sets

Decision trees of low depth are beneficial for understanding and interpreting the data they represent. Unfortunately, finding a decision tree of lowest complexity (depth or size) that correctly represents given data is NP-hard. Hence known algorithms either (i) utilize heuristics that do not minimize the depth or (ii) are exact but scale only to small or medium-sized instances. We propose a new hybrid approach to decision tree learning, combining heuristic and exact methods in a novel way. More specifically, we employ SAT encodings repeatedly to local parts of a decision tree provided by a standard heuristic, leading to an overall reduction in complexity. This allows us to scale the power of exact SAT-based methods to comparatively very large data sets. We evaluate our new approach experimentally on a range of real-world instances that contain up to several thousand samples. In almost all cases, our method successfully decreases the complexity of the initial decision tree; often, the decrease is significant.

MEDIA POHON ANGKA DAN PENGGUNAANYA DALAM PEMBELAJARAN ASPEK KOGNITIF ANAK USIA DINI

The purpose of this developmental research is to create a number tree learning media for the cognitive aspect of recognizing and naming numbers or numerical symbols in Ade Irma Kindergarden. Additionally, this study aims to assess the feasibility of using the number tree learning media and obtain feedback on its usage from teachers at Ade Irma Kindergarden and the course instructor. The research method uses descriptive qualitative methods with data collection techniques through observation, interviews and documentation. The research subjects were children aged 4-6 years.The research results, especially from the aspect of cognitive development, show development that is in line with expectations. Based on the results of research and validation by experts, the number tree media developed in this research is very suitable for use in learning cognitive aspects of early childhood. These findings support the importance of developing contextual learning media that suit the needs and characteristics of young children. It is hoped that the implementation of number tree media can improve the quality of learning at Ade Irma Mataloko Kindergarten and provide a model for the development of learning media in other places.

Smartphone technology adoption of university students for distance learning in Pakistan: COVID-19 data-driven decision tree learning

Smartphone technology adoption of university students for distance learning in Pakistan: COVID-19 data-driven decision tree learning

Cost‐sensitive tree SHAP for explaining cost‐sensitive tree‐based models

AbstractCost‐sensitive ensemble learning as a combination of two approaches, ensemble learning and cost‐sensitive learning, enables generation of cost‐sensitive tree‐based ensemble models using the cost‐sensitive decision tree (CSDT) learning algorithm. In general, tree‐based models characterize nice graphical representation that can explain a model's decision‐making process. However, the depth of the tree and the number of base models in the ensemble can be a limiting factor in comprehending the model's decision for each sample. The CSDT models are widely used in finance (e.g., credit scoring and fraud detection) but lack effective explanation methods. We previously addressed this gap with cost‐sensitive tree Shapley Additive Explanation Method (CSTreeSHAP), a cost‐sensitive tree explanation method for the single‐tree CSDT model. Here, we extend the introduced methodology to cost‐sensitive ensemble models, particularly cost‐sensitive random forest models. The paper details the theoretical foundation and implementation details of CSTreeSHAP for both single CSDT and ensemble models. The usefulness of the proposed method is demonstrated by providing explanations for single and ensemble CSDT models trained on well‐known benchmark credit scoring datasets. Finally, we apply our methodology and analyze the stability of explanations for those models compared to the cost‐insensitive tree‐based models. Our analysis reveals statistically significant differences between SHAP values despite seemingly similar global feature importance plots of the models. This highlights the value of our methodology as a comprehensive tool for explaining CSDT models.

A novel ensemble approach for Twitter sentiment classification with ML and LSTM algorithms for real-time tweets analysis

Social media sentiment classification was an essential consideration in natural language processing (NLP) for evaluating normal people’s perspectives on a given topic. With Twitter’s massive rise in popularity in recent years, the capacity to extract information about public sentiment from tweets became a major focus. This paper not only analyzed public sentiment through data from Twitter but introduced a novel ensemble approach in the methods employed for Twitter sentiment classification. Real-time tweets on various topics, including “covid,” “crime,” “spam,” “flipkart,” “migraine,” and “airlines,” were extracted and thoroughly examined to gain insight into public opinions. Leveraging the Twitter API for real-time tweet extraction, natural language processing techniques were applied to clean the tweet data. Subsequently, we applied several machine learning (ML) algorithms Naïve Bayes, decision tree (DT), random forest (RF), logistic regression (LGR), and deep learning (DL) algorithms recurrent neural network (RNN), LSTM, and GRU individually. Later, we proposed a novel ensemble of ML and DL algorithms for sentiment classification, with a novel emphasis on ensemble techniques and enhanced the accuracy with a significance compared to individual ML or DL model applied. The experimental results demonstrated that our novel ensemble approach achieved high accuracy when compared to existing work.

Smart Data Driven Decision Trees Ensemble Methodology for Imbalanced Big Data

Differences in data size per class, also known as imbalanced data distribution, have become a common problem affecting data quality. Big Data scenarios pose a new challenge to traditional imbalanced classification algorithms, since they are not prepared to work with such amount of data. Split data strategies and lack of data in the minority class due to the use of MapReduce paradigm have posed new challenges for tackling the imbalance between classes in Big Data scenarios. Ensembles have been shown to be able to successfully address imbalanced data problems. Smart Data refers to data of enough quality to achieve high-performance models. The combination of ensembles and Smart Data, achieved through Big Data preprocessing, should be a great synergy. In this paper, we propose a novel Smart Data driven Decision Trees Ensemble methodology for addressing the imbalanced classification problem in Big Data domains, namely SD_DeTE methodology. This methodology is based on the learning of different decision trees using distributed quality data for the ensemble process. This quality data is achieved by fusing random discretization, principal components analysis, and clustering-based random oversampling for obtaining different Smart Data versions of the original data. Experiments carried out in 21 binary adapted datasets have shown that our methodology outperforms random forest.

Active intrusion detection and prediction based on temporal big data analytics

Computer security consists in protecting access and manipulating system data by several mechanisms. However, conventional protection technologies are ineffective against current attacks. Thus, new tools have appeared, such as the intrusion detection and prediction systems which are important defense elements for network security since they detect the ongoing intrusions and predict the upcoming attacks. Besides, most of conventional protection technologies remain insufficient in terms of actions since they are all passive systems, unable to provide recommendations in order to block or stop the attacks. In this paper, a distributed detection and prediction system, composed of three major parts, is proposed. The first part deals with the detection of intrusions based on the decision tree learning algorithm. The second part deals with intrusions prediction using the chronicle algorithm. The third part proposes an expert system for security recommendations in response to detected intrusions, able to provide appropriate recommendations to stop the attacks. The proposed system gives good results in terms of accuracy and precision in detecting and predicting attacks, and efficiency in proposing the right recommendations to stop the attacks.

Revolutionizing Anemia Classification with Multilayer Extremely Randomized Tree Learning Machine for Unprecedented Accuracy

Anemia is a prevalent global health issue that is characterized by a deficit in red blood cells or low levels of hemoglobin. This condition is influenced by various causes, including nutritional inadequacies, chronic diseases, and genetic predisposition. The incidence of the phenomenon exhibits variation across different geographical regions and demographic groups. This pioneering research investigates the identification and classification of anemia, potentially leading to transformative advancements in the discipline. The classification of anemia encompasses four distinct groups, namely Beta Thalassemia Trait, Iron Deficiency Anemia, Hemoglobin E, and Combination. This comprehensive categorization offers clinicians a more refined and detailed comprehension of the condition. The integration of deep learning and machine learning in the Multilayer Extremely Randomized Tree Learning Machine (MERTLM) model represents a departure from traditional approaches and a significant advancement in the field of medical categorization accuracy. The MERTLM approach integrates randomized tree with multilayer extreme learning machine (M-ELM) representation learning, hence emphasizing the possibility of interdisciplinary collaboration in the field of diagnostics. In addition to its impact on anemia, artificial intelligence (AI) is playing a significant role in revolutionizing medical diagnosis by emphasizing the integration of innovative methods. This study utilizes the combined capabilities of machine learning and deep learning to improve accuracy. Notably, recent developments have resulted in an exceptional accuracy rate of 99.67%, precision of 99.60%, sensitivity of 99.47%, and an amazing F1-Score of 99.53%. This study represents a significant advancement in the field of anemia research, providing valuable insights that may be applied to intricate medical issues and enhancing the quality of patient care.