Multi-level Machine Learning Research Articles

Permanent pastures identification in Portugal using remote sensing and multi-level machine learning

IntroductionThe Common Agricultural Policy (CAP) is a vital policy framework implemented by the European Union to regulate and support agricultural production within member states. The Land Parcel Identification System (LPIS) is a key component that provides reliable land identification for administrative control procedures. On-the-spot checks (OTSC) are carried out to verify compliance with CAP requirements, typically relying on visual interpretation or field visits. However, the CAP is embracing advanced technologies to enhance its efficiency.MethodsThis study focuses on using Sentinel-2 time series data and a two-level approach involving recurrent neural networks (RNN) and convolutional neural networks (CNN) to accurately identify permanent pastures.ResultsIn the first step, using RNN, the model achieved an accuracy of 68%, a precision of 36%, a recall of 97% and a F1-score of 52%, which indicates the model’s ability to identify all the true positive parcels (correctly identified permanent pasture parcels) and minimize the false negative parcels (non-identified permanent pasture parcels). This occurs due to the difficulty in distinguishing between permanent pastures and other similar land covers (such as temporary pastures and shrublands). In the second step, it was possible to distinguish the permanent pasture parcels from the others. The obtained results improved significantly from the first to the second step. Using CNN, an accuracy of 93%, a precision of 89%, and a recall of 98% were achieved for the “Permanent pasture” class. The F1-score was 94%, indicating a balanced measure of the model’s performance.DiscussionThe integration of advanced technologies in the CAP’s control mechanisms, as demonstrated, has the potential to automate the verification of farmers’ declarations and subsequent subsidy payments.

Multi-Level Machine Learning Model to Improve the Effectiveness of Predicting Customers Churn Banks

Abstract This study presents a novel multi-level Stacking model designed to enhance the accuracy of customer churn prediction in the banking sector, a critical aspect for improving customer retention. Our approach integrates four distinct machine-learning algorithms – K-Nearest Neighbor (KNN), XGBoost, Random Forest (RF), and Support Vector Machine (SVM) – at the first level (Level 0). These algorithms generate initial predictions, which are then combined and fed into higher-level models (Level 1) comprising Logistic Regression, Recurrent Neural Network (RNN), and Deep Neural Network (DNN). We evaluated the model through three scenarios: Scenario 1 uses Logistic Regression at Level 1, Scenario 2 employs a Deep Convolutional Neural Network (DNN), and Scenario 3 utilizes a Deep Recurrent Neural Network (RNN). Our experiments on multiple datasets demonstrate significant improvements over traditional methods. In particular, Scenario 1 achieved an accuracy of 91.08%, a ROC-AUC of 98%, and an AUC-PR of 98.15%. Comparisons with existing research further underscore the enhanced performance of our proposed model.

Reducing training data needs with minimal multilevel machine learning (M3L)

For many machine learning applications in science, data acquisition, not training, is the bottleneck even when avoiding experiments and relying on computation and simulation. Correspondingly, and in order to reduce cost and carbon footprint, training data efficiency is key. We introduce minimal multilevel machine learning (M3L) which optimizes training data set sizes using a loss function at multiple levels of reference data in order to minimize a combination of prediction error with overall training data acquisition costs (as measured by computational wall-times). Numerical evidence has been obtained for calculated atomization energies and electron affinities of thousands of organic molecules at various levels of theory including HF, MP2, DLPNO-CCSD(T), DFHFCABS, PNOMP2F12, and PNOCCSD(T)F12, and treating them with basis sets TZ, cc-pVTZ, and AVTZ-F12. Our M3L benchmarks for reaching chemical accuracy in distinct chemical compound sub-spaces indicate substantial computational cost reductions by factors of ∼1.01, 1.1, 3.8, 13.8, and 25.8 when compared to heuristic sub-optimal multilevel machine learning (M2L) for the data sets QM7b, QM9 LCCSD(T) , Electrolyte Genome Project, QM9 AECCSD(T) , and QM9 EACCSD(T) , respectively. Furthermore, we use M2L to investigate the performance for 76 density functionals when used within multilevel learning and building on the following levels drawn from the hierarchy of Jacobs Ladder: LDA, GGA, mGGA, and hybrid functionals. Within M2L and the molecules considered, mGGAs do not provide any noticeable advantage over GGAs. Among the functionals considered and in combination with LDA, the three on average top performing GGA and Hybrid levels for atomization energies on QM9 using M3L correspond respectively to PW91, KT2, B97D, and τ-HCTH, B3LYP ∗ (VWN5), and TPSSH.

Improving cyberbullying detection through multi-level machine learning

Cyberbullying is a known risk factor for mental health issues, demanding immediate attention. This study aims to detect cyberbullying on social media in alignment with the third sustainable development goal (SDG) for health and well-being. Many previous studies employ single-level classification, but this research introduces a multi-class multi-level (MCML) algorithm for a more detailed approach. The MCML approach incorporates two levels of classification: level one for cyberbullying or not cyberbullying, and level two for classifying cyberbullying by type. This study used a dataset of 47,000 tweets from Twitter with six class labels and employed an 80:20 training and testing data split. By integrating bidirectional encoder representations from transformers (BERT) and MCML at level two, we achieved a remarkable 99% accuracy, surpassing BERT-based single-level classification at 94%. In conclusion, the combination of MCML and BERT offers enhanced cyberbullying classification accuracy, contributing to the broader goal of promoting mental health and well-being.

Ecological Momentary Assessments and Passive Sensing in the Prediction of Short-Term Suicidal Ideation in Young Adults

Advancements in technology, including mobile-based ecological momentary assessments (EMAs) and passive sensing, have immense potential to identify short-term suicide risk. However, the extent to which EMA and passive data, particularly in combination, have utility in detecting short-term risk in everyday life remains poorly understood. To examine whether and what combinations of self-reported EMA and sensor-based assessments identify next-day suicidal ideation. In this intensive longitudinal prognostic study, participants completed EMAs 4 times daily and wore a sensor wristband (Fitbit Charge 3) for 8 weeks. Multilevel machine learning methods, including penalized generalized estimating equations and classification and regression trees (CARTs) with repeated 5-fold cross-validation, were used to optimize prediction of next-day suicidal ideation based on time-varying features from EMAs (affective, cognitive, behavioral risk factors) and sensor data (sleep, activity, heart rate). Young adult patients who visited an emergency department with recent suicidal ideation and/or suicide attempt were recruited. Identified via electronic health record screening, eligible individuals were contacted remotely to complete enrollment procedures. Participants (aged 18 to 25 years) completed 14 708 EMA observations (64.4% adherence) and wore a sensor wristband approximately half the time (55.6% adherence). Data were collected between June 2020 and July 2021. Statistical analysis was performed from January to March 2023. The outcome was presence of next-day suicidal ideation. Among 102 enrolled participants, 83 (81.4%) were female; 6 (5.9%) were Asian, 5 (4.9%) were Black or African American, 9 (8.8%) were more than 1 race, and 76 (74.5%) were White; mean (SD) age was 20.9 (2.1) years. The best-performing model incorporated features from EMAs and showed good predictive accuracy (mean [SE] cross-validated area under the receiver operating characteristic curve [AUC], 0.84 [0.02]), whereas the model that incorporated features from sensor data alone showed poor prediction (mean [SE] cross-validated AUC, 0.56 [0.02]). Sensor-based features did not improve prediction when combined with EMAs. Suicidal ideation-related features were the strongest predictors of next-day ideation. When suicidal ideation features were excluded, an alternative EMA model had acceptable predictive accuracy (mean [SE] cross-validated AUC, 0.76 [0.02]). Both EMA models included features at different timescales reflecting within-day, end-of-day, and time-varying cumulative effects. In this prognostic study, self-reported risk factors showed utility in identifying near-term suicidal thoughts. Best-performing models required self-reported information, derived from EMAs, whereas sensor-based data had negligible predictive accuracy. These results may have implications for developing decision algorithms identifying near-term suicidal thoughts to guide risk monitoring and intervention delivery in everyday life.

Damage Detection in Largely Unobserved Structures under Varying Environmental Conditions: An AutoRegressive Spectrum and Multi-Level Machine Learning Methodology.

Vibration-based damage detection in civil structures using data-driven methods requires sufficient vibration responses acquired with a sensor network. Due to technical and economic reasons, it is not always possible to deploy a large number of sensors. This limitation may lead to partial information being handled for damage detection purposes, under environmental variability. To address this challenge, this article proposes an innovative multi-level machine learning method by employing the autoregressive spectrum as the main damage-sensitive feature. The proposed method consists of three levels: (i) distance calculation by the log-spectral distance, to increase damage detectability and generate distance-based training and test samples; (ii) feature normalization by an improved factor analysis, to remove environmental variations; and (iii) decision-making for damage localization by means of the Jensen–Shannon divergence. The major contributions of this research are represented by the development of the aforementioned multi-level machine learning method, and by the proposal of the new factor analysis for feature normalization. Limited vibration datasets relevant to a truss structure and consisting of acceleration time histories induced by shaker excitation in a passive system, have been used to validate the proposed method and to compare it with alternate, state-of-the-art strategies.

Early-predicting dropout of university students: an application of innovative multilevel machine learning and statistical techniques

ABSTRACT This paper combines a theoretical-based model with a data-driven approach to develop an Early Warning System that detects students who are more likely to dropout. The model uses innovative multilevel statistical and machine learning methods. The paper demonstrates the validity of the approach by applying it to administrative data from a leading Italian university.

Analysis of the Impact of Sustained Load and Temperature on the Performance of the Electromechanical Impedance Technique through Multilevel Machine Learning and FBG Sensors.

The electro-mechanical impedance (EMI) technique has been applied successfully to detect minor damage in engineering structures including reinforced concrete (RC). However, in the presence of temperature variations, it can cause false alarms in structural health monitoring (SHM) applications. This paper has developed an innovative approach that integrates the EMI methodology with multilevel hierarchical machine learning techniques and the use of fiber Bragg grating (FBG) temperature and strain sensors to evaluate the mechanical performance of RC beams strengthened with near surface mounted (NSM)-fiber reinforced polymer (FRP) under sustained load and varied temperatures. This problem is a real challenge since the bond behavior at the concrete–FRP interface plays a key role in the performance of this type of structure, and additionally, its failure occurs in a brittle and sudden way. The method was validated in a specimen tested over a period of 1.5 years under different conditions of sustained load and temperature. The analysis of the experimental results in an especially complex problem with the proposed approach demonstrated its effectiveness as an SHM method in a combined EMI–FBG framework.

Integrating across neuroimaging modalities boosts prediction accuracy of cognitive ability.

Variation in cognitive ability arises from subtle differences in underlying neural architecture. Understanding and predicting individual variability in cognition from the differences in brain networks requires harnessing the unique variance captured by different neuroimaging modalities. Here we adopted a multi-level machine learning approach that combines diffusion, functional, and structural MRI data from the Human Connectome Project (N = 1050) to provide unitary prediction models of various cognitive abilities: global cognitive function, fluid intelligence, crystallized intelligence, impulsivity, spatial orientation, verbal episodic memory and sustained attention. Out-of-sample predictions of each cognitive score were first generated using a sparsity-constrained principal component regression on individual neuroimaging modalities. These individual predictions were then aggregated and submitted to a LASSO estimator that removed redundant variability across channels. This stacked prediction led to a significant improvement in accuracy, relative to the best single modality predictions (approximately 1% to more than 3% boost in variance explained), across a majority of the cognitive abilities tested. Further analysis found that diffusion and brain surface properties contribute the most to the predictive power. Our findings establish a lower bound to predict individual differences in cognition using multiple neuroimaging measures of brain architecture, both structural and functional, quantify the relative predictive power of the different imaging modalities, and reveal how each modality provides unique and complementary information about individual differences in cognitive function.

MULTI-SOURCE MODEL OF HETEROGENEOUS DATA ANALYSIS FOR OIL PRICE FORECASTING

This article sheds light on the question of whether it is possible to create fairly accurate forecasts of real oil prices. For this purpose, a multi-level machine learning model has been created to analyze several sources of heterogeneous data to predict future prices. The article uses different types of data: market condition data, titles, and transaction data. Then, they have been processed to be able to load them into the model. The validation of the regression neural network results showed that the model is more accurate than in previous studies. In fact, this paper presents an artificial neural network model that solves the problem of determining the most informative relationship between different types of oil price data. Keywords: artificial neural network, oil forecasting, machine learning, price prediction, energy resources.JEL Classifications: C45, C51, Q43, Q47DOI: https://doi.org/10.32479/ijeep.10853

Individual yarn fibre extraction from micro-CT: multilevel machine learning approach

The internal structure and mechanics of the fibre materials, such as yarn or woven textile, are highly complex. Exploring the fibre structure is an essential step in material engineering either from the experimental or computational point of view. In this study, a new method to extract geometrical and morphological parameters of fibre structures is proposed. The method benefits from standard image analysis and machine learning technique to efficiently extract fibre segments from microcomputer tomography data. The proposed algorithm is tested on the yarn and woven textile materials with different resolution and quality. The developed method can extract the individual fibres with varying accuracy from 73% to 100% with processing time 2–5 s on the tested samples.

Three-Tier Control of Energy Costs in Transport Corporation with Self-Learning

According to the concept of Industry 4.0, the control of energy costs in a large transport corporation should be based on the use of artificial intelligence, the main focus of which is machine learning. It is also necessary to take into account the human factor - the unwanted activity of corporation employees who use information available only to them to achieve their own goals. The article examines the problem of reducing energy costs in a 3-tier transport corporation, which has regional offices, which are subordinate to local enterprises. The plant management knows own abilities of energy savings better than the regional management. Respectively, regional management knows these abilities better than the corporate governance. Therefore, both regional and plant management can manipulate energy expenditures to get more stimuli from the corporate governance. Such unwanted activity can result in excess energy consumption by the corporation. The goal is to minimize the energy costs of the corporation. To achieve this goal, a mechanism for organizational control of energy costs in a 3-tier transport corporation is proposed. This mechanism includes multilevel machine learning and stimulating procedures. Sufficient conditions are found for the synthesis of such a mechanism in which the existing possibilities of reducing energy costs in a transport corporation are used. The advantage of this mechanism over existing methods of learning classification in organizational systems (such as artificial neural networks or self-organizing maps) is to stimulate energy savings. The use of such a mechanism is illustrated by the example of 3-tier control system for energy consumption of JSC Russian Railways.

A Case Study Based Approach for Remote Fault Detection Using Multi-Level Machine Learning in A Smart Building

Due to the increased awareness of issues ranging from green initiatives, sustainability, and occupant well-being, buildings are becoming smarter, but with smart requirements come increasing complexity and monitoring, ultimately carried out by humans. Building heating ventilation and air-conditioning (HVAC) units are one of the major units that consume large percentages of a building’s energy, for example through their involvement in space heating and cooling, the greatest energy consumption in buildings. By monitoring such components effectively, the entire energy demand in buildings can be substantially decreased. Due to the complex nature of building management systems (BMS), many simultaneous anomalous behaviour warnings are not manageable in a timely manner; thus, many energy related problems are left unmanaged, which causes unnecessary energy wastage and deteriorates equipment’s lifespan. This study proposes a machine learning based multi-level automatic fault detection system (MLe-AFD) focusing on remote HVAC fan coil unit (FCU) behaviour analysis. The proposed method employs sequential two-stage clustering to identify the abnormal behaviour of FCU. The model’s performance is validated by implementing well-known statistical measures and further cross-validated via expert building engineering knowledge. The method was experimented on a commercial building based in central London, U.K., as a case study and allows remotely identifying three types of FCU faults appropriately and informing building management staff proactively when they occur; this way, the energy expenditure can be further optimized.

Прогнозирование халькошпинелей состава ABCX4 (X — S или Se)

New chalcospinels of the most common compositions were predicted: AIBIIICIVX4 (X — S or Se) and AIIBIIICIIIS4 (A, B, and C are various chemical elements). They are promising for the search for new materials for magneto-optical memory elements, sensors and anodes in sodium-ion batteries. The parameter “a” values of their crystal lattice are estimated. When predicting only the values of chemical elements properties were used. The calculations were carried out using machine learning programs that are part of the information-analytical system developed by the authors (various ensembles of algorithms of: the binary decision trees, the linear machine, the search for logical regularities of classes, the support vector machine, Fisher linear discriminant, the k-nearest neighbors, the learning a multilayer perceptron and a neural network), — for predicting chalcospinels not yet obtained, as well as an extensive family of regression methods, presented in the scikit-learn package for the Python language, and multilevel machine learning methods that were proposed by the authors — for estimation of the new chalcospinels lattice parameter value). The prediction accuracy of new chalcospinels according to the results of the cross-validation is not lower than 80%, and the prediction accuracy of the parameter of their crystal lattice (according to the results of calculating the mean absolute error (when cross-validation in the leave-one-out mode)) is ± 0.1 Å. The effectiveness of using multilevel machine learning methods to predict the physical properties of substances was shown.

A Tunable-Q wavelet transform and quadruple symmetric pattern based EEG signal classification method

Electroencephalography (EEG) signals have been widely used to diagnose brain diseases for instance epilepsy, Parkinson’s Disease (PD), Multiple Skleroz (MS), and many machine learning methods have been proposed to develop automated disease diagnosis methods using EEG signals. In this method, a multilevel machine learning method is presented to diagnose epilepsy disease. The proposed multilevel EEG classification method consists of pre-processing, feature extraction, feature concatenation, feature selection and classification phases. In order to create levels, Tunable-Q wavelet transform (TQWT) is chosen and 25 frequency coefficients sub-bands are calculated by using TQWT in the pre-processing. In the feature extraction phase, quadruple symmetric pattern (QSP) is chosen as feature extractor and extracts 256 features from the raw EEG signal and the extracted 25 sub-bands. In the feature selection phase, neighborhood component analysis (NCA) is used. The 128, 256, 512 and 1024 most significant features are selected in this phase. In the classification phase, k nearest neighbors (kNN) classifier is utilized as classifier. The proposed method is tested on seven cases using Bonn EEG dataset. The proposed method achieved 98.4% success rate for 5 classes case. Therefore, our proposed method can be used in bigger datasets for more validation.

Multi-Level Model to Predict Antibody Response to Influenza Vaccine Using Gene Expression Interaction Network Feature Selection.

Vaccination is an effective prevention of influenza infection. However, certain individuals develop a lower antibody response after vaccination, which may lead to susceptibility to subsequent infection. An important challenge in human health is to find baseline gene signatures to help identify individuals who are at higher risk for infection despite influenza vaccination. We developed a multi-level machine learning strategy to build a predictive model of vaccine response using pre−vaccination antibody titers and network interactions between pre−vaccination gene expression levels. The first-level baseline−antibody model explains a significant amount of variation in post-vaccination response, especially for subjects with large pre−existing antibody titers. In the second level, we clustered individuals based on pre−vaccination antibody titers to focus gene−based modeling on individuals with lower baseline HAI where additional response variation may be predicted by baseline gene expression levels. In the third level, we used a gene−association interaction network (GAIN) feature selection algorithm to find the best pairs of genes that interact to influence antibody response within each baseline titer cluster. We used ratios of the top interacting genes as predictors to stabilize machine learning model generalizability. We trained and tested the multi-level approach on data with young and older individuals immunized against influenza vaccine in multiple cohorts. Our results indicate that the GAIN feature selection approach improves model generalizability and identifies genes enriched for immunologically relevant pathways, including B Cell Receptor signaling and antigen processing. Using a multi-level approach, starting with a baseline HAI model and stratifying on baseline HAI, allows for more targeted gene−based modeling. We provide an interactive tool that may be extended to other vaccine studies.

Multi-level machine learning prediction of protein-protein interactions in Saccharomyces cerevisiae.

Accurate identification of protein–protein interactions (PPI) is the key step in understanding proteins’ biological functions, which are typically context-dependent. Many existing PPI predictors rely on aggregated features from protein sequences, however only a few methods exploit local information about specific residue contacts. In this work we present a two-stage machine learning approach for prediction of protein–protein interactions. We start with the carefully filtered data on protein complexes available for Saccharomyces cerevisiae in the Protein Data Bank (PDB) database. First, we build linear descriptions of interacting and non-interacting sequence segment pairs based on their inter-residue distances. Secondly, we train machine learning classifiers to predict binary segment interactions for any two short sequence fragments. The final prediction of the protein–protein interaction is done using the 2D matrix representation of all-against-all possible interacting sequence segments of both analysed proteins. The level-I predictor achieves 0.88 AUC for micro-scale, i.e., residue-level prediction. The level-II predictor improves the results further by a more complex learning paradigm. We perform 30-fold macro-scale, i.e., protein-level cross-validation experiment. The level-II predictor using PSIPRED-predicted secondary structure reaches 0.70 precision, 0.68 recall, and 0.70 AUC, whereas other popular methods provide results below 0.6 threshold (recall, precision, AUC). Our results demonstrate that multi-scale sequence features aggregation procedure is able to improve the machine learning results by more than 10% as compared to other sequence representations. Prepared datasets and source code for our experimental pipeline are freely available for download from: http://zubekj.github.io/mlppi/ (open source Python implementation, OS independent).

Edge-over-Erosion Error Prediction Method Based on Multi-Level Machine Learning Algorithm

SUMMARY As VLSI process node continue to shrink, chemical mechanical planarization (CMP) process for copper interconnect has become an essential technique for enabling many-layer interconnection. Recently, Edge-over-Erosion error (EoE-error), which originates from overpolishing and could cause yield loss, is observed in various CMP processes, while its mechanism is still unclear. To predict these errors, we propose an EoEerror prediction method that exploits machine learning algorithms. The proposed method consists of (1) error analysis stage, (2) layout parameter extraction stage, (3) model construction stage and (4) prediction stage. In the error analysis and parameter extraction stages, we analyze test chips and identify layout parameters which have an impact on EoE phenomenon. In the model construction stage, we construct a prediction model using the proposed multi-level machine learning method, and do predictions for designed layouts in the prediction stage. Experimental results show that the proposed method attained 2.7∼19.2% accuracy improvement of EoE-error prediction and 0.8∼10.1% improvement of non-EoE-error prediction compared with general machine learning methods. The proposed method makes it possible to prevent unexpected yield loss by recognizing EoE-errors be