Competitive Prediction Accuracy Research Articles

Interpretable machine learning model for activation energy prediction based on biomass properties

A thorough understanding of pyrolysis kinetics for biomass is crucial for the process design, feasibility assessment, and scale-up in industrial scenarios. To reduce the time and economic cost invested in obtaining experimental kinetic data by performing the thermogravimetric analysis, a data-driven machine learning method was proposed in the present work. Random forest (RF), gradient boosting decision tree (GBDT), and extreme gradient boosting (XGB) algorithms were utilized to predict the activation energy versus conversion range with ten input features from biomass characteristics. Among these XGB has shown the most competitive prediction accuracy with a coefficient of determination (R2) higher than 0.98 and a root mean squared error (RMSE) = 7.7 kJ/mol. To enhance the model interpretability of the derived model, a visualized analysis for individual impact and interactions of variables was conducted by Shapley additive explanation (SHAP). The top three significant inputs for the model established by the current dataset are conversion rate, volatile content, and moisture content. This work provides a quick and accurate kinetic prediction method for unexploited biomass and favors detailed insights into the complicated associations between biomass characteristics and pyrolysis kinetics.

GRAformer: A gated residual attention transformer for multivariate time series forecasting

Recurrent Neural Networks (RNNs), particularly when equipped with output windows – a standard practice in contemporary time series forecasting – have shown proficiency in handling short-term dependencies. Nonetheless, RNNs can encounter challenges in maintaining hidden states over extended forecasting periods, particularly in longer-term predictions where increased hidden state sizes and extended look-back windows can lead to gradient instability. In contrast, Transformer-based models, with their distinctive architecture designed to encode complex contextual relationships and enable computations to be done in parallel, are emerging as a popular alternative in this field. However, current research has mainly focused on modifying attention mechanisms, overlooking opportunities to improve the feedforward layer, which could lead to efficiency limitations. Moreover, prevailing methods often assume absolute independence between channels, disregarding distinct features among variables and failing to fully leverage channel-specific information. To address these gaps, we propose an efficient transformer design for multivariate time series prediction. Our approach integrates two key components: (i) a gated residual attention unit that enhances predictive accuracy and computational efficiency, and (ii) a channel embedding technique that differentiates between series and boosts performance. Theoretically, we prove that our model has recurrent dynamics introduced by the RNN layer. Through extensive experiments on real-world data, we demonstrate that our proposed method achieves competitive predictive accuracy compared to prior approaches while exhibiting accelerated processing relative to state-of-the-art transformers. Our code, data, and trained models are available at https://github.com/MythosAd/GRAformer.

Linked shrinkage to improve estimation of interaction effects in regression models.

The addition of two-way interactions is a classic problem in statistics, and comes with the challenge of quadratically increasing dimension. We aim to a) devise an estimation method that can handle this challenge and b) to aid interpretation of the resulting model by developing computational tools for quantifying variable importance. Existing strategies typically overcome the dimensionality problem by only allowing interactions between relevant main effects. Building on this philosophy, and aiming for settings with moderate n to p ratio, we develop a local shrinkage model that links the shrinkage of interaction effects to the shrinkage of their corresponding main effects. In addition, we derive a new analytical formula for the Shapley value, which allows rapid assessment of individual-specific variable importance scores and their uncertainties. We empirically demonstrate that our approach provides accurate estimates of the model parameters and very competitive predictive accuracy. In our Bayesian framework, estimation inherently comes with inference, which facilitates variable selection. Comparisons with key competitors are provided. Large-scale cohort data are used to provide realistic illustrations and evaluations. The implementation of our method in RStan is relatively straightforward and flexible, allowing for adaptation to specific needs. Our method is an attractive alternative for existing strategies to handle interactions in epidemiological and/or clinical studies, as its linked local shrinkage can improve parameter accuracy, prediction and variable selection. Moreover, it provides appropriate inference and interpretation, and may compete well with less interpretable machine learners in terms of prediction.

Collaborative analysis for drug discovery by federated learning on non-IID data.

With the increasing availability of large-scale QSAR (Quantitative Structure-Activity Relationship) datasets, collaborative analysis has become a promising approach for drug discovery. Traditional centralized analysis which typically concentrates data on a central server for training faces challenges such as data privacy and security. Distributed analysis such as federated learning offers a solution by enabling collaborative model training without sharing raw data. However, it may fail when the training data in the local devices are non-independent and identically distributed (non-IID). In this paper, we propose a novel framework for collaborative drug discovery using federated learning on non-IID datasets. We address the difficulty of training on non-IID data by globally sharing a small subset of data among all institutions. Our framework allows multiple institutions to jointly train a robust predictive model while preserving the privacy of their individual data. We leverage the federated learning paradigm to distribute the model training process across local devices, eliminating the need for data exchange. The experimental results on 15 benchmark datasets demonstrate that the proposed method achieves competitive predictive accuracy to centralized analysis while respecting data privacy. Moreover, our framework offers benefits such as reduced data transmission and enhanced scalability, making it suitable for large-scale collaborative drug discovery efforts.

Parkinsonian Tremor Detection with Compact Convolutional Transformer from Bispectrum Representation of tri-Axial Accelerometer Signals.

After the breakthroughs of Transformer networks in Natural Language Processing (NLP) tasks, they have led to exciting progress in visual tasks as well. Nonetheless, there has been a parallel growth in the number of parameters and the amount of training data, which led to the conclusion that Transformers are not suited for small datasets. This paper is the first to convey the feasibility of Compact Convolutional Transformers (CCT) for the prediction of Parkinsonian postural tremor based on the Bispectrum (BS) representation of IMU accelerometer time series. The dataset includes tri-axial accelerometer signals collected unobtrusively in-the-wild while subjects are on a phone call, and labelled by neurologists and signal processing experts. The BS is a noise-immune, higher-order representation that reflects a signal's deviation from Gaussianity and measures quadratic phase coupling. We performed comparative classification experiments using the CCT, pre-trained CNNs such as VGG-16 and ResNet-50, and the conventional Vision Transformer (ViT). Our model achieves competitive prediction accuracy and F1 score of 96% with only 1.016 M trainable parameters, compared to the ViT with 21.659 M trainable parameters, in a five-fold cross-validation scheme. Our model also outperforms pre-trained CNNs such as VGG-16 and ResNet-50. Furthermore, we show that the performance gains are maintained when training on a larger dataset of BS images. Our effort here is motivated by the hypothesis that data-efficient transformers outperform transfer learning using pre-trained CNNs, paving the way for promising deep learning architecture for small-scale, novel and noisy medical imaging datasets.Clinical relevance- Novel deep learning model for unobtrusive prediction of Parkinsonian Postural Tremor from Bispectrum image representation of tri-axial accelerometer signals collected in-the-wild.

Knowledge-embedded meta-learning model for lift coefficient prediction of airfoils

The evaluation of aerodynamic performance is a crucial aspect of aircraft design optimization. However, traditional methods are time-consuming and expensive. While machine learning techniques can achieve high accuracy, their application in engineering is challenging due to their poor generalization and ”black box” nature. In this paper, a knowledge-embedded meta-learning model is proposed that integrates data with theoretical knowledge to predict lift coefficients for a supercritical airfoil at various angles of attack. The model employs a primary network to capture the relationship between lift and angle of attack, and a hyper network to encode geometry information and predict unknown parameters in the primary network. Three models are trained with different architectures to provide various interpretations. The model exhibits better generalization capabilities and competitive prediction accuracy compared to conventional neural networks. Additionally, interpretable analysis is performed using Integrated Gradients and Saliency methods, revealing that this model can assess the influence of airfoil geometry on physical characteristics. Furthermore, exceptional cases are investigated where the model’s performance is reduced, analyzing the impact of knowledge limitations on a knowledge-embedded predictive model.

A two-stage online remaining useful life prediction framework for supercapacitors based on the fusion of deep learning network and state estimation algorithm

Supercapacitors have been widely used in many fields. The safe and stable operation of supercapacitors requires accurate remaining useful life (RUL) prediction. This paper proposes a two-stage online RUL prediction framework based on the bidirectional long short-term memory (BiLSTM) network and the H∞ observer. In Stage 1, the BiLSTM network as well as the Bayesian Optimization algorithm is used to estimate capacitance. The Bayesian Optimization (BO) algorithm aims to optimize the hyperparameters of the BiLSTM network. In Stage 2, the estimated capacitance is handled by the moving average filter (MAF) to alleviate short-term fluctuations. The double exponential model is employed to describe the degradation trajectory, and the model is iteratively updated by the H∞ observer with the estimated capacitance as the measurement. Experiments are implemented to verify the validation of the proposed framework. The results indicate that the BiLSTM network can explain more than 99% variation of capacitance. The deviation of RUL prediction is less than 6% in most cases, and it is even less than 2.5% in late phase. The proposed framework takes the advantages of online deployment, and achieves competitive prediction accuracy when compared with offline methods.

On a nonlinear extension of the principal fitted component model

We propose a nonlinear sufficient dimension reduction method called the kernel principal fitted component model using the kernel method under a reproducing kernel Hilbert space. The kernel principal fitted component is a nonlinear extension of the principal fitted component model, and it is found in the theory of mapping low dimensional input space to the higher dimensional feature space so that we can apply well-developed linear methods to the nonlinear dataset. We derive our method coincides with the generalized sliced inverse regression under some mild assumptions and show the dimension reduction subspace extracted from the kernel principal fitted component model is contained in the central class. In the numerical experiments, we present the kernel principal fitted component model with the Gaussian kernel can extract the linear and nonlinear features well for the models from both forward and inverse regression settings. By applying our method to ovarian cancer microarray dataset, we demonstrate the kernel principal fitted component can provide a competitive prediction accuracy and computational efficiency in the high-dimensional classification problem.

A statistical boosting framework for polygenic risk scores based on large-scale genotype data.

Polygenic risk scores (PRS) evaluate the individual genetic liability to a certain trait and are expected to play an increasingly important role in clinical risk stratification. Most often, PRS are estimated based on summary statistics of univariate effects derived from genome-wide association studies. To improve the predictive performance of PRS, it is desirable to fit multivariable models directly on the genetic data. Due to the large and high-dimensional data, a direct application of existing methods is often not feasible and new efficient algorithms are required to overcome the computational burden regarding efficiency and memory demands. We develop an adapted component-wise L 2-boosting algorithm to fit genotype data from large cohort studies to continuous outcomes using linear base-learners for the genetic variants. Similar to the snpnet approach implementing lasso regression, the proposed snpboost approach iteratively works on smaller batches of variants. By restricting the set of possible base-learners in each boosting step to variants most correlated with the residuals from previous iterations, the computational efficiency can be substantially increased without losing prediction accuracy. Furthermore, for large-scale data based on various traits from the UK Biobank we show that our method yields competitive prediction accuracy and computational efficiency compared to the snpnet approach and further commonly used methods. Due to the modular structure of boosting, our framework can be further extended to construct PRS for different outcome data and effect types-we illustrate this for the prediction of binary traits.

Health Assessment of Rotating Equipment With Unseen Conditions Using Adversarial Domain Generalization Toward Self-Supervised Regularization Learning

Traditional health assessment models work well under the assumption that the test and training samples obey a similar distribution. However, it is practically impossible to eliminate domain shifts between different tasks. Thus, most work tries to establish a data-driven approach via domain adaptation to accomplish transfer learning between different operating conditions. Sufficient target data are needed to participate in the training, which may not normally be available due to most working scenarios being unseen. An adversarial domain generalization framework with regularization learning (ADGR) is proposed for the health assessment to mine latent domains. Also, the latent domain is expanded to the unseen domain as possible. More specifically, the diversity of the sample distribution is augmented by adversarial training and the maximization of the domain discrepancy between the latent and source domains. Meanwhile, self-supervised interdomain regularization and semantical consistent regularization are proposed to mitigate the feature drift of the domain classifier and semantic divergence between source and latent domains. The case study shows that the ADGR-based health assessment approach achieves competitive prediction accuracy under unseen conditions, demonstrating its potential as a diagnostic solution.

BoostMEC: predicting CRISPR-Cas9 cleavage efficiency through boosting models

BackgroundIn the CRISPR-Cas9 system, the efficiency of genetic modifications has been found to vary depending on the single guide RNA (sgRNA) used. A variety of sgRNA properties have been found to be predictive of CRISPR cleavage efficiency, including the position-specific sequence composition of sgRNAs, global sgRNA sequence properties, and thermodynamic features. While prevalent existing deep learning-based approaches provide competitive prediction accuracy, a more interpretable model is desirable to help understand how different features may contribute to CRISPR-Cas9 cleavage efficiency.ResultsWe propose a gradient boosting approach, utilizing LightGBM to develop an integrated tool, BoostMEC (Boosting Model for Efficient CRISPR), for the prediction of wild-type CRISPR-Cas9 editing efficiency. We benchmark BoostMEC against 10 popular models on 13 external datasets and show its competitive performance.ConclusionsBoostMEC can provide state-of-the-art predictions of CRISPR-Cas9 cleavage efficiency for sgRNA design and selection. Relying on direct and derived sequence features of sgRNA sequences and based on conventional machine learning, BoostMEC maintains an advantage over other state-of-the-art CRISPR efficiency prediction models that are based on deep learning through its ability to produce more interpretable feature insights and predictions.

Restricted Relevance Vector Machine for Missing Data and Application to Virtual Metrology

In semiconductor manufacturing, virtual metrology (VM) is a method of predicting physical measurements of wafer qualities using in-process information from sensors on production equipment. The relevance vector machine (RVM) is a sparse Bayesian kernel machine that has been widely used for VM modeling in semiconductor manufacturing. Missing values from equipment sensors, however, preclude training an RVM model due to missing kernels from incomplete instances. Moreover, imputation for such kernels can lead to a loss of model sparsity. In this work, we propose a restricted RVM (RRVM) that selects its basis functions from only complete instances to handle incomplete data for VM. We conduct the experiments using toy data and real-life data from an etching process for wafer fabrication. The results indicate the model’s competitive prediction accuracy with massive missing data while maintaining model sparsity. Note to Practitioners—In recent decades, virtual metrology (VM) has focused on wafer fabrication in semiconductor manufacturing due to its advantages for process monitoring and automation. Typically, signals from production process equipment can predict wafer qualities in VM, which often leads to high data dimensionality. The relevance vector machine (RVM) is an algorithm that can provide a sparse solution to a Bayesian kernel method for a prediction model. Missing components in incomplete data due to sensor failures in wafer fabrication processes, however, hinder model training, and the existing approaches to handling missing data using imputation may lead to a loss of model sparsity. This article proposes a new method for RVM with incomplete data to train a model built on fully available instances by incorporating the available components of incomplete instances into model training. Using the proposed method, one can predict wafer qualities building a model trained to maintain its sparsity. Experiments indicate that the proposed model achieves competitive prediction performance and maintains model sparsity when incomplete instances are used.

Efficient Learning of Interpretable Classification Rules

Machine learning has become omnipresent with applications in various safety-critical domains such as medical, law, and transportation. In these domains, high-stake decisions provided by machine learning necessitate researchers to design interpretable models, where the prediction is understandable to a human. In interpretable machine learning, rule-based classifiers are particularly effective in representing the decision boundary through a set of rules comprising input features. Examples of such classifiers include decision trees, decision lists, and decision sets. The interpretability of rule-based classifiers is in general related to the size of the rules, where smaller rules are considered more interpretable. To learn such a classifier, the brute-force direct approach is to consider an optimization problem that tries to learn the smallest classification rule that has close to maximum accuracy. This optimization problem is computationally intractable due to its combinatorial nature and thus, the problem is not scalable in large datasets. To this end, in this paper we study the triangular relationship among the accuracy, interpretability, and scalability of learning rule-based classifiers. The contribution of this paper is an interpretable learning framework IMLI, that is based on maximum satisfiability (MaxSAT) for synthesizing classification rules expressible in proposition logic. IMLI considers a joint objective function to optimize the accuracy and the interpretability of classification rules and learns an optimal rule by solving an appropriately designed MaxSAT query. Despite the progress of MaxSAT solving in the last decade, the straightforward MaxSAT-based solution cannot scale to practical classification datasets containing thousands to millions of samples. Therefore, we incorporate an efficient incremental learning technique inside the MaxSAT formulation by integrating mini-batch learning and iterative rule-learning. The resulting framework learns a classifier by iteratively covering the training data, wherein in each iteration, it solves a sequence of smaller MaxSAT queries corresponding to each mini-batch. In our experiments, IMLI achieves the best balance among prediction accuracy, interpretability, and scalability. For instance, IMLI attains a competitive prediction accuracy and interpretability w.r.t. existing interpretable classifiers and demonstrates impressive scalability on large datasets where both interpretable and non-interpretable classifiers fail. As an application, we deploy IMLI in learning popular interpretable classifiers such as decision lists and decision sets. The source code is available at https://github.com/meelgroup/mlic.

LIFT AND DRAG FORCE PREDICTION USING CONVOLUTIONAL NEURAL NETWORKS


 
 
 The adaptability of the convolutional neural network (CNN) technique is probed for aerodynamic meta- modeling task. The primary objective is to develop a suitable architecture for variable flow conditions and object geometry, in addition to identifying a sufficient data preparation process. Multiple CNN structures were trained to learn the lift coefficients of the airfoils with a variety of shapes in multiple flow Mach numbers, Reynolds numbers, and diverse angles of attack. This was conducted to illustrate the concept of the methodology. Multi-layered perceptron (MLP) solutions were also obtained and compared with the CNN results. The newly proposed meta-modeling concept has been found to be comparable with the MLP in learning capability; and more importantly, our CNN model exhibits a competitive prediction accuracy with minimal constraints in geometric representation.
 
 

An active learning Bayesian ensemble surrogate model for structural reliability analysis

AbstractSurrogate models have been proven to be powerful tools to alleviate the computational burden of structural reliability analysis. An appropriate surrogate model can guarantee prediction accuracy with limited samples. However, the traditional single modeling technique ignores the model‐form uncertainty due to insufficient knowledge of the physical system, leading to unreliable prediction results or time‐consuming computation. To overcome the aforementioned deficiencies, an active learning ensemble surrogate model under the framework of Bayesian inference is proposed for structural reliability analysis. Based on the derived Bayesian posterior distribution of the predicted response, a learning function integrating the modified U function and the distance information between design points is developed to sequentially select the next point. Besides, in order to further enhance the computational efficiency, we propose an adaptive method to identify the sampling region according to the prediction uncertainty of the estimated limit state surface. Five benchmark examples are employed to verify the effectiveness and efficiency of the proposed algorithm. Comparison results show that the proposed active learning reliability analysis method based on the Bayesian ensemble surrogate model can greatly reduce the computational expense with a competitive prediction accuracy. Taking the 10‐bar truss problem as an example, compared with AK‐MCS+U, ALR‐Bpce, and ALR‐SVR, the improved rate of the proposed method in efficiency is 51.58%, 12.78%, and 25.96%, respectively. Meanwhile, its prediction accuracy is high and much better than ALR‐ELSM. In addition, the superior performance is robust in a wide range of application cases.

A Multi-View Ensemble Width-Depth Neural Network for Short-Term Wind Power Forecasting

Short-term wind power forecasting (SWPF) is essential for managing wind power systems management. However, most existing forecasting methods fail to fully consider how to rationally integrate multi-view learning technologies with attention mechanisms. In this case, some potential features cannot be fully extracted, degenerating the predictive accuracy and robustness in SWPF. To solve this problem, this paper proposes a multi-view ensemble width-depth neural network (MVEW-DNN) for SWPF. Specifically, MVEW-DNN consists of local and global view learning subnetworks, which can effectively achieve more potential global and local view features of the original wind power data. In MVEW-DNN, the local view learning subnetwork is developed by introducing the deep belief network (DBN) model, which can efficiently extract the local view features. On the other hand, by introducing the attention mechanism, a new deep encoder board learning system (deBLS) is developed as the global view learning subnetwork, which provides more comprehensive global information. Therefore, by rationally learning the effective local and global view features, MVEW-DNN can achieve competitive predictive performance in SWPF. MVEW-DNN is compared with the state-of-the-art models in SWPF. The experiment results indicate that MVEW-DNN can provide competitive predictive accuracy and robustness.

Construction and Evaluation of Robust Interpretation Models for Breast Cancer Metastasis Prediction.

Interpretability of machine learning (ML) models represents the extent to which a model's decision-making process can be understood by model developers and/or end users. Transcriptomics-based cancer prognosis models, for example, while achieving good accuracy, are usually hard to interpret, due to the high-dimensional feature space and the complexity of models. As interpretability is critical for the transparency and fairness of ML models, several algorithms have been proposed to improve the interpretability of arbitrary classifiers. However, evaluation of these algorithms often requires substantial domain knowledge. Here, we propose a breast cancer metastasis prediction model using a very small number of biologically interpretable features, and a simple yet novel model interpretation approach that can provide personalized interpretations. In addition, we contributed, to the best of our knowledge, the first method to quantitatively compare different interpretation algorithms. Experimental results show that our model not only achieved competitive prediction accuracy, but also higher inter-classifier interpretation consistency than state-of-the-art interpretation methods. Importantly, our interpretation results can improve the generalizability of the prediction models. Overall, this work provides several novel ideas to construct and evaluate interpretable ML models that can be valuable to both the cancer machine learning community and related application domains.

A PID-incorporated Latent Factorization of Tensors Approach to Dynamically Weighted Directed Network Analysis

A large-scale dynamically weighted directed network (DWDN) involving numerous entities and massive dynamic interaction is an essential data source in many big-data-related applications, like in a terminal interaction pattern analysis system (TIPAS). It can be represented by a high-dimensional and incomplete (HDI) tensor whose entries are mostly unknown. Yet such an HDI tensor contains a wealth knowledge regarding various desired patterns like potential links in a DWDN. A latent factorization-of-tensors (LFT) model proves to be highly efficient in extracting such knowledge from an HDI tensor, which is commonly achieved via a stochastic gradient descent (SGD) solver. However, an SGD-based LFT model suffers from slow convergence that impairs its efficiency on large-scale DWDNs. To address this issue, this work proposes a proportional-integral-derivative (PID)-incorporated LFT model. It constructs an adjusted instance error based on the PID control principle, and then substitutes it into an SGD solver to improve the convergence rate. Empirical studies on two DWDNs generated by a real TIPAS show that compared with state-of-the-art models, the proposed model achieves significant efficiency gain as well as highly competitive prediction accuracy when handling the task of missing link prediction for a given DWDN.

Toward building a fair peer recommender to support help-seeking in online learning

Help-seeking is a valuable practice in online discussion forums. However, the asynchronicity and information overload of online discussion forums have made it challenging for help seekers and providers to connect effectively. This study formulated a new method to provide fair and accurate insights toward building a peer recommender to support help-seeking in online learning. Specifically, we developed the fair network embedding (Fair-NE) model and compared it with existing popular models. We trained and evaluated the models with a large dataset consisting of 187,450 discussion post-reply pairs by 10,182 Algebra I online learners from 2015 to 2020. Finally, we examined models with representation fairness, predictive accuracy, and predictive fairness. The results showed that the Fair-NE can achieve superior fairness in genders and races while retaining competitive predictive accuracy. This study marks a paradigm change from previous investigation and evaluation of fair artificial intelligence to proactively build fair artificial intelligence in education.

Popularity prediction for social media over arbitrary time horizons

Predicting the popularity of social media content in real time requires approaches that efficiently operate at global scale. Popularity prediction is important for many applications, including detection of harmful viral content to enable timely content moderation. The prediction task is difficult because views result from interactions between user interests, content features, resharing, feed ranking, and network structure. We consider the problem of accurately predicting popularity both at any given prediction time since a content item's creation and for arbitrary time horizons into the future. In order to achieve high accuracy for different prediction time horizons, it is essential for models to use static features (of content and user) as well as observed popularity growth up to prediction time. We propose a feature-based approach based on a self-excited Hawkes point process model, which involves prediction of the content's popularity at one or more reference horizons in tandem with a point predictor of an effective growth parameter that reflects the timescale of popularity growth. This results in a highly scalable method for popularity prediction over arbitrary prediction time horizons that also achieves a high degree of accuracy, compared to several leading baselines, on a dataset of public page content on Facebook over a two-month period, covering billions of content views and hundreds of thousands of distinct content items. The model has shown competitive prediction accuracy against a strong baseline that consists of separately trained models for specific prediction time horizons.