Complex Machine Learning Methods Research Articles

XAI-TRIS: non-linear image benchmarks to quantify false positive post-hoc attribution of feature importance

AbstractThe field of ‘explainable’ artificial intelligence (XAI) has produced highly acclaimed methods that seek to make the decisions of complex machine learning (ML) methods ‘understandable’ to humans, for example by attributing ‘importance’ scores to input features. Yet, a lack of formal underpinning leaves it unclear as to what conclusions can safely be drawn from the results of a given XAI method and has also so far hindered the theoretical verification and empirical validation of XAI methods. This means that challenging non-linear problems, typically solved by deep neural networks, presently lack appropriate remedies. Here, we craft benchmark datasets for one linear and three different non-linear classification scenarios, in which the important class-conditional features are known by design, serving as ground truth explanations. Using novel quantitative metrics, we benchmark the explanation performance of a wide set of XAI methods across three deep learning model architectures. We show that popular XAI methods are often unable to significantly outperform random performance baselines and edge detection methods, attributing false-positive importance to features with no statistical relationship to the prediction target rather than truly important features. Moreover, we demonstrate that explanations derived from different model architectures can be vastly different; thus, prone to misinterpretation even under controlled conditions.

False Discovery Rate Control For Structured Multiple Testing: Asymmetric Rules And Conformal Q-values

The effective utilization of structural information in data while ensuring statistical validity poses a significant challenge in false discovery rate (FDR) analyses. Conformal inference provides rigorous theory for grounding complex machine learning methods without relying on strong assumptions or highly idealized models. However, existing conformal methods have limitations in handling structured multiple testing, as their validity often requires the deployment of symmetric decision rules, which assume the exchangeability of data points and permutation-invariance of fitting algorithms. To overcome these limitations, we introduce the pseudo local index of significance (PLIS) procedure, which is capable of accommodating asymmetric rules and requires only pairwise exchangeability between the null conformity scores. We demonstrate that PLIS offers finite-sample guarantees in FDR control and the ability to assign higher weights to relevant data points. Numerical results confirm the effectiveness and robustness of PLIS and demonstrate improvements in power compared to existing model-free methods in various scenarios.

An explainable AI framework for robust and transparent data-driven wind turbine power curve models

In recent years, increasingly complex machine learning methods have become state-of-the-art in modelling wind turbine power curves based on operational data. While these methods often exhibit superior performance on test sets, they face criticism due to a perceived lack of transparency and concerns about their robustness in dynamic, non-stationary environments encountered by wind turbines. In this work, we address these issues and present a framework that leverages explainable artificial intelligence methods to gain systematic insights into data-driven power curve models. At its core, we propose a metric to quantify how well a learned model strategy aligns with the underlying physical principles of the problem. This novel tool enables model validation beyond the conventional error metrics in an automated manner. We demonstrate, for instance, its capacity as an indicator for model generalization even when limited data is available. Moreover, it facilitates understanding how decisions made during the machine learning development process, such as data selection, pre-processing, or training parameters, affect learned strategies. As a result, we obtain physically more reasonable models, a prerequisite not only for robustness but also for meaningful insights into turbine operation by domain experts. The latter, we illustrate in the context of wind turbine performance monitoring. In summary, the framework aims to guide researchers and practitioners alike toward a more informed selection and utilization of data-driven wind turbine power curve models.

Improving prediction of mountain snowfall in the southwestern United States using machine learning methods

AbstractSnowfall forecasting has historically been an area of difficulty for operational meteorologists, particularly in regions of complex terrain, such as the western United States. Attempts at improving forecasts have been made, but skill is still poor, with snowfall routinely overpredicted. A major reason for this overprediction has been the failure to accurately predict snow–liquid ratios (SLR) ahead of major events. This research proposes, develops, and tests multiple machine learning methods for dynamic SLR prediction for the Sky Islands of southeast Arizona by objectively comparing a multiple linear regression (MLR) against several more complex and flexible machine learning methods. Input parameters for each method were chosen based on variables found by previous studies to have a regression‐based relationship with SLR, with a focus on the lower mid‐levels of the troposphere. These parameters were also used to construct the MLR model, and its performance was compared objectively with the machine learning methods. When tested on historical events, a very high percentage of the network‐predicted SLR values fall within the margin of error of observed SLRs, which were calculated using gridded snow depth and snow water equivalent (SWE) data from the University of Arizona daily 4‐km SWE, SD, and SCE dataset (UASnow). A support vector machine (SVM), a k‐nearest neighbor (KNN) algorithm, and a random forest also showed high accuracies when tested on the dataset, and each showed a significant gain in skill compared with the MLR model, with skill being evaluated by multiple metrics.

Rationalizing general limitations in assessing and comparing methods for compound potency prediction

Compound potency predictions play a major role in computational drug discovery. Predictive methods are typically evaluated and compared in benchmark calculations that are widely applied. Previous studies have revealed intrinsic limitations of potency prediction benchmarks including very similar performance of increasingly complex machine learning methods and simple controls and narrow error margins separating machine learning from randomized predictions. However, origins of these limitations are currently unknown. We have carried out an in-depth analysis of potential reasons leading to artificial outcomes of potency predictions using different methods. Potency predictions on activity classes typically used in benchmark settings were found to be determined by compounds with intermediate potency close to median values of the compound data sets. The potency of these compounds was consistently predicted with high accuracy, without the need for learning, which dominated the results of benchmark calculations, regardless of the activity classes used. Taken together, our findings provide a clear rationale for general limitations of compound potency benchmark predictions and a basis for the design of alternative test systems for methodological comparisons.

Experimental tracking of an ultrasonic source with unknown dynamics using a stereo sensor

Sound source localization (SSL) is the ability to successfully estimate the bearing and distance of a sound in space relative to the sensing position and pose. SSL as a topic of interest for engineers often revolves around the ability of robots to track other robots, human voices, or other acoustic objects. Common approaches to this goal frequently use large arrays, computationally intensive and complex machine learning methods, or require known dynamic models of a system which may not always be available. In this work we seek to experimentally verify a solution to SSL using a minimal amount of inexpensive equipment on a two microphone, or stereo, sensing platform. A previously developed Bayesian estimator allows for localization of an emitter using easily available a priori information and timing data received from the sensor platform. Our results show that our approach is accurate for the tested paths and that the estimator can correct itself when dynamic assumptions are broken for short times due to hardware and software limitations.

Processing of Al/SiC/Gr Hybrid Composite on EDM by Different Electrode Materials Using RSM-COPRAS Approach

The present research used the stir-casting method to develop an Al-based composite. The developed composite exhibited challenges while being processed on conventional machining. Thus, a non-traditional machining process was opted to process the composite. The machining variables selected for the current research were the pulse off time (Toff), pulse on time (Ton), servo voltage (SV), current (I), and tool electrode. Three tool electrodes (SS-304, copper, and brass) were used to process the developed composite (Al/SiC/Gr). The experimental plan was designed using response surface methodology (RSM). The output responses recorded for the analysis were the material removal rate (MRR) and tool wear rate (TWR). The obtained data was optimized using complex proportional assessment (COPRAS) and machine learning methods. The optimized settings predicted by the RSM–COPRAS method were Ton: 60 µs; Toff: 60 µs; SV: 7 V; I: 12 A; and tool: brass. The maximum MRR and TWR at the suggested settings were 1.11 g/s and 0.0114 g/s, respectively. A morphological investigation of the machined surface and tool surface was conducted with scanning electron microscopy. The morphological examination of the surface (machined) presented the presence of cracks, lumps, etc.

Complex modeling with detailed temporal predictors does not improve health records-based suicide risk prediction

Suicide risk prediction models can identify individuals for targeted intervention. Discussions of transparency, explainability, and transportability in machine learning presume complex prediction models with many variables outperform simpler models. We compared random forest, artificial neural network, and ensemble models with 1500 temporally defined predictors to logistic regression models. Data from 25,800,888 mental health visits made by 3,081,420 individuals in 7 health systems were used to train and evaluate suicidal behavior prediction models. Model performance was compared across several measures. All models performed well (area under the receiver operating curve [AUC]: 0.794–0.858). Ensemble models performed best, but improvements over a regression model with 100 predictors were minimal (AUC improvements: 0.006–0.020). Results are consistent across performance metrics and subgroups defined by race, ethnicity, and sex. Our results suggest simpler parametric models, which are easier to implement as part of routine clinical practice, perform comparably to more complex machine learning methods.

Design of an efficient energy management system for renewables based wireless electric vehicle charging station

AbstractThe radical decline in climatic conditions coerces the world to take steps progressive toward implementing green energy technologies in all facets. The electric vehicle is one such huge leap toward providing a pollution‐free environment to the future generation, but the charging techniques in most cases utilize the power that is obtained using thermal power stations, and so on. This concludes it to be not completely green energy, therefore in this paper, considered two far‐reaching sources viz. photovoltaic and wind energy storage systems. The sources are also supported with the presence of an additional storage unit of battery energy storage system (BESS) and the overall system is managed by energy management system (EMS) that is designed using a simple state flow algorithm instead of complex machine learning methods. The electric vehicle (EV) is charged using wireless power transfer technology. The model is built using MATLAB—Simulink software and the simulation results are validated against the physical calculation carried out over the equivalent circuit model reduction method. The values were taken from the TATA Nexon EV for consideration and the output values are 400 V and 15 A to be provided for the charging of the EV at the receiving end of the wireless charging.

Distributed Intermittent Fault Diagnosis in Wireless Sensor Network Using Likelihood Ratio Test

In current days, sensor nodes are deployed in hostile environments for various military and commercial applications. Sensor nodes are becoming faulty and having adverse effects in the network if they are not diagnosed and inform the fault status to other nodes. Fault diagnosis is difficult when the nodes behave faulty some times and provide good data at other times. The intermittent disturbances may be random or kind of spikes either in regular or irregular intervals. In literature, the fault diagnosis algorithms are based on statistical methods using repeated testing or machine learning. To avoid more complex and time consuming repeated test processes and computationally complex machine learning methods, we proposed a one shot likelihood ratio test (LRT) here to determine the fault status of the sensor node. The proposed method measures the statistics of the received data over a certain period of time and then compares the likelihood ratio with the threshold value associated with a certain tolerance limit. The simulation results using a real time data set shows that the new method provides better detection accuracy (DA) with minimum false positive rate (FPR) and false alarm rate (FAR) over the modified three sigma test. LRT based hybrid fault diagnosis method detecting the fault status of a sensor node in wireless sensor network (WSN) for real time measured data with 100% DA, 0% FAR and 0% FPR if the probability of the data from faulty node exceeds 25%.

Embracing imperfection: Machine-assisted invertebrate classification in real-world datasets

Despite growing concerns over the health of global invertebrate diversity, terrestrial invertebrate monitoring efforts remain poorly geographically distributed. Machine-assisted classification has been proposed as a potential solution to quickly gather large amounts of data; however, previous studies have often used unrealistic or idealized datasets to train and test their models.In this study, we describe a practical methodology for including machine learning in ecological data acquisition pipelines. Here we train and test machine learning algorithms to classify over 72,000 terrestrial invertebrate specimens from morphometric data and contextual metadata. All vouchered specimens were collected in pitfall traps by the National Ecological Observatory Network (NEON) at 45 locations across the United States from 2016 to 2019. Specimens were photographed, and two separate machine learning paradigms were used to classify them. In the first, we used a convolutional neural network (ResNet-50), and in the second, we extracted morphometric data as feature vectors using ImageJ and used traditional machine learning methods to classify specimens. Issues stemming from inconsistent taxonomic label specificity were resolved by making classifications at the lowest identified taxonomic level (LITL). Taxa with too few specimens to be included in the training dataset were classified by the model using zero-shot classification.When classifying specimens that were known and seen by our models, we reached a maximum accuracy of 72.7% using eXtreme Gradient Boosting (XGBoost) at the LITL. This nearly matched the maximum accuracy achieved by the CNN of 72.8% at the LITL. Models that were trained without contextual metadata underperformed models with contextual metadata. We also classified invertebrate taxa that were unknown to the model using zero-shot classification, reaching a maximum accuracy of 65.5% when using the ResNet-50, compared to 39.4% when using XGBoost.The general methodology outlined here represents a realistic application of machine learning as a tool for ecological studies. We found that more advanced and complex machine learning methods such as convolutional neural networks are not necessarily more accurate than traditional machine learning methods. Hierarchical and LITL classifications allow for flexible taxonomic specificity at the input and output layers. These methods also help address the ‘long tail’ problem of underrepresented taxa missed by machine learning models. Finally, we encourage researchers to consider more than just morphometric data when training their models, as we have shown that the inclusion of contextual metadata can provide significant improvements to accuracy.

An investigation of privacy preservation in deep learning-based eye-tracking

BackgroundThe expanding usage of complex machine learning methods such as deep learning has led to an explosion in human activity recognition, particularly applied to health. However, complex models which handle private and sometimes protected data, raise concerns about the potential leak of identifiable data. In this work, we focus on the case of a deep network model trained on images of individual faces.Materials and methodsA previously published deep learning model, trained to estimate the gaze from full-face image sequences was stress tested for personal information leakage by a white box inference attack. Full-face video recordings taken from 493 individuals undergoing an eye-tracking- based evaluation of neurological function were used. Outputs, gradients, intermediate layer outputs, loss, and labels were used as inputs for a deep network with an added support vector machine emission layer to recognize membership in the training data.ResultsThe inference attack method and associated mathematical analysis indicate that there is a low likelihood of unintended memorization of facial features in the deep learning model.ConclusionsIn this study, it is showed that the named model preserves the integrity of training data with reasonable confidence. The same process can be implemented in similar conditions for different models.

A Case Study Investigating a User-Centred and Expert Informed 'Companion Guide' for a Complex Sensor-based Platform

We present a case study that informs the creation of a 'companion guide' providing transparency to potential non-expert users of a ubiquitous machine learning (ML) platform during the initial onboarding. Ubiquitous platforms (e.g., smart home systems, including smart meters and conversational agents) are increasingly commonplace and increasingly apply complex ML methods. Understanding how non-ML experts comprehend these platforms is important in supporting participants in making an informed choice about if and how they adopt these platforms. To aid this decision-making process, we created a companion guide for a home health platform through an iterative user-centred-design process, seeking additional input from platform experts at all stages of the process to ensure the accuracy of explanations. This user-centred and expert informed design process highlights the need to present the platform's entire ecosystem at an appropriate level for those with differing backgrounds to understand, in order to support informed consent and decision making.

Abstract 6394: The SB Image Processing Toolkit: Machine learning for cancer research on the CGC

Abstract Applying machine learning algorithms to analysis of multi-omics datasets has provided a wealth of analysis and interpretations of data not easily achievable by conventional methods. The Cancer Genomics Cloud (CGC), powered by Seven Bridges, offers many helpful features to simplify and streamline the performance of machine learning (ML) tasks on the platform. The CGC platform contains features and tools for all of the steps of a ML project, from data exploration, to model generation, to production, so that all key steps occur on the platform. The Data Cruncher interactive analysis tool enables users to perform data exploration, modeling, training, and visualization using familiar frameworks like Jupyterlab and RStudio. We applied this ML methodology to imaging tools, to create The SB Image Processing Toolkit. Here we demonstrate the use of this toolkit with specific cancer datasets, including training and analysis, with all steps taking place on the cloud. The SB Image Processing Toolkit is a collection of various deep-learning, preprocessing, and utility tools and workflows created with the purpose of performing image class prediction on any type of image data. The toolkit supports a variety of images used in cancer research, such as histopathology and radiology images, and common formats like JPG or PNG. There are nine distinct tools within the toolkit, including methods for quality control, normalization, preprocessing for histopathology or radiology images, classification, and deep learning. Together, these tools create an easy-to-use infrastructure to enable various stages of ML and Image Processing. Most importantly, it was designed to enable users to create and use ML image classifiers on the CGC without any coding experience. All workflows can be searched for and implemented via the Public Apps Gallery on the CGC. Users can import image data, run analysis, and compare results all in one environment. By harnessing the scale and flexibility of cloud computing, the SB Image Processing Toolkit brings massive speed and price improvements compared to manual writing and running model training scripts. In conjunction with the access to data and ease of use, the CGC makes complex machine learning methods accessible to any researcher. Citation Format: Soner Koc, Jovana Babic, Nevena Nikolic, Ana Stankovic, Daniel Ventre, Manisha Ray, Zelia F. Worman, Sai Lakshmi Subramanian, Dennis A. Dean, Jack DiGiovanna, Brandi Davis Dusenbery. The SB Image Processing Toolkit: Machine learning for cancer research on the CGC [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2022; 2022 Apr 8-13. Philadelphia (PA): AACR; Cancer Res 2022;82(12_Suppl):Abstract nr 6394.

Deep Reinforcement Learning for Stock Prediction

Investors are frequently concerned with the potential return from changes in a company’s stock price. However, stock price fluctuations are frequently highly nonlinear and nonstationary, rendering them to be uncontrollable and the primary reason why the majority of investors earn low long-term returns. Historically, people have always simulated and predicted using classic econometric models and simple machine learning models. In recent years, an increasing amount of research has been conducted using more complex machine learning and deep learning methods to forecast stock prices, and their research reports also indicate that their prediction accuracy is gradually improving. While the prediction results and accuracy of these models improve over time, their adaptability in a volatile market environment is questioned. Highly optimized machine learning algorithms include the following: FNN and the RNN are incapable of predicting the stock price of random walks and their results are frequently not consistent with stock price movements. The purpose of this article is to increase the accuracy and speed of stock price volatility prediction by incorporating the PG method’s deep reinforcement learning model. Finally, our tests demonstrate that the new algorithm’s prediction accuracy and reward convergence speed are significantly higher than those of the traditional DRL algorithm. As a result, the new algorithm is more adaptable to fluctuating market conditions.

Fair-AdaBoost: Extending AdaBoost method to achieve fair classification

Machine learning-based classification models are ubiquitous in a wide variety of classification tasks. These models usually rely on complex machine learning methods and vast historical data to make accurate predictions. Recently, a concern grows that these models lead to the unfair classifications, i.e., model predictions would hurt or benefit particular group of people based on sensitive features (e.g., race, age). To achieve fair classification, a popular machine learning method, AdaBoost, is extended into the Fair-AdaBoost method in this study. Fair-AdaBoost can achieve fair classification while preserving the advantages (i.e., interpretability, scalability, and accuracy) of the basic AdaBoost. To further enhance the performance of Fair-AdaBoost, the non-dominated sorting genetic algorithm-II is extended in this study to optimize the hyper-parameters of the base classifiers in Fair-AdaBoost. In the experiments, the proposed method and algorithm have been tested on three standard benchmark datasets widely used in the literature of fair classification, which has proven their superior performance.

Multidimensional nearest neighbors classification based system for incomplete lip print identification

This paper presents a method of personal identification via the analysis of lip print images. This remains a little explored field even within the most serious and leading biometric research teams. Biometrics is the scientific study of the identification and verification of individuals based on their physiological and behavioral traits. Such traits are permanent, unique, and can be used to separately identify any one individual from any another. In the method we present here, we integrated complex image processing techniques, machine learning, and statistical methods. We then evaluated this new method on previously collected realistic (i.e. low quality) real-world lip print images. Multi-variant experimental protocols, specifically designed for this work, then confirmed the accuracy of our new technique. Our results have extended the knowledge of – and the collection of methods available for – the identification of biometric objects from incomplete data sets. Biometric analysis techniques are rapidly gaining in importance. The approach we propose here will be useful in many areas including biometrics, forensics, and forensic medicine. The novelty of this proposed method is its ability to work on lip print images that are partially corrupted or incomplete, as often occurs in practice. Low quality areas are recognized and effectively eliminated. These areas are neither taken into account during the classifier learning process nor later during the samples’ classification. This was confirmed in a series of experiments in which the best classification accuracy achieved was 94.40%.

In silico prediction models for thyroid peroxidase inhibitors and their application to synthetic flavors

Systematic toxicity tests are often waived for the synthetic flavors as they are added in a very small amount in foods. However, their safety for some endpoints such as endocrine disruption should be concerned as they are likely to be active in low levels. In this case, structure–activity-relationship (SAR) models are good alternatives. In this study, therefore, binary, ternary, and quaternary prediction models were designed using simple or complex machine-learning methods. Overall, hard-voting classifiers outperformed other methods. The test scores for the best binary, ternary, and quaternary models were 0.6635, 0.5083, and 0.5217, respectively. Along with model development, some substructures including primary aromatic amine, (enol)ether, phenol, heterocyclic sulfur, and heterocyclic nitrogen, dominantly occurred in the most highly active compounds. The best predicting models were applied to synthetic flavors, and 22 agents appeared to have a strong inhibitory potential towards TPO activities.

Error Quantification for the Assessment of Data-Driven Turbulence Models

Data-driven turbulence modelling is becoming common practice in the field of fluid mechanics. Complex machine learning methods are applied to large high fidelity data sets in an attempt to discover relationships between mean flow features and turbulence model parameters. However, a clear discrepancy is emerging between complex models that appear to fit the high fidelity data well a priori and simpler models which subsequently hold up in a posteriori testing through CFD simulations. With this in mind, a novel error quantification technique is proposed consisting of an upper and lower bound, against which data-driven turbulence models can be systematically assessed. At the lower bound is models that are linear in either the full set or a subset of the input features, where feature selection is used to determine the best model. Any machine learning technique must be able to improve on this performance for the extra complexity in training to be of practical use. The upper bound is found by the stable insertion of the high fidelity data for the Reynolds stresses into CFD simulation. Three machine learning methods, Gene Expression Programming, Deep Neural Networks and Gaussian Mixtures Models are presented and assessed on this error quantification technique. We further show that for the simple canonical cases often used to develop data-driven methods, lower bound linear models can provide very satisfactory accuracy and stability with limited scope for substantial improvement through more complex machine learning methods.

SARS-CoV-2 infection and cardiovascular or pulmonary complications in ambulatory care: A risk assessment based on routine data.

Risk factors of severe COVID-19 have mainly been investigated in the hospital setting. We investigated pre-defined risk factors for testing positive for SARS-CoV-2 infection and cardiovascular or pulmonary complications in the outpatient setting. The present cohort study makes use of ambulatory claims data of statutory health insurance physicians in Bavaria, Germany, with polymerase chain reaction (PCR) test confirmed or excluded SARS-CoV-2 infection in first three quarters of 2020. Statistical modelling and machine learning were used for effect estimation and for hypothesis testing of risk factors, and for prognostic modelling of cardiovascular or pulmonary complications. A cohort of 99 811 participants with PCR test was identified. In a fully adjusted multivariable regression model, dementia (odds ratio (OR) = 1.36), type 2 diabetes (OR = 1.14) and obesity (OR = 1.08) were identified as significantly associated with a positive PCR test result. Significant risk factors for cardiovascular or pulmonary complications were coronary heart disease (CHD) (OR = 2.58), hypertension (OR = 1.65), tobacco consumption (OR = 1.56), chronic obstructive pulmonary disease (COPD) (OR = 1.53), previous pneumonia (OR = 1.53), chronic kidney disease (CKD) (OR = 1.25) and type 2 diabetes (OR = 1.23). Three simple decision rules derived from prognostic modelling based on age, hypertension, CKD, COPD and CHD were able to identify high risk patients with a sensitivity of 74.8% and a specificity of 80.0%. The decision rules achieved a high prognostic accuracy non-inferior to complex machine learning methods. They might help to identify patients at risk, who should receive special attention and intensified protection in ambulatory care.