Black Box Solutions Research Articles

Retrospective validation of automatic sleep analysis with grey areas model for human-in-the-loop scoring approach.

State-of-the-art automatic sleep staging methods have demonstrated comparable reliability and superior time efficiency to manual sleep staging. However, fully automatic black-box solutions are difficult to adapt into clinical workflow due to the lack of transparency in decision-making processes. Transparency would be crucial for interaction between automatic methods and the work of sleep experts, i.e., in human-in-the-loop applications. To address these challenges, we propose an automatic sleep staging model (aSAGA) that effectively utilises both electroencephalography and electro-oculography channels while incorporating transparency of uncertainty in the decision-making process. We validated the model through extensive retrospective testing using a range of datasets, including open-access, clinical, and research-driven sources. Our channel-wise ensemble model, trained on both electroencephalography and electro-oculography signals, demonstrated robustness and the ability to generalise across various types of sleep recordings, including novel self-applied home polysomnography. Additionally, we compared model uncertainty with human uncertainty in sleep staging and studied various uncertainty mapping metrics to identify ambiguous regions, or "grey areas", that may require manual re-evaluation. The validation of this grey area concept revealed its potential to enhance sleep staging accuracy and to highlight regions in the recordings where sleep experts may struggle to reach a consensus. In conclusion, this study provides a technical basis and understanding of automatic sleep staging uncertainty. Our approach has the potential to improve the integration of automatic sleep staging into clinical practice; however, further studies are needed to test the model prospectively in real-world clinical settings and human-in-the-loop scoring applications.

Grey-box solution for predicting thermo-mechanical response of rocks

Evaluating the thermo-mechanical response of rocks under high temperature treatments is crucial for various engineering geology projects. Current predictions of rock thermo-mechanical response rely on simplistic mathematical fittings treating temperature as a reduction factor, while existing machine learning algorithms often present practical challenges due to their black-box solutions. In this study, highly practical grey-box solutions, utilizing gene expression programming (GEP) are proposed for forecasting rock strength following high-temperature treatments. The dataset, comprising temperature, rock type, rock density, sample size, crack damage stress, confining pressure, and elastic modulus, serves as input parameters, with rock strength from triaxial compression tests as the output. Three grey-box solutions (mathematical formulations) based on distinct input parameter sets are proposed, all demonstrating excellent accuracy with high R2-values (R2 > 0.95) and low error values across both the training and testing phases. Feature importance analysis highlights crack damage stress, confining pressure, and elastic modulus as statistically significant parameters influencing the strength of rocks subjected to high temperatures. External validation of the proposed models indicates strong generalization capabilities, underscoring their ability to perform well beyond the training data. Furthermore, a monotonicity study demonstrates that the proposed models align with the expected physical processes. The proposed formulations offer valuable field implications, effectively addressing the limitations of labor-intensive and costly laboratory processes for evaluating rock thermo-mechanical responses.

Navigating speed–accuracy trade-offs for multi-physics simulations

AbstractThis paper introduces a novel approach aimed at addressing persistent challenges inherent in conventional multiphysics modeling methodologies. Existing techniques, such as numerical modeling and analytical calculations, often suffer from time-consuming and computationally intensive processes, leading to inefficiencies, particularly in intricate simulations. The proposed methodology employs regression machine learning algorithms as a black-box solution to anticipate errors and execution times in multiphysics simulations. Diverging from conventional methods, this approach streamlines the exploration of simulation options, providing discernible choices for balancing speed and precision. The efficacy of the methodology is exemplified through successful applications to heat transfer and fluid–structure interaction problems, illustrating its adaptability across diverse scenarios. Notably, the approach upholds the integrity of physics equations and simulation convergence while markedly reducing the trial-and-error efforts and computational burdens associated with traditional methodologies. In summary, the proposed approach emerges as an innovative and promising solution to augment the accuracy, efficiency, and dependability of multiphysics simulations.

Few-Shot Learning via Repurposing Ensemble of Black-Box Models

This paper investigates the problem of exploiting existing solution models of previous tasks to address a related target task with limited training data. Existing approaches addressing this problem often require access to the internal parameterization of the existing solution models and possibly their training data, which is not possible in many practical settings. To relax this requirement, We approach this problem from a new perspective of black-box re-purposing, which augments the target inputs and leverages their corresponding outputs generated by existing black-box APIs into a feature ensemble. We hypothesize that such feature ensemble can be learned to incorporate and encode relevant black-box knowledge into the feature representation of target data, which will compensate for their scarcity. This hypothesis is confirmed via the reported successes of our proposed black-box ensemble in solving multiple few-shot learning tasks derived from various benchmark datasets. All reported results show consistently that the set of heterogeneous black-box solutions of previous tasks can indeed be reused and combined effectively to solve a reasonably related target task without requiring access to a large training dataset. This is the first step towards enabling new possibilities to further supplement existing techniques in transfer or meta learning with black-box knowledge.

Radioport: a radiomics-reporting network for interpretable deep learning in BI-RADS classification of mammographic calcification

Objective. Generally, due to a lack of explainability, radiomics based on deep learning has been perceived as a black-box solution for radiologists. Automatic generation of diagnostic reports is a semantic approach to enhance the explanation of deep learning radiomics (DLR). Approach. In this paper, we propose a novel model called radiomics-reporting network (Radioport), which incorporates text attention. This model aims to improve the interpretability of DLR in mammographic calcification diagnosis. Firstly, it employs convolutional neural networks to extract visual features as radiomics for multi-category classification based on breast imaging reporting and data system. Then, it builds a mapping between these visual features and textual features to generate diagnostic reports, incorporating an attention module for improved clarity. Main results. To demonstrate the effectiveness of our proposed model, we conducted experiments on a breast calcification dataset comprising mammograms and diagnostic reports. The results demonstrate that our model can: (i) semantically enhance the interpretability of DLR; and, (ii) improve the readability of generated medical reports. Significance. Our interpretable textual model can explicitly simulate the mammographic calcification diagnosis process.

Automated potential development workflow: Application to BaZrO3

We present a Python workflow, Automated Potential Development (APD), for automating the development of interatomic potentials, including calculation of density functional theory (DFT) fitting data, optimization of potentials, and potential-driven molecular dynamics (MD) simulations. The workflow currently supports CASTEP and VASP DFT codes and the MEAMfit potential optimization code for optimization of reference-free modified embedded atom method (RF-MEAM) potentials. The LAMMPS software is supported for calculating the relaxed geometry, elastic constants, phonon dispersion, thermal expansion and radial distribution functions using the optimized potentials. These same properties are also computed at the DFT level and APD automatically generates plots and tables of these data. Query-by-committee active learning is supported, using multiple fitted potentials to evaluate the energies of atomic configurations generated from LAMMPS MD runs. The workflow is demonstrated on BaZrO3, an oxide-based perovskite material, with RF-MEAM results found in good agreement with DFT. Program summaryProgram title: Automated potential development (APD) workflowCPC Library link to program files:https://doi.org/10.17632/pd9gbxyy8d.1Developer's repository link:https://gitlab.com/AndyDuff123/meamfitLicensing provisions: BSD 3-clauseProgramming language: Python, BashNature of problem: Development of new interatomic potentials is a time-consuming process and requires expertise in density functional theory (DFT), potential optimization, and molecular dynamics (and statics) simulations using the optimized potential. Furthermore repeated cycles are usually required to fix numerical issues and improve the potential, requiring further DFT calculations and potential reoptimization. There are also many tedious steps including setting up inputs, submitting jobs (including continuation jobs where jobs terminated prematurely), analyzing outputs, and validating predictions of the interatomic potential. The latter requires multiple calculations of properties computed at both the DFT and potential level.Solution method: The APD workflow provides a black-box solution to fully automate the generation of interatomic potentials. The workflow takes as input a Materials Project id and temperature range of interest. All DFT, potential optimization, and potential molecular dynamics simulations are automatically set-up, submitted and analyzed. Active-learning is incorporated, and properties are computed both at the DFT and potential-level for validation purposes, with tables and graphs for relaxed geometry, elastic constants, phonon dispersion, partial radial distribution functions, and thermal expansion automatically generated.Additional comments including restrictions and unusual features: A full guide to installation, including setting up an environment with all relevant packages and libraries, are included in the distribution. The APD workflow is currently restricted to RF-MEAM potentials and the properties listed above. Some materials which are particularly challenging at the DFT level, e.g. complex magnetic structures, may require intervention in tuning DFT input parameters.

Deep neural networks for explainable feature extraction in orchid identification

AbstractAutomated image-based plant identification systems are black-boxes, failing to provide an explanation of a classification. Such explanations are seen as being essential by taxonomists and are part of the traditional procedure of plant identification. In this paper, we propose a different method by extracting explicit features from flower images that can be employed to generate explanations. We take the benefit of feature extraction derived from the taxonomic characteristics of plants, with the orchids as an example domain. Feature classifiers were developed using deep neural networks. Two different methods were studied: (1) a separate deep neural network was trained for every individual feature, and (2) a single, multi-label, deep neural network was trained, combining all features. The feature classifiers were tested in predicting 63 orchid species using naive Bayes (NB) and tree-augmented Bayesian networks (TAN). The results show that the accuracy of the feature classifiers is in the range 83-93%. By combining these features using NB and TAN the species can be predicted with an accuracy of 88.9%, which is better than a standard pre-trained deep neural-network architecture, but inferior to a deep learning architecture after fine-tuning of multiple layers. The proposed novel feature extraction method still performs well for identification and is explainable, as opposed to black-box solutions that only aim for the best performance. Graphical abstract

PINN-FORM: A new physics-informed neural network for reliability analysis with partial differential equation

The first-order reliability method (FORM) is commonly used in the field of structural reliability analysis, which transforms the reliability analysis problem into the solution of an optimization problem with equality constraint. However, when the limit state functions (LSFs) in mechanical and engineering problems are complex, particularly for implicit partial differential equations (PDEs), FORM encounters computation difficulty and incurs unbearable computational effort. In this study, the physics-informed neural network (PINN), which is a new branch of deep learning technology for addressing forward and inverse problems with PDEs, is applied as a black-box solution tool. For LSFs with implicit PDE expressions, PINN-FORM is constructed by combining PINN with FORM, which can avoid the calculation of the real structure response. Moreover, a loss function model with an optimization target item is established. Then, an adaptive weight strategy, which can balance the interplay between different parts of the loss function, is suggested to enhance the predictive accuracy. To demonstrate the effectiveness of PINN-FORM, five benchmark examples with LSFs expressed by implicit PDEs, including two-dimensional and three-dimensional problems, and steady state and transient state problems are tested. The results illustrate the proposed PINN-FORM not only is very accurate, but also can simultaneously predict the solutions of PDEs and reliability index within a single training process.

Ensemble Quantile Networks: Uncertainty-Aware Reinforcement Learning With Applications in Autonomous Driving

Reinforcement learning (RL) can be used to create a decision-making agent for autonomous driving. However, previous approaches provide black-box solutions, which do not offer information on how confident the agent is about its decisions. An estimate of both the aleatoric and epistemic uncertainty of the agent’s decisions is fundamental for real-world applications of autonomous driving. Therefore, this paper introduces the Ensemble Quantile Networks (EQN) method, which combines distributional RL with an ensemble approach, to obtain a complete uncertainty estimate. The distribution over returns is estimated by learning its quantile function implicitly, which gives the aleatoric uncertainty, whereas an ensemble of agents is trained on bootstrapped data to provide a Bayesian estimation of the epistemic uncertainty. A criterion for classifying which decisions that have an unacceptable uncertainty is also introduced. The results show that the EQN method can balance risk and time efficiency in different occluded intersection scenarios, by considering the estimated aleatoric uncertainty. Furthermore, it is shown that the trained agent can use the epistemic uncertainty information to identify situations that the agent has not been trained for and thereby avoid making unfounded, potentially dangerous, decisions outside of the training distribution.

Tail Prediction for Heterogeneous Data Center Clusters

Service providers need to meet their service level objectives (SLOs) to ensure better client experiences. Predicting tail sojourn times of applications is an essential step to combat long tail latency. Therefore, as an attempt to further unravel the power of our prediction model, new study scenarios for heterogeneous environments will be introduced in this research by using either of two methods: white- or black-box solutions. This research presents several techniques for modeling clusters of inhomogeneous nodes. Those techniques are recognized as heterogeneous fork-join queuing networks (HFJQNs). Moreover, included in the research is a nested-event-based simulation model, borrowing help from multi-core technologies. This model adopts the multiprocessing technique to take part in its design to enable different architectural designs for all computing nodes. This novel implementation of the simulation model is believed to be the next logical step for research studies targeting heterogeneous clusters in addition to the several provided scenarios. Experimental results confirm that even with the existence of such heterogeneous conditions, the tail latency can be predicted at high-load regions with an approximated relative error of less than 15%.

Fuzzy rough nearest neighbour methods for detecting emotions, hate speech and irony

Due to the ever-expanding volumes of information available on social media, the need for reliable and efficient automated text understanding mechanisms becomes evident. Unfortunately, most current approaches rely on black-box solutions rooted in deep learning technologies. In order to provide a more transparent and interpretable framework for extracting intrinsic text characteristics like emotions, hate speech and irony, we propose to integrate fuzzy rough set techniques and text embeddings. We apply our methods to different classification problems originating from Semantic Evaluation (SemEval) competitions, and demonstrate that their accuracy is on par with leading deep learning solutions.

Computing hydrodynamic eigenmodes of channel flow with slip—A highly accurate algorithm

AbstractThe transient start‐up flow solution with slip is a useful tool to verify computational fluid dynamics (CFD) simulations. However, a highly accurate, open‐source black box solution does not seem to be available. Our method provides a fast, automated, and rigorously verified open‐source implementation that can compute the hydrodynamic eigenmodes of a two‐dimensional channel flow beyond the standard floating‐point precision. This allows for a very accurate computation of the corresponding Fourier series solution. We prove that all roots are found in all special cases for the general flow problem with different slip lengths on the channel walls. The numerical results confirm analytically derived asymptotic power laws for the leading hydrodynamic eigenmode and the characteristic timescale in the limiting cases of small and large slip. The code repository including test cases is publicly available (DOI: 10.5281/zenodo.6806351). The Navier slip boundary condition for numerical simulations in OpenFOAM is also publicly available (DOI: 10.5281/zenodo.7037712).

Machine learning of material properties: Predictive and interpretable multilinear models.

Machine learning models can provide fast and accurate predictions of material properties but often lack transparency. Interpretability techniques can be used with black box solutions, or alternatively, models can be created that are directly interpretable. We revisit material datasets used in several works and demonstrate that simple linear combinations of nonlinear basis functions can be created, which have comparable accuracy to the kernel and neural network approaches originally used. Linear solutions can accurately predict the bandgap and formation energy of transparent conducting oxides, the spin states for transition metal complexes, and the formation energy for elpasolite structures. We demonstrate how linear solutions can provide interpretable predictive models and highlight the new insights that can be found when a model can be directly understood from its coefficients and functional form. Furthermore, we discuss how to recognize when intrinsically interpretable solutions may be the best route to interpretability.

CCT: Conditional Co-Training for Truly Unsupervised Remote Sensing Image Segmentation in Coastal Areas

As the fastest growing trend in big data analysis, deep learning technology has proven to be both an unprecedented breakthrough and a powerful tool in many fields, particularly for image segmentation tasks. Nevertheless, most achievements depend on high-quality pre-labeled training samples, which are labor-intensive and time-consuming. Furthermore, different from conventional natural images, coastal remote sensing ones generally carry far more complicated and considerable land cover information, making it difficult to produce pre-labeled references for supervised image segmentation. In our research, motivated by this observation, we take an in-depth investigation on the utilization of neural networks for unsupervised learning and propose a novel method, namely conditional co-training (CCT), specifically for truly unsupervised remote sensing image segmentation in coastal areas. In our idea, a multi-model framework consisting of two parallel data streams, which are superpixel-based over-segmentation and pixel-level semantic segmentation, is proposed to simultaneously perform the pixel-level classification. The former processes the input image into multiple over-segments, providing self-constrained guidance for model training. Meanwhile, with this guidance, the latter continuously processes the input image into multi-channel response maps until the model converges. Incentivized by multiple conditional constraints, our framework learns to extract high-level semantic knowledge and produce full-resolution segmentation maps without pre-labeled ground truths. Compared to the black-box solutions in conventional supervised learning manners, this method is of stronger explainability and transparency for its specific architecture and mechanism. The experimental results on two representative real-world coastal remote sensing datasets of image segmentation and the comparison with other state-of-the-art truly unsupervised methods validate the plausible performance and excellent efficiency of our proposed CCT.

Convolutional neural networks can decode eye movement data: A black box approach to predicting task from eye movements.

Previous attempts to classify task from eye movement data have relied on model architectures designed to emulate theoretically defined cognitive processes and/or data that have been processed into aggregate (e.g., fixations, saccades) or statistical (e.g., fixation density) features. Black box convolutional neural networks (CNNs) are capable of identifying relevant features in raw and minimally processed data and images, but difficulty interpreting these model architectures has contributed to challenges in generalizing lab-trained CNNs to applied contexts. In the current study, a CNN classifier was used to classify task from two eye movement datasets (Exploratory and Confirmatory) in which participants searched, memorized, or rated indoor and outdoor scene images. The Exploratory dataset was used to tune the hyperparameters of the model, and the resulting model architecture was retrained, validated, and tested on the Confirmatory dataset. The data were formatted into timelines (i.e., x-coordinate, y-coordinate, pupil size) and minimally processed images. To further understand the informational value of each component of the eye movement data, the timeline and image datasets were broken down into subsets with one or more components systematically removed. Classification of the timeline data consistently outperformed the image data. The Memorize condition was most often confused with Search and Rate. Pupil size was the least uniquely informative component when compared with the x- and y-coordinates. The general pattern of results for the Exploratory dataset was replicated in the Confirmatory dataset. Overall, the present study provides a practical and reliable black box solution to classifying task from eye movement data.

An Ensemble Learning Solution for Predictive Maintenance of Wind Turbines Main Bearing.

A novel and innovative solution addressing wind turbines’ main bearing failure predictions using SCADA data is presented. This methodology enables to cut setup times and has more flexible requirements when compared to the current predictive algorithms. The proposed solution is entirely unsupervised as it does not require the labeling of data through work orders logs. Results of interpretable algorithms, which are tailored to capture specific aspects of main bearing failures, are merged into a combined health status indicator making use of Ensemble Learning principles. Based on multiple specialized indicators, the interpretability of the results is greater compared to black-box solutions that try to address the problem with a single complex algorithm. The proposed methodology has been tested on a dataset covering more than two year of operations from two onshore wind farms, counting a total of 84 turbines. All four main bearing failures are anticipated at least one month of time in advance. Combining individual indicators into a composed one proved effective with regard to all the tracked metrics. Accuracy of 95.1%, precision of 24.5% and F1 score of 38.5% are obtained averaging the values across the two windfarms. The encouraging results, the unsupervised nature and the flexibility and scalability of the proposed solution are appealing, making it particularly attractive for any online monitoring system used on single wind farms as well as entire wind turbine fleets.

Combining Supervised Learning and Digital Twin for Autonomous Path-planning

Abstract Over the last decade, the evolution of autonomous automobiles based on artificial intelligence has increased rapidly with significant success. Naturally, this has caught the interest of the maritime industry and the development of autonomous vessels. However, unlike the highway, the ocean is considered a complex environment carrying unpredictable environmental forces, such as current, waves and wind-condition. For autonomous path-following and path-planning, particularly within the machine learning-field, Deep Reinforcement Learning (DRL) have generally been the favored approach. This follows from the fact that resulting models have demonstrated staggering performance. However, for practical implementations, Deep learning-based models are generally considered black box-solutions, and hence often introduce uncertainties in the operating domain. Therefore, in this paper an autonomous path-planner based on Supervised learning is proposed. Different Supervised learning models were investigated, and Gradient Boosting Regressor was found to be the most adequate model based on hyperparameter-tuning. The model was developed on constraints proposed by the class society DNV GL combined with International Regulations for Preventing Collision at Sea (COLREGs) rule 14 for collision-avoidance. Following this, the model was trained to design a suitable path based on parametrization of a cubic Bezier curve. To follow the parametrized path, a maneuvering-controller derived from the Maneuvering problem presented in Skjetne (2005) was applied. However, a drawback of Supervised learning is the necessity for large-scale training data. Hence, a digital twin of the own vessel was developed and utilized to generate sufficient training data. To demonstrate the performance of the autonomous path-planner, a number of simulation scenarios were introduced.

EFFICIENT COMPUTATION OF POSTERIOR COVARIANCE IN BUNDLE ADJUSTMENT IN DBAT FOR PROJECTS WITH LARGE NUMBER OF OBJECT POINTS

Abstract. One of the major quality control parameters in bundle adjustment are the posterior estimates of the covariance of the estimated parameters. Posterior covariance computations have been part of the open source Damped Bundle Adjustment Toolbox in Matlab (DBAT) since its first public release. However, for large projects, the computation of especially the posterior covariances of object points have been time consuming.The non-zero structure of the normal matrix depends on the ordering of the parameters to be estimated. For some algorithms, the ordering of the parameters highly affect the computational effort needed to compute the results. If the parameters are ordered to have the object points first, the non-zero structure of the normal matrix forms an arrowhead.In this paper, the legacy DBAT posterior computation algorithm was compared to three other algorithms: The Classic algorithm based on the reduced normal equation, the Sparse Inverse algorithm by Takahashi, and the novel Inverse Cholesky algorithm. The Inverse Cholesky algorithm computes the explicit inverse of the Cholesky factor of the normal matrix in arrowhead ordering.The algorithms were applied to normal matrices of ten data sets of different types and sizes. The project sizes ranged from 21 images and 100 object points to over 900 images and 400,000 object points. Both self-calibration and non-self-calibration cases were investigated. The results suggest that the Inverse Cholesky algorithm is the fastest for projects up to about 300 images. For larger projects, the Classic algorithm is faster. Compared to the legacy DBAT implementation, the Inverse Cholesky algorithm provides a performance increase by one to two orders of magnitude. The largest data set was processed in about three minutes on a five year old workstation.The legacy and Inverse Cholesky algorithms were implemented in Matlab. The Classic and Sparse Inverse algorithms included code written in C. For a general toolbox as DBAT, a pure Matlab implementation is advantageous, as it removes any dependencies on, e.g., compilers. However, for a specific lab with mostly large projects, compiling and using the classic algorithm will most likely give the best performance. Nevertheless, the Inverse Cholesky algorithm is a significant addition to DBAT as it enables a relatively rapid computation of more statistical metrics, further reinforcing its application for reprocessing bundle adjustment results of black-box solutions.

Combining machine learning and process engineering physics towards enhanced accuracy and explainability of data-driven models

Machine learning models are often considered as black-box solutions which is one of the main reasons why they are still not widely used in operation of process engineering systems. One approach to overcome this problem is to combine machine learning with first principles models of a process engineering system. In this work, we investigate different methods of combining machine learning with first principles and test them on a case study of multiphase flowrate estimation in a petroleum production system. However, the methods can be applied to any process engineering system. The results show that by adding physics-based models to machine learning, it is possible not only to improve the performance of the purely black-box machine learning models, but also to make them more transparent and interpretable. We also propose a step-by-step procedure for selecting a method for combining physics and machine learning depending on the process engineering system conditions.

Machine Intelligence in Cardiovascular Medicine.

The computer science technology trend called artificial intelligence (AI) is not new. Both machine learning and deep learning AI applications have recently begun to impact cardiovascular medicine. Scientists working in the AI domain have long recognized the importance of data quality and provenance to AI algorithm efficiency and accuracy. A diverse array of cardiovascular raw data sources of variable quality-electronic medical records, radiological picture archiving and communication systems, laboratory results, omics, etc.-are available to train AI algorithms for predictive modeling of clinical outcomes (in-hospital mortality, acute coronary syndrome risk stratification, etc.), accelerated image interpretation (edge detection, tissue characterization, etc.) and enhanced phenotyping of heterogeneous conditions (heart failure with preserved ejection fraction, hypertension, etc.). A number of software as medical device narrow AI products for cardiac arrhythmia characterization and advanced image deconvolution are now Food and Drug Administration approved, and many others are in the pipeline. Present and future health professionals using AI-infused analytics and wearable devices have 3 critical roles to play in their informed development and ethical application in practice: (1) medical domain experts providing clinical context to computer and data scientists, (2) data stewards assuring the quality, relevance and provenance of data inputs, and (3) real-time and post-hoc interpreters of AI black box solutions and recommendations to patients. The next wave of so-called contextual adaption AI technologies will more closely approximate human decision-making, potentially augmenting cardiologists' real-time performance in emergency rooms, catheterization laboratories, imaging suites, and clinics. However, before such higher order AI technologies are adopted in the clinical setting and by healthcare systems, regulatory agencies, and industry must jointly develop robust AI standards of practice and transparent technology insertion rule sets.