Probabilistic Machine Learning Research Articles

Data-driven prediction of critical flutter velocity of long-span suspension bridges using a probabilistic machine learning approach

Among the consequences of wind-induced excitation on long-span cable-supported bridges, flutter instability is the most dangerous and can collapse bridge structures. Until now, the flutter velocity of long-span bridges can be computed after conducting an experimental wind tunnel test or by means of computational fluid dynamics (CFD). However, the approaches mentioned above are cost-prohibitive and time-consuming especially when there are many sampling designs to evaluate. Therefore, it is crucial to develop an alternative solution for evaluating the flutter performance of such structures while minimizing the associated cost and computation time without worsening the accuracy of the results. Scholars have proposed machine learning (ML) techniques to address this issue. Unfortunately, some ML techniques do not provide excellent generalization for small datasets and are prone to bias even when the accuracy is relatively high. Besides, conventional ML models do not consider the uncertainty in data. This study proposes a probabilistic machine learning approach to overcome the weaknesses of the conventional ML models. The dataset used to train the proposed model was generated from 73 sampling designs using a fully numerical approach that involved 2D URANS (Unsteady Reynolds-averaged Navier-Stokes) CFD simulations, quasi-steady flutter theory, finite element model, and multi-mode approach for critical flutter velocity computation. The deck cross-section's most influential parameters and the critical flutter velocity constitute the datasets used to build the proposed probabilistic machine learning model. Our finding substantiates that the probabilistic machine learning approach based on hierarchical Bayesian modeling (PML-HBM) overperformed the conventional machine learning models, including extreme gradient boosting regression (XGBR), random forest (RF), and the support vector regression(SVR). Therefore, PML-HBM can accurately predict the critical flutter velocity. This model is an interesting approach to circumvent the time and cost associated with wind tunnel tests and CFD simulations at the earlier design stage of long-span bridge aerodynamic study.

Vaccine hesitancy, parental concerns, and COVID-19 in a digital leisure context: Implications for out-of-school time

Due to the increasing prevalence of parental vaccine hesitancy or refusal, it is important to understand parental motives for vaccine hesitancy. This study examines social media conversations and commentaries regarding concerns about parenting, vaccine hesitancy, and the COVID-19 pandemic within an in-person leisure and recreation context: out-of-school time (OST) programs. A generative, probabilistic Bayesian machine learning model was used to analyze 31,925 tweets and group them into seven categories: Government, Feelings, School, Public Health, Christmas, Risk and Safety, and Families. As a result, recommendations for research and practice are discussed in connection to both OST programs and digital leisure, including a diverse range of vaccine hesitancy motivations related to children and parents, communication management strategies for OST professionals, and the impact of the politization of leisure in a digital leisure context.

Toward a taxonomy of trust for probabilistic machine learning.

Probabilistic machine learning increasingly informs critical decisions in medicine, economics, politics, and beyond. To aid the development of trust in these decisions, we develop a taxonomy delineating where trust in an analysis can break down: (i) in the translation of real-world goals to goals on a particular set of training data, (ii) in the translation of abstract goals on the training data to a concrete mathematical problem, (iii) in the use of an algorithm to solve the stated mathematical problem, and (iv) in the use of a particular code implementation of the chosen algorithm. We detail how trust can fail at each step and illustrate our taxonomy with two case studies. Finally, we describe a wide variety of methods that can be used to increase trust at each step of our taxonomy. The use of our taxonomy highlights not only steps where existing research work on trust tends to concentrate and but also steps where building trust is particularly challenging.

Implementation of Gaussian Process Regression to strain data in residual stress measurements by hole drilling

Incremental hole drilling is a prominent tool in the surface and sub-surface residual stress measurements. Generally, strains are measured at drilling steps not corresponding to the stress calculation steps of the integral method. Consequently, strain measurements are fitted using polynomials or splines. However, this process is greatly influenced by user’s experience, affecting the calculated stresses as an additional source of uncertainty. The present work assesses the uncertainty associated with the strain fitting procedure by exploiting Gaussian Process Regression (GPR), a probabilistic machine learning framework which can yield uncertainties on the fit. These uncertainties were propagated to residual stresses by using Monte Carlo simulation. Laser shock peened AA 7050-T7451 specimens were considered as case study.The proposed methodology was corroborated by comparing residual stress results with those obtained exploiting traditional fitting procedures and X-ray diffraction measurements. Finally, the GPR-based approach demonstrated its ability to successfully discriminate different strain signal sources.

A Machine Learning Approach for PLGA Nanoparticles in Antiviral Drug Delivery

In recent years, nanoparticles have been highly investigated in the laboratory. However, only a few laboratory discoveries have been translated into clinical practice. These findings in the laboratory are limited by trial-and-error methods to determine the optimum formulation for successful drug delivery. A new paradigm is required to ease the translation of lab discoveries to clinical practice. Due to their previous success in antiviral activity, it is vital to accelerate the discovery of novel drugs to treat and manage viruses. Machine learning is a subfield of artificial intelligence and consists of computer algorithms which are improved through experience. It can generate predictions from data inputs via an algorithm which includes a method built from inputs and outputs. Combining nanotherapeutics and well-established machine-learning algorithms can simplify antiviral-drug development systems by automating the analysis. Other relationships in bio-pharmaceutical networks would eventually aid in reaching a complex goal very easily. From previous laboratory experiments, data can be extracted and input into machine learning algorithms to generate predictions. In this study, poly (lactic-co-glycolic acid) (PLGA) nanoparticles were investigated in antiviral drug delivery. Data was extracted from research articles on nanoparticle size, polydispersity index, drug loading capacity and encapsulation efficiency. The Gaussian Process, a form of machine learning algorithm, could be applied to this data to generate graphs with predictions of the datasets. The Gaussian Process is a probabilistic machine learning model which defines a prior over function. The mean and variance of the data can be calculated via matrix multiplications, leading to the formation of prediction graphs-the graphs generated in this study which could be used for the discovery of novel antiviral drugs. The drug load and encapsulation efficiency of a nanoparticle with a specific size can be predicted using these graphs. This could eliminate the trial-and-error discovery method and save laboratory time and ease efficiency.

Advanced ensemble modeling method for space object state prediction accounting for uncertainty in atmospheric density

For objects in the low Earth orbit region, uncertainty in atmospheric density estimation is an important source of orbit prediction error, which is critical for space traffic management activities such as the satellite conjunction analysis. This paper investigates the evolution of orbit error distribution in the presence of atmospheric density uncertainties, which are modeled using probabilistic machine learning techniques. The recently proposed “HASDM-ML,” “CHAMP-ML,” and “MSIS-UQ” machine learning models for density estimation (Licata and Mehta, 2022b; Licata et al., 2022b) are used in this work. The investigation is convoluted because of the spatial and temporal correlation of the atmospheric density values. We develop several Monte Carlo methods, each capturing a different spatiotemporal density correlation, to study the effects of density uncertainty on orbit uncertainty propagation. However, Monte Carlo analysis is computationally expensive, so a faster method based on the Kalman filtering technique for orbit uncertainty propagation is also explored. It is difficult to translate the uncertainty in atmospheric density to the uncertainty in orbital states under a standard extended Kalman filter or unscented Kalman filter framework. This work uses the so-called “consider covariance sigma point (CCSP)” filter that can account for the density uncertainties during orbit propagation. As a test-bed for validation purposes, a comparison between CCSP and Monte Carlo methods of orbit uncertainty propagation is carried out. Finally, using the HASDM-ML, CHAMP-ML, and MSIS-UQ density models, we propose an ensemble approach for orbit uncertainty quantification for four different space weather conditions.

Improving multi-scale attention networks: Bayesian optimization for segmenting medical images

ABSTRACT Current deep learning-based image segmentation methods are notable for their use of large number of parameters and extensive computational resources in training. There is a persistent need for more efficient flexible systems without compromising on precision. This work proposes a novel model that combines the best of deep learning and probabilistic machine learning to segment a wide variety of medical image datasets with state-of-the-art accuracy and limited resources. The approach benefits from the introduction of new diverse attention modules that serve multiple purposes including capturing of relevant information at different scales. These proposed attention modules are generic and can potentially be used with other architectures to boost performance. In addition, Bayesian optimization is employed to tune multi-scale weight hyperparameters of the model. The architecture combined with one of the proposed novel attention modules and tuned hyperparameters achieves the best results in segmenting ISIC 2017, LUNGS, NERVE, Skin Lesion and CHEST datasets. Finally, the explainability of the network is analysed by visualizing the feature map learned from the attention modules.

Environmental sensor placement with convolutional Gaussian neural processes

Abstract Environmental sensors are crucial for monitoring weather conditions and the impacts of climate change. However, it is challenging to place sensors in a way that maximises the informativeness of their measurements, particularly in remote regions like Antarctica. Probabilistic machine learning models can suggest informative sensor placements by finding sites that maximally reduce prediction uncertainty. Gaussian process (GP) models are widely used for this purpose, but they struggle with capturing complex non-stationary behaviour and scaling to large datasets. This paper proposes using a convolutional Gaussian neural process (ConvGNP) to address these issues. A ConvGNP uses neural networks to parameterise a joint Gaussian distribution at arbitrary target locations, enabling flexibility and scalability. Using simulated surface air temperature anomaly over Antarctica as training data, the ConvGNP learns spatial and seasonal non-stationarities, outperforming a non-stationary GP baseline. In a simulated sensor placement experiment, the ConvGNP better predicts the performance boost obtained from new observations than GP baselines, leading to more informative sensor placements. We contrast our approach with physics-based sensor placement methods and propose future steps towards an operational sensor placement recommendation system. Our work could help to realise environmental digital twins that actively direct measurement sampling to improve the digital representation of reality.

Optical network physical layer parameter optimization for digital backpropagation using Gaussian processes

Abstract We present a novel methodology for optimizing fiber optic network performance by determining the ideal values for attenuation, nonlinearity, and dispersion parameters in terms of achieved signal-to-noise ratio (SNR) gain from digital backpropagation (DBP). Our approach uses Gaussian process regression, a probabilistic machine learning technique, to create a computationally efficient model for mapping these parameters to the resulting SNR after applying DBP. We then use simplicial homology global optimization to find the parameter values that yield maximum SNR for the Gaussian process model within a set of a priori bounds. This approach optimizes the parameters in terms of the DBP gain at the receiver. We demonstrate the effectiveness of our method through simulation and experimental testing, achieving optimal estimates of the dispersion, nonlinearity, and attenuation parameters. Our approach also highlights the limitations of traditional one-at-a-time grid search methods and emphasizes the interpretability of the technique. This methodology has broad applications in engineering and can be used to optimize performance in various systems beyond optical networks.

A machine learning route between band mapping and band structure

The electronic band structure and crystal structure are the two complementary identifiers of solid-state materials. Although convenient instruments and reconstruction algorithms have made large, empirical, crystal structure databases possible, extracting the quasiparticle dispersion (closely related to band structure) from photoemission band mapping data is currently limited by the available computational methods. To cope with the growing size and scale of photoemission data, here we develop a pipeline including probabilistic machine learning and the associated data processing, optimization and evaluation methods for band-structure reconstruction, leveraging theoretical calculations. The pipeline reconstructs all 14 valence bands of a semiconductor and shows excellent performance on benchmarks and other materials datasets. The reconstruction uncovers previously inaccessible momentum-space structural information on both global and local scales, while realizing a path towards integration with materials science databases. Our approach illustrates the potential of combining machine learning and domain knowledge for scalable feature extraction in multidimensional data.

Investigating the Effect of Temperature History on Crystal Morphology of Thermoplastic Composites Using In Situ Polarized Light Microscopy and Probabilistic Machine Learning.

Processing parameters including temperature history affect the morphology of semicrystalline thermoplastic composites, and in turn their performance. In addition, the competition between spherulite growth in resin-rich areas, and transcrystallinity growth from fiber surfaces, determines the final morphology. In this study, growth of crystals in low volume fraction PEEK-carbon fiber composites was studied in situ, using a polarized microscope equipped with a heating and cooling controlled stage and a probabilistic machine learning approach, Gaussian Process Regression (GPR). GPR showed that for spherulites, growth kinetics follows the established Lauritzen-Hoffman equation, while transcrystallinity growth deviates from the theory. Combined GPR model and Lauritzen-Hoffman equation were used to deconvolute the underlying competition between diffusion and secondary nucleation at growth front of spherulites and transcrystalline regions.

A probabilistic framework for product health monitoring in multistage manufacturing using Unsupervised Artificial Neural Networks and Gaussian Processes

The emergence of highly instrumented manufacturing systems has enabled the paradigm of smart manufacturing that provides high levels of prognostics functionality. Of particular interest is to precisely determine geometric conformance or non-conformance of workpieces during manufacturing. This paper presents a new dimensional product health monitoring system that learns from in-process sensor data and updates the prediction of the product quality as the product is manufactured. The system uses data from multiple manufacturing stages, unlike from a single stage at a time, to predict the dimensional quality of the finished product that is updated with subsequent measurements such as On-Machine Measurements (OMMs), in on-line incremental learning fashion. It is based on self-supervised neural networks for dimensionality reduction, Gaussian Process Regression (GPR) models for probabilistic prediction about the end product condition and the associated uncertainty, and Bayesian information fusion for updating the conditional probability distribution of the end product quality in the light of new information. The monitoring approach was tested on the prediction of diameter deviations with validation results showing its ability to achieve an average accuracy better than 5 μm in terms of the Root Mean Squared Error (RMSE). Having obtained a Probability Density Function (PDF) for the measurand of interest, the conformance and non-conformance probabilities given the tolerance specifications are computed to support the principle of inspection by exception. This ability to construct a conformance probability-based product quality monitoring system using probabilistic machine learning methods constitute a step change to manufacturing prognostics.

Short-term rainfall forecasting using cumulative precipitation fields from station data: a probabilistic machine learning approach

Rainfall nowcasting supports emergency decision-making in hydrological, agricultural, and economical sectors. However, short-term prediction is challenging because meteorological variables are strongly interconnected and rapidly change during an event. Since machine learning do not require any previous physical assumption, this study investigates their ability to provide reliable and quick forecasts. This paper proposes a machine learning model for probabilistic rainfall nowcasting at 10 min intervals for short lead times - from 30 min up to 6 h. The model employs cumulative rainfall fields from station data as inputs for a feed forward neural network to predict rainfall interval and the corresponding probability of occurrence. Cumulative rainfall depths from station data were used to overcome the lack of temporal memory of the feed forward neural networks. In this way, using only the current rain field as input, the model exploited pattern recognition techniques combining both temporal - cumulative rainfall depth - and spatial - cumulative rainfall field – information. Based on 359 rain events observed in an area of 1619 km2 in Southern Italy, 95 machine learning models were independently trained for 19 recording stations and for each target lead-time (30, 60, 120, 180 and 360 min). Comprehensive nowcasts verification was performed to analyse the reliability of probabilistic nowcasts using both continuous (RMSE and RAE) and categorical (POD, FAR and CSI) indicators. The performance of the models was also compared with the results of Eulerian Persistence. All the models produced consistent nowcasts and learnt the complex relationship describing space–time rainfall evolution. As expected, predictive accuracy gradually decreased as the lead-time increase, according to physically based models. Results showed that the use of both temporal and spatial information enables the model to predict short-term rainfall using only the current measurements as input, resulting in a rapid, easily replicable and convenient nowcasting approach. The procedure is an effective way to predict multi-step rainfall fields and is suitable for operational early warning system.

Chronic stress causes striatal disinhibition mediated by SOM-interneurons in male mice

Chronic stress (CS) is associated with a number of neuropsychiatric disorders, and it may also contribute to or exacerbate motor function. However, the mechanisms by which stress triggers motor symptoms are not fully understood. Here, we report that CS functionally alters dorsomedial striatum (DMS) circuits in male mice, by affecting GABAergic interneuron populations and somatostatin positive (SOM) interneurons in particular. Specifically, we show that CS impairs communication between SOM interneurons and medium spiny neurons, promoting striatal overactivation/disinhibition and increased motor output. Using probabilistic machine learning to analyze animal behavior, we demonstrate that in vivo chemogenetic manipulation of SOM interneurons in DMS modulates motor phenotypes in stressed mice. Altogether, we propose a causal link between dysfunction of striatal SOM interneurons and motor symptoms in models of chronic stress.

Non-linear probabilistic calibration of low-cost environmental air pollution sensor networks for neighborhood level spatiotemporal exposure assessment.

Low-cost sensor networks for monitoring air pollution are an effective tool for expanding spatial resolution beyond the capabilities of existing state and federal reference monitoring stations. However, low-cost sensor data commonly exhibit non-linear biases with respect to environmental conditions that cannot be captured by linear models, therefore requiring extensive lab calibration. Further, these calibration models traditionally produce point estimates or uniform variance predictions which limits their downstream in exposure assessment. Build direct field-calibration models using probabilistic gradient boosted decision trees (GBDT) that eliminate the need for resource-intensive lab calibration and that can be used to conduct probabilistic exposure assessments on the neighborhood level. Using data from Plantower A003 particulate matter (PM) sensors deployed in Baltimore, MD from November 2018 through November 2019, a fully probabilistic NGBoost GBDT was trained on raw data from sensors co-located with a federal reference monitoring station and compared against linear regression trained on lab calibrated sensor data. The NGBoost predictions were then used in a Monte Carlo interpolation process to generate high spatial resolution probabilistic exposure gradients across Baltimore. We demonstrate that direct field-calibration of the raw PM2.5 sensor data using a probabilistic GBDT has improved point and distribution accuracies compared to the linear model, particularly at reference measurements exceeding 25 μg/m3, and also on monitors not included in the training set. We provide a framework for utilizing the GBDT to conduct probabilistic spatial assessments of human exposure with inverse distance weighting that predicts the probability of a given location exceeding an exposure threshold and provides percentiles of exposure. These probabilistic spatial exposure assessments can be scaled by time and space with minimal modifications. Here, we used the probabilistic exposure assessment methodology to create high quality spatial-temporal PM2.5 maps on the neighborhood-scale in Baltimore, MD. We demonstrate how the use of open-source probabilistic machine learning models for in-place sensor calibration outperforms traditional linear models and does not require an initial laboratory calibration step. Further, these probabilistic models can create uniquely probabilistic spatial exposure assessments following a Monte Carlo interpolation process.

Model Uncertainty and Correctability for Directed Graphical Models

Probabilistic graphical models are a fundamental tool in probabilistic modeling, machine learning and artificial intelligence. They allow us to integrate in a natural way expert knowledge, physical modeling, heterogeneous and correlated data and quantities of interest. For exactly this reason, multiple sources of model uncertainty are inherent within the modular structure of the graphical model. In this paper we develop information-theoretic, robust uncertainty quantification methods and non-parametric stress tests for directed graphical models to assess the effect and the propagation through the graph of multi-sourced model uncertainties to quantities of interest. These methods allow us to rank the different sources of uncertainty and correct the graphical model by targeting its most impactful components with respect to the quantities of interest. Thus, from a machine learning perspective, we provide a mathematically rigorous approach to correctability that guarantees a systematic selection for improvement of components of a graphical model while controlling potential new errors created in the process in other parts of the model. We demonstrate our methods in two physico-chemical examples, namely quantum scale-informed chemical kinetics and materials screening to improve the efficiency of fuel cells.

Numerical Investigation of a Class of Nonlinear Time-Dependent Delay PDEs Based on Gaussian Process Regression

Probabilistic machine learning and data-driven methods gradually show their high efficiency in solving the forward and inverse problems of partial differential equations (PDEs). This paper will focus on investigating the forward problem of solving time-dependent nonlinear delay PDEs with multi-delays based on multi-prior numerical Gaussian processes (MP-NGPs), which are constructed by us to solve complex PDEs that may involve fractional operators, multi-delays and different types of boundary conditions. We also quantify the uncertainty of the prediction solution by the posterior distribution of the predicted solution. The core of MP-NGPs is to discretize time firstly, then a Gaussian process regression based on multi-priors is considered at each time step to obtain the solution of the next time step, and this procedure is repeated until the last time step. Different types of boundary conditions are studied in this paper, which include Dirichlet, Neumann and mixed boundary conditions. Several numerical tests are provided to show that the methods considered in this paper work well in solving nonlinear time-dependent PDEs with delay, where delay partial differential equations, delay partial integro-differential equations and delay fractional partial differential equations are considered. Furthermore, in order to improve the accuracy of the algorithm, we construct Runge–Kutta methods under the frame of multi-prior numerical Gaussian processes. The results of the numerical experiments prove that the prediction accuracy of the algorithm is obviously improved when the Runge–Kutta methods are employed.

Probabilistic physics-informed machine learning for dynamic systems

This paper develops a physics-informed machine learning approach for response prediction in dynamic systems, by augmenting a physics-based model with a machine learning model for model error; both models are probabilistic. The physics-based dynamic system model has discrepancy in the predicted output (due to incomplete physics or model form error in the physics-based model) when compared against observation data. The model form error is quantified using Bayesian state estimation with observation data, and a probabilistic machine learning model is trained to predict the output discrepancy for untrained inputs. Two different computational options are developed to implement this overall approach, with different levels of sophistication and computational effort. The methods are developed for problems with stationary and non-stationary, and Gaussian and non-Gaussian random process inputs. Strategies for computational effort reduction at different stages, such as model training, model error quantification, and reliability prediction, are discussed. The proposed approach is demonstrated using two numerical illustrations: (a) a deep beam subjected to dynamic loading (a single physics system), and (b) hypersonic flow over a flexible aircraft panel (a multi-physics system).

Machine Learning for Risk Prediction of Oesophago-Gastric Cancer in Primary Care: Comparison with Existing Risk-Assessment Tools.

Simple SummaryOesophago-gastric cancer is one of the commonest cancers worldwide, yet it can be particularly difficult to diagnose given that initial symptoms are often non-specific and routine screening is not available. Cancer risk-assessment tools, which calculate cancer risk based on symptoms and other risk factors present in the primary care record, can aid decisions on referrals for cancer investigations, facilitating earlier diagnosis. Diagnosing common cancers earlier could help improve survival rates. Using UK primary care electronic health record data, we compared five different machine learning techniques for probabilistic classification of cancer patients against a current widely used UK primary care cancer risk-assessment tool. The machine learning algorithms outperformed the current risk-assessment tool, with a higher overall accuracy and an ability to reasonably identify 11–25% more cancer patients. We conclude that machine-learning-based risk-assessment tools could help better identify suitable patients for further investigation and support earlier diagnosis.Oesophago-gastric cancer is difficult to diagnose in the early stages given its typical non-specific initial manifestation. We hypothesise that machine learning can improve upon the diagnostic performance of current primary care risk-assessment tools by using advanced analytical techniques to exploit the wealth of evidence available in the electronic health record. We used a primary care electronic health record dataset derived from the UK General Practice Research Database (7471 cases; 32,877 controls) and developed five probabilistic machine learning classifiers: Support Vector Machine, Random Forest, Logistic Regression, Naïve Bayes, and Extreme Gradient Boosted Decision Trees. Features included basic demographics, symptoms, and lab test results. The Logistic Regression, Support Vector Machine, and Extreme Gradient Boosted Decision Tree models achieved the highest performance in terms of accuracy and AUROC (0.89 accuracy, 0.87 AUROC), outperforming a current UK oesophago-gastric cancer risk-assessment tool (ogRAT). Machine learning also identified more cancer patients than the ogRAT: 11.0% more with little to no effect on false positives, or up to 25.0% more with a slight increase in false positives (for Logistic Regression, results threshold-dependent). Feature contribution estimates and individual prediction explanations indicated clinical relevance. We conclude that machine learning could improve primary care cancer risk-assessment tools, potentially helping clinicians to identify additional cancer cases earlier. This could, in turn, improve survival outcomes.

Automated geological model updates during TBM operation – An approach based on probabilistic machine learning concepts

AbstractGeological models are commonly used to predict the position of relevant geological features, such as rock types or faults in the subsurface. These models can contain significant uncertainties, as the geological input parameters are often not perfectly known. Predictions of geological features, for example, on the level of a tunnel during an excavation process, are therefore uncertain. This work shows how these uncertainties can be estimated using probabilistic concepts. Furthermore, an approach is presented to automatically adjust the geological model predictions using measurements of the tunnel boring machine (TBM) operation. To this aim, the geological forward model is combined with a measurement model and both are integrated in a probabilistic machine learning framework. This integration enables a Bayesian inference process using computational methods, enabling an update of the parameters of the geological and measurement models. Based on the inferred parameter distributions, the model predictions on the tunnel level are subsequently updated. The application of the concept in a simple schematic application shows that such a combination can accurately and precisely predict features ahead of the operation within the limits of the model capabilities. In future work, the methods need to be tested with a real case study to evaluate the accuracy of predictions using real‐world TBM data and more complex geological models. The framework presented here could directly be extended to this purpose.