Probabilistic Machine Learning Methods Research Articles

Data-driven prediction of natural period for existing RC high-rise buildings using probabilistic machine learning methods

The natural period of existing reinforced concrete (RC) high-rise buildings is a fundamental structural parameter for dynamic analysis and condition assessment, while the estimation of such structural characteristic is an important task for life-cycle maintenance. A commonly adopted approach for predicting natural periods involves the use of design-oriented empirical formulas. However, as shown by real-world observations in both this study and others, these formulas with limited building features may not fully capture the complexity and uncertainty of the natural period for existing buildings. Therefore, it is desired to develop an alternative method that is both feature-based and data-driven, to predict natural periods accurately. This study proposes a feature-based probabilistic machine learning method to determine the natural periods and quantify the uncertainty intervals of the predictions for existing RC high-rise buildings. Specifically, the proposed method comprises a random-forest method using quantile regression. Through in situ measurements and an extensive literature review, a database comprising over 500 actual records was constructed, including the building features and their dynamic properties. The results show that the proposed method achieves a coefficient of determination of 0.96 and an uncertainty interval confined within 20 % for the test set. These findings indicate that the proposed method may be both more accurate and reliable. Furthermore, an in-depth investigation into the model interpretability was conducted to reveal the relative importance of different building features. The analyses indicate that the building height emerges as the most crucial feature, while the building width and its lateral resistant system may also demonstrate noticeable influence on predicting natural periods.

Prediction and correlations estimation of seismic capacities of pier columns: Extended Gaussian process regression models

Assessing the seismic capacity of pier columns is a crucial element in the performance-based seismic design of bridges. Such assessment necessitates a probabilistic approach to accurately determine the marginal probability distributions of seismic capacities and to characterize the dependencies among these variables. In response to this need, this paper employs Multi-Output Gaussian Process Regression (MO-GPR), a probabilistic machine learning method, to jointly model the multiple seismic capacities of pier columns. We initially introduce a probabilistic seismic capacity model that utilizes MO-GPR for pier columns and validate its predictive accuracy in comparison to Bayesian linear regression and existing empirical methods. Subsequently, the methodology is augmented by the integration of hierarchical modeling within the MO-GPR framework, resulting in a Multi-Output Hierarchical Gaussian Process Regression (MO-HGPR) model that effectively estimates intraclass correlation coefficients for specific types of datasets. It is postulated that these correlation coefficients also reflect correlations associated with multiple components of the real structure. This study employs MO-HGPR and MO-GPR separately to investigate the potential correlations of seismic capacities among pier columns and distinct limit states. The results demonstrate that the MO-GPR model exhibits superior prediction accuracy and more effectively portrays uncertainty compared to existing empirical models. Moreover, the correlations of seismic capacities among piers and limit states are both robust and significantly impact the seismic fragility of bridges. This finding highlights the essential nature of considering capacities correlations in seismic fragility or risk assessment processes.

Automated experimental design of safe rampdowns via probabilistic machine learning

Typically the rampdown phase of a shot consists of a decrease in current and injected power and optionally a change in shape, but there is considerable flexibility in the rate, sequencing, and duration of these changes. On the next generation of tokamaks it is essential that this is done safely as the device could be damaged by the stored thermal and electromagnetic energy present in the plasma. This works presents a procedure for automatically choosing experimental rampdown designs to rapidly converge to an effective rampdown trajectory. This procedure uses probabilistic machine learning methods paired with acquisition functions taken from Bayesian optimization. In a set of 2022 experiments at DIII-D, the rampdown designs produced by our method maintained plasma control down to substantially lower current and energy levels than are typically observed. The actions predicted by the model significantly improved as the model was able to explore over the course of the experimental campaign.

Using Probabilistic Machine Learning Methods to Improve Beef Cattle Price Modeling and Promote Beef Production Efficiency and Sustainability in Canada

Accurate agricultural commodity price models enable efficient allocation of limited natural resources, leading to improved sustainability in agriculture. Because of climate change, price volatility and uncertainty in the sector are expected to increase in the future, increasing the need for improved price modeling. With the emergence of machine learning (ML) algorithms, novel tools are now available to enhance the modeling of agricultural commodity prices. This research explores both univariate and multivariate ML techniques to perform probabilistic price prediction modeling for the Canadian beef industry, taking into account beef production, commodity markets, and international trade features to enhance accuracy. We model Alberta fed steer prices using three multivariate ML algorithms (support vector regression (SVR), random forest (RF), and Adaboost (AB)) and three univariate ML algorithms (autoregressive integrated moving average (ARIMA), seasonal ARIMA (SARIMA), and the seasonal autoregressive integrated moving average with exogenous factors (SARIMAX)). We apply these models to monthly fed steer price data between January 2005 and September 2023 and compare predicted prices with observed prices using several validation metrics. The outcomes indicate that both random forest (RF) and Adaboost (AB) show superior overall performance in accurately predicting Alberta fed steer prices in comparison to other algorithms. To better account for the variance of the best model performance, we subsequently adopted a probabilistic approach by considering uncertainty in our best-selected ML model. The beef industry can use these improved price models to minimize resource waste and inefficiency in the sector and improve the long-term sustainability prospects for beef producers in Canada.

Mapping the global free expression landscape using machine learning

Freedom of expression is a core human right, yet the forces that seek to suppress it have intensified, increasing the need to develop tools that can measure the rates of freedom globally. In this study, we propose a novel freedom of expression index to gain a nuanced and data-led understanding of the level of censorship across the globe. For this, we used an unsupervised, probabilistic machine learning method, to model the status of the free expression landscape. This index seeks to provide legislators and other policymakers, activists and governments, and non-governmental and intergovernmental organisations, with tools to better inform policy or action decisions. The global nature of the proposed index also means it can become a vital resource/tool for engagement with international and supranational bodies.

Efficient seismic fragility analysis method utilizing ground motion clustering and probabilistic machine learning

Machine learning (ML) techniques have been recently adopted in engineering practice to define the relationship between seismic intensity measure (IM) and structural damage measure (DM) based on a limited set of numerical simulations. However, they only offer deterministic prediction, which failing to reflect the aleatoric uncertainty related to input variables (e.g. seismic excitation and structural properties) and the epistemic uncertainty associated with modeling. This paper proposes a probabilistic ML method combined with ground motion clustering for seismic fragility analysis of structures. In the probabilistic ML method, by the natural gradient boosting (NGBoost), the conditional probability distribution can be evaluated for each structural response instead of producing point estimation. In addition, ground motion clustering is based on the time series K-means, which can capture the hidden features and select the representative subset of ground motions. The proposed framework was implemented for seismic fragility analysis of a typical 3-span, 6-storey reinforced concrete (RC) frame system. Analysis results indicated that the point estimation accuracy of the NGBoost was comparable to that based on excellent deterministic ML techniques, e.g. artificial neural network (ANN), random forest (RF), extreme gradient boosting (XGBoost). Moreover, the probabilistic prediction can efficiently provide the conditional probability of exceeding a damaged state in the structure given an IM level, eliminating the need for additional input of uncertainties from structural properties in traditional ML methods. Ultimately, the cluster-based ground motion selection reduced the model uncertainty and improves the prediction accuracy of the probabilistic ML model.

A probabilistic framework for product health monitoring in multistage manufacturing using Unsupervised Artificial Neural Networks and Gaussian Processes

The emergence of highly instrumented manufacturing systems has enabled the paradigm of smart manufacturing that provides high levels of prognostics functionality. Of particular interest is to precisely determine geometric conformance or non-conformance of workpieces during manufacturing. This paper presents a new dimensional product health monitoring system that learns from in-process sensor data and updates the prediction of the product quality as the product is manufactured. The system uses data from multiple manufacturing stages, unlike from a single stage at a time, to predict the dimensional quality of the finished product that is updated with subsequent measurements such as On-Machine Measurements (OMMs), in on-line incremental learning fashion. It is based on self-supervised neural networks for dimensionality reduction, Gaussian Process Regression (GPR) models for probabilistic prediction about the end product condition and the associated uncertainty, and Bayesian information fusion for updating the conditional probability distribution of the end product quality in the light of new information. The monitoring approach was tested on the prediction of diameter deviations with validation results showing its ability to achieve an average accuracy better than 5 μm in terms of the Root Mean Squared Error (RMSE). Having obtained a Probability Density Function (PDF) for the measurand of interest, the conformance and non-conformance probabilities given the tolerance specifications are computed to support the principle of inspection by exception. This ability to construct a conformance probability-based product quality monitoring system using probabilistic machine learning methods constitute a step change to manufacturing prognostics.

Impedance-based forecasting of lithium-ion battery performance amid uneven usage

Accurate forecasting of lithium-ion battery performance is essential for easing consumer concerns about the safety and reliability of electric vehicles. Most research on battery health prognostics focuses on the research and development setting where cells are subjected to the same usage patterns. However, in practical operation, there is great variability in use across cells and cycles, thus making forecasting challenging. To address this challenge, here we propose a combination of electrochemical impedance spectroscopy measurements with probabilistic machine learning methods. Making use of a dataset of 88 commercial lithium-ion coin cells generated via multistage charging and discharging (with currents randomly changed between cycles), we show that future discharge capacities can be predicted with calibrated uncertainties, given the future cycling protocol and a single electrochemical impedance spectroscopy measurement made immediately before charging, and without any knowledge of usage history. The results are robust to cell manufacturer, the distribution of cycling protocols, and temperature. The research outcome also suggests that battery health is better quantified by a multidimensional vector rather than a scalar state of health.

Probabilistic machine learning based soft-sensors for product quality prediction in batch processes

The major promise of the 4th industrial revolution is encompassed by the utilization of real-time process data to inform operational decision-making. Research focus relating to the monitoring of batch process end product qualities has typically been dominated by the use of latent variable models. In this work, we combined latent variable modeling with probabilistic machine learning methods to construct novel soft-sensors, which are able to synchronously capture nonlinearities expressed within process data, as well as identify accurate uncertainty estimates for predictions. Specifically, we explored the combination of multiway projection to latent structures (MPLS) with Gaussian processes (GPs), Bayesian neural networks (BNNs) and heteroscedastic noise neural networks (HNNs). We demonstrated performance of the soft-sensors for industrial batch process quality control, which involves estimating the end viscosity of different variants of a non-newtonian liquid product manufactured over different periods. Through experimental validation, it is concluded that the use of the MPLS-HNN soft-sensor provides particular promise for industrial batch process monitoring given its high accuracy and reliability, as well as ease for practical implementation.

Structural health monitoring by a novel probabilistic machine learning method based on extreme value theory and mixture quantile modeling

This article proposes a novel probabilistic machine learning method based on unsupervised novelty detection for health monitoring of civil structures. The core of this method is based on extreme value theory (EVT) and mixture quantile modeling. Accordingly, a mixture quantile value by combining non-parametric and parametric quantile estimators is proposed as a new decision-making or novelty score. The non-parametric estimator relies on an empirical quantile function, while the parametric estimator stems from modeling a generalized extreme value distribution. Generally, the proposed method is composed of some main steps; that is, calculating distances between feature samples, sorting the distances in ascending order, changing their signs for providing negated quantities, selecting some negated maximum distances or extreme values in an iterative algorithm by using a goodness-of-fit test based on Kullback-Leibler information, and computing a mixture quantile. Furthermore, an EVT-based approach is proposed to estimate an alarming threshold. The major innovation in this article is to develop a novel probabilistic novelty detector with a new score for decision-making. The advantages of the proposed method contain preparing discriminative novelty scores for damage detection, dealing with the major challenge of environmental and/or operational variability, estimating a reliable threshold, and implementing both the procedures of decision-making and threshold estimation within a single framework. Dynamic and statistical features of two full-scale bridge structures are utilized to validate the proposed method along with comparative studies. Results demonstrate that the method is a reliable and influential tool for health monitoring of civil structures under varying various environmental and/or operational conditions.

Probabilistic machine learning for detection of tightening torque in bolted joints

Observing the loss of tightening torque using modal parameters is challenging due to the variability and nonlinear effects in bolted joints. Thus, this paper proposes a combined application of two probabilistic machine learning methods. First, a Gaussian mixture model (GMM) is learned using estimated natural frequencies, assuming the tightening torque in a safe situation. This probabilistic model can assuredly detect the lack of torque using indirect vibration measures in other unknown states by computing a damage index. A Gaussian process regression (GPR) is also learned considering a set of torque and damage index pairs in several conditions. The GPR model interpolates a curve to supply an estimative of the tightening torque for other conditions not used in this learning. An illustrative application is performed considering the Orion beam, an academic-scale specimen composed of a lap-joint configuration that retains the friction surface in contact patches. The structure is subjected to a random vibration with a controlled RMS level and several tightening torque conditions to identify the modal parameters. The probabilistic model learning via the GMM and GPR can detect adequately, with a low number of false diagnoses, the actual state of torque using an indirect measure of vibration, that is, without the need for a torque sensor on each bolt.

A backward SDE method for uncertainty quantification in deep learning

<p style='text-indent:20px;'>We develop a backward stochastic differential equation based probabilistic machine learning method, which formulates a class of stochastic neural networks as a stochastic optimal control problem. An efficient stochastic gradient descent algorithm is introduced with the gradient computed through a backward stochastic differential equation. Convergence analysis for stochastic gradient descent optimization and numerical experiments for applications of stochastic neural networks are carried out to validate our methodology in both theory and performance.</p>

Comparing cell population balance model simulation through Gaussian processes and discretisation

Phenotypic variability in cell populations can be described through population balance models (PBM). In this paper, the simulation of cell PBMs via two different methods is compared. We compare the cell average technique (CAT) with the recently introduced numerical Gaussian processes (GPs). While the CAT discretises the PBM in the individual variables, i.e. in the cellular properties, numerical GPs circumvent this discretisation. Numerical GPs are a probabilistic machine-learning method that additionally provides uncertainty quantification. Using two simulation case studies, we compare both methods with regards to computational load, accuracy, and implementation effort. In one of the two considered case studies, numerical GPs are inferior to the CAT. Nonetheless, they are an attractive option when uncertainty quantification is very much required, such as in a subsequent state estimation for PBMs.

Probabilistic Forecasting of Wind Turbine Icing Related Production Losses Using Quantile Regression Forests

A probabilistic machine learning method is applied to icing related production loss forecasts for wind energy in cold climates. The employed method, called quantile regression forests, is based on the random forest regression algorithm. Based on the performed tests on data from four Swedish wind parks available for two winter seasons, it has been shown to produce valuable probabilistic forecasts. Even with the limited amount of training and test data that were used in the study, the estimated forecast uncertainty adds more value to the forecast when compared to a deterministic forecast and a previously published probabilistic forecast method. It is also shown that the output from a physical icing model provides useful information to the machine learning method, as its usage results in an increased forecast skill when compared to only using Numerical Weather Prediction data. A potential additional benefit in machine learning for some stations was also found when using information in the training from other stations that are also affected by icing. This increases the amount of data, which is otherwise a challenge when developing forecasting methods for wind energy in cold climates.

The role of phosphorus and nitrogen on chlorophyll a: Evidence from hundreds of lakes

The effect of nutrients on phytoplankton biomass in lakes continues to be a subject of debate by aquatic scientists. However, determining whether or not chlorophyll a (CHL) is limited by phosphorus (P) and/or nitrogen (N) is rarely considered using a probabilistic method in studies of hundreds of lakes across broad spatial extents. Several studies have applied a unified CHL-nutrient relationship to determine nutrient limitation, but pose a risk of ecological fallacy because they neglect spatial heterogeneity in ecological contexts. To examine whether or not CHL is limited by P, N, or both nutrients in hundreds of lakes and across diverse ecological settings, a probabilistic machine learning method, Bayesian Network, was applied. Spatial heterogeneity in ecological context was accommodated by the probabilistic nature of the results. We analyzed data from 1382 lakes in 17 US states to evaluate the cause-effect relationships between CHL and nutrients. Observations of CHL, total phosphorus (TP), and total nitrogen (TN) were discretized into three trophic states (oligo-mesotrophic, eutrophic, and hypereutrophic) to train the model. We found that although both nutrients were related to CHL trophic state, TP was more related to CHL than TN, especially under oligo-mesotrophic and eutrophic CHL conditions. However, when the CHL trophic state was hypereutrophic, both TP and TN were important. These results provide additional evidence that P-limitation is more likely under oligo-mesotrophic or eutrophic CHL conditions and that co-limitation of P and N occurs under hypereutrophic CHL conditions. We also found a decreasing pattern of the TN/TP ratio with increasing CHL concentrations, which might be a key driver for the role change of nutrients. Previous work performed at smaller scales support our findings, indicating potential for extension of our findings to other regions. Our findings enhance the understanding of nutrient limitation at macroscales and revealed that the current debate on the limiting nutrient might be caused by failure to consider CHL trophic state. Our findings also provide prior information for the site-specific eutrophication management of unsampled or data-limited lakes.

Learning Coarse-Grained Potentials for Binary Fluids.

For a multiple-fluid system, CG models capable of accurately predicting the interfacial properties as a function of curvature are still lacking. In this work, we propose a new probabilistic machine learning (ML) model for learning CG potentials for binary fluids. The water-hexane mixture is selected as a typical immiscible binary liquid-liquid system. We develop a new CG force field (FF) using the Shinoda-DeVane-Klein (SDK) FF framework and compute parameters in this CG FF using the proposed probabilistic ML method. It is shown that a standard response-surface approach does not provide a unique set of parameters, as it results in a loss function with multiple shallow minima. To address this challenge, we develop a probabilistic ML approach where we compute the probability density function (PDF) of parameters that minimize the loss function. The PDF has a well-defined peak corresponding to a unique set of parameters in the CG FF that reproduces the desired properties of a liquid-liquid interface. We compare the performance of the new CG FF with several existing FFs for the water-hexane mixture, including two atomistic and three CG FFs with respect to modeling the interface structure and thermodynamic properties. It is demonstrated that the new FF significantly improves the CG model prediction of both the interfacial tension and structure for the water-hexane mixture.

Retrieval of aboveground crop nitrogen content with a hybrid machine learning method

Hyperspectral acquisitions have proven to be the most informative Earth observation data source for the estimation of nitrogen (N) content, which is the main limiting nutrient for plant growth and thus agricultural production. In the past, empirical algorithms have been widely employed to retrieve information on this biochemical plant component from canopy reflectance. However, these approaches do not seek for a cause-effect relationship based on physical laws. Moreover, most studies solely relied on the correlation of chlorophyll content with nitrogen, and thus neglected the fact that most N is bound in proteins. Our study presents a hybrid retrieval method using a physically-based approach combined with machine learning regression to estimate crop N content. Within the workflow, the leaf optical properties model PROSPECT-PRO including the newly calibrated specific absorption coefficients (SAC) of proteins, was coupled with the canopy reflectance model 4SAIL to PROSAIL-PRO. The latter was then employed to generate a training database to be used for advanced probabilistic machine learning methods: a standard homoscedastic Gaussian process (GP) and a heteroscedastic GP regression that accounts for signal-to-noise relations. Both GP models have the property of providing confidence intervals for the estimates, which sets them apart from other machine learners. Moreover, a GP-based sequential backward band removal algorithm was employed to analyze the band-specific information content of PROSAIL-PRO simulated spectra for the estimation of aboveground N. Data from multiple hyperspectral field campaigns, carried out in the framework of the future satellite mission Environmental Mapping and Analysis Program (EnMAP), were exploited for validation. In these campaigns, corn and winter wheat spectra were acquired to simulate spectral EnMAP data. Moreover, destructive N measurements from leaves, stalks and fruits were collected separately to enable plant-organ-specific validation. The results showed that both GP models can provide accurate aboveground N simulations, with slightly better results of the heteroscedastic GP in terms of model testing and against in situ N measurements from leaves plus stalks, with root mean square error (RMSE) of 2.1 g/m². However, the inclusion of fruit N content for validation deteriorated the results, which can be explained by the inability of the radiation to penetrate the thick tissues of stalks, corn cobs and wheat ears. GP-based band analysis identified optimal spectral settings with ten bands mainly situated in the shortwave infrared (SWIR) spectral region. Use of well-known protein absorption bands from the literature showed comparative results. Finally, the heteroscedastic GP model was successfully applied on airborne hyperspectral data for N mapping. We conclude that GP algorithms, and in particular the heteroscedastic GP, should be implemented for global agricultural monitoring of aboveground N from future imaging spectroscopy data.

Mutual Information Input Selector and Probabilistic Machine Learning Utilisation for Air Pollution Proxies

An air pollutant proxy is a mathematical model that estimates an unobserved air pollutant using other measured variables. The proxy is advantageous to fill missing data in a research campaign or to substitute a real measurement for minimising the cost as well as the operators involved (i.e., virtual sensor). In this paper, we present a generic concept of pollutant proxy development based on an optimised data-driven approach. We propose a mutual information concept to determine the interdependence of different variables and thus select the most correlated inputs. The most relevant variables are selected to be the best proxy inputs, where several metrics and data loss are also involved for guidance. The input selection method determines the used data for training pollutant proxies based on a probabilistic machine learning method. In particular, we use a Bayesian neural network that naturally prevents overfitting and provides confidence intervals around its output prediction. In this way, the prediction uncertainty could be assessed and evaluated. In order to demonstrate the effectiveness of our approach, we test it on an extensive air pollution database to estimate ozone concentration.

Comparing unsupervised probabilistic machine learning methods for market basket analysis

We compare several unsupervised probabilistic machine learning methods for market basket analysis, namely binary factor analysis, two topic models (latent Dirichlet allocation and the correlated topic model), the restricted Boltzmann machine and the deep belief net. After an overview of previous applications of unsupervised probabilistic machine learning methods to market basket analysis we shortly present the methods which we investigate and outline their estimation. Performance is measured by tenfold cross-validated log likelihood values. Binary factor analysis vastly outperforms topic models. The restricted Boltzmann machine attains a similar performance advantage over binary factor analysis. Overall, a deep belief net with 45 variables in the first and 15 variables in the second hidden layers turns out to be the best model. We also compare the investigated machine learning methods with respect to ease of interpretation and runtimes. In addition, we show how to interpret the relationships between hidden variables and observed category purchases. To demonstrate managerial implications we estimate the effect of promoting each category both on purchase probability increases of other product categories and the relative increase of basket size. Finally, we indicate several possibilities to extend restricted Boltzmann machines and deep belief nets for market basket analysis.

A non-homogeneous dynamic Bayesian network with a hidden Markov model dependency structure among the temporal data points

In the topical field of systems biology there is considerable interest in learning regulatory networks, and various probabilistic machine learning methods have been proposed to this end. Popular approaches include non-homogeneous dynamic Bayesian networks (DBNs), which can be employed to model time-varying regulatory processes. Almost all non-homogeneous DBNs that have been proposed in the literature follow the same paradigm and relax the homogeneity assumption by complementing the standard homogeneous DBN with a multiple changepoint process. Each time series segment defined by two demarcating changepoints is associated with separate interactions, and in this way the regulatory relationships are allowed to vary over time. However, the configuration space of the data segmentations (allocations) that can be obtained by changepoints is restricted. A complementary paradigm is to combine DBNs with mixture models, which allow for free allocations of the data points to mixture components. But this extension of the configuration space comes with the disadvantage that the temporal order of the data points can no longer be taken into account. In this paper I present a novel non-homogeneous DBN model, which can be seen as a consensus between the free allocation mixture DBN model and the changepoint-segmented DBN model. The key idea is to assume that the underlying allocation of the temporal data points follows a Hidden Markov model (HMM). The novel HMM---DBN model takes the temporal structure of the time series into account without putting a restriction onto the configuration space of the data point allocations. I define the novel HMM---DBN model and the competing models such that the regulatory network structure is kept fixed among components, while the network interaction parameters are allowed to vary, and I show how the novel HMM---DBN model can be inferred with Markov Chain Monte Carlo (MCMC) simulations. For the new HMM---DBN model I also present two new pairs of MCMC moves, which can be incorporated into the recently proposed allocation sampler for mixture models to improve convergence of the MCMC simulations. In an extensive comparative evaluation study I systematically compare the performance of the proposed HMM---DBN model with the performances of the competing DBN models in a reverse engineering context, where the objective is to learn the structure of a network from temporal network data.