Probabilistic Machine Learning Research Articles

The role of phosphorus and nitrogen on chlorophyll a: Evidence from hundreds of lakes

The effect of nutrients on phytoplankton biomass in lakes continues to be a subject of debate by aquatic scientists. However, determining whether or not chlorophyll a (CHL) is limited by phosphorus (P) and/or nitrogen (N) is rarely considered using a probabilistic method in studies of hundreds of lakes across broad spatial extents. Several studies have applied a unified CHL-nutrient relationship to determine nutrient limitation, but pose a risk of ecological fallacy because they neglect spatial heterogeneity in ecological contexts. To examine whether or not CHL is limited by P, N, or both nutrients in hundreds of lakes and across diverse ecological settings, a probabilistic machine learning method, Bayesian Network, was applied. Spatial heterogeneity in ecological context was accommodated by the probabilistic nature of the results. We analyzed data from 1382 lakes in 17 US states to evaluate the cause-effect relationships between CHL and nutrients. Observations of CHL, total phosphorus (TP), and total nitrogen (TN) were discretized into three trophic states (oligo-mesotrophic, eutrophic, and hypereutrophic) to train the model. We found that although both nutrients were related to CHL trophic state, TP was more related to CHL than TN, especially under oligo-mesotrophic and eutrophic CHL conditions. However, when the CHL trophic state was hypereutrophic, both TP and TN were important. These results provide additional evidence that P-limitation is more likely under oligo-mesotrophic or eutrophic CHL conditions and that co-limitation of P and N occurs under hypereutrophic CHL conditions. We also found a decreasing pattern of the TN/TP ratio with increasing CHL concentrations, which might be a key driver for the role change of nutrients. Previous work performed at smaller scales support our findings, indicating potential for extension of our findings to other regions. Our findings enhance the understanding of nutrient limitation at macroscales and revealed that the current debate on the limiting nutrient might be caused by failure to consider CHL trophic state. Our findings also provide prior information for the site-specific eutrophication management of unsampled or data-limited lakes.

Learning Coarse-Grained Potentials for Binary Fluids.

For a multiple-fluid system, CG models capable of accurately predicting the interfacial properties as a function of curvature are still lacking. In this work, we propose a new probabilistic machine learning (ML) model for learning CG potentials for binary fluids. The water-hexane mixture is selected as a typical immiscible binary liquid-liquid system. We develop a new CG force field (FF) using the Shinoda-DeVane-Klein (SDK) FF framework and compute parameters in this CG FF using the proposed probabilistic ML method. It is shown that a standard response-surface approach does not provide a unique set of parameters, as it results in a loss function with multiple shallow minima. To address this challenge, we develop a probabilistic ML approach where we compute the probability density function (PDF) of parameters that minimize the loss function. The PDF has a well-defined peak corresponding to a unique set of parameters in the CG FF that reproduces the desired properties of a liquid-liquid interface. We compare the performance of the new CG FF with several existing FFs for the water-hexane mixture, including two atomistic and three CG FFs with respect to modeling the interface structure and thermodynamic properties. It is demonstrated that the new FF significantly improves the CG model prediction of both the interfacial tension and structure for the water-hexane mixture.

Incorporating physical constraints in a deep probabilistic machine learning framework for coarse-graining dynamical systems

Data-based discovery of effective, coarse-grained (CG) models of high-dimensional dynamical systems presents a unique challenge in computational physics and particularly in the context of multiscale problems. The present paper offers a data-based, probabilistic perspective that enables the quantification of predictive uncertainties. One of the outstanding problems has been the introduction of physical constraints in the probabilistic machine learning objectives. The primary utility of such constraints stems from the undisputed physical laws such as conservation of mass, energy etc. that they represent. Furthermore and apart from leading to physically realistic predictions, they can significantly reduce the requisite amount of training data which for high-dimensional, multiscale systems are expensive to obtain (Small Data regime). We formulate the coarse-graining process by employing a probabilistic state-space model and account for the aforementioned equality constraints as virtual observables in the associated densities. We demonstrate how deep neural nets in combination with probabilistic inference tools can be employed to identify the coarse-grained variables and their evolution model without ever needing to define a fine-to-coarse (restriction) projection and without needing time-derivatives of state variables.We advocate a sparse Bayesian learning perspective which avoids overfitting and reveals the most salient features in the CG evolution law. The formulation adopted enables the quantification of a crucial, and often neglected, component in the CG process, i.e. the predictive uncertainty due to information loss. Furthermore, it is capable of reconstructing the evolution of the full, fine-scale system and therefore the observables of interest need not be selected a priori. We demonstrate the efficacy of the proposed framework by applying it to systems of interacting particles and a series of images of a nonlinear pendulum. In both cases we identify the underlying coarse dynamics and can generate extrapolative predictions including the forming and propagation of a shock for the particle systems and a stable trajectory in the phase space for the pendulum.

Retrieval of aboveground crop nitrogen content with a hybrid machine learning method

Hyperspectral acquisitions have proven to be the most informative Earth observation data source for the estimation of nitrogen (N) content, which is the main limiting nutrient for plant growth and thus agricultural production. In the past, empirical algorithms have been widely employed to retrieve information on this biochemical plant component from canopy reflectance. However, these approaches do not seek for a cause-effect relationship based on physical laws. Moreover, most studies solely relied on the correlation of chlorophyll content with nitrogen, and thus neglected the fact that most N is bound in proteins. Our study presents a hybrid retrieval method using a physically-based approach combined with machine learning regression to estimate crop N content. Within the workflow, the leaf optical properties model PROSPECT-PRO including the newly calibrated specific absorption coefficients (SAC) of proteins, was coupled with the canopy reflectance model 4SAIL to PROSAIL-PRO. The latter was then employed to generate a training database to be used for advanced probabilistic machine learning methods: a standard homoscedastic Gaussian process (GP) and a heteroscedastic GP regression that accounts for signal-to-noise relations. Both GP models have the property of providing confidence intervals for the estimates, which sets them apart from other machine learners. Moreover, a GP-based sequential backward band removal algorithm was employed to analyze the band-specific information content of PROSAIL-PRO simulated spectra for the estimation of aboveground N. Data from multiple hyperspectral field campaigns, carried out in the framework of the future satellite mission Environmental Mapping and Analysis Program (EnMAP), were exploited for validation. In these campaigns, corn and winter wheat spectra were acquired to simulate spectral EnMAP data. Moreover, destructive N measurements from leaves, stalks and fruits were collected separately to enable plant-organ-specific validation. The results showed that both GP models can provide accurate aboveground N simulations, with slightly better results of the heteroscedastic GP in terms of model testing and against in situ N measurements from leaves plus stalks, with root mean square error (RMSE) of 2.1 g/m². However, the inclusion of fruit N content for validation deteriorated the results, which can be explained by the inability of the radiation to penetrate the thick tissues of stalks, corn cobs and wheat ears. GP-based band analysis identified optimal spectral settings with ten bands mainly situated in the shortwave infrared (SWIR) spectral region. Use of well-known protein absorption bands from the literature showed comparative results. Finally, the heteroscedastic GP model was successfully applied on airborne hyperspectral data for N mapping. We conclude that GP algorithms, and in particular the heteroscedastic GP, should be implemented for global agricultural monitoring of aboveground N from future imaging spectroscopy data.

Naïve bayes classifier models for cerebral infarction classification

Cerebral infarction is a condition of tissue damage in the brain caused by inadequate oxygen supply caused by obstruction of the flow of regions to the area (ischemia). Brain ischemia is more common than hemorrhagic, but surgery is most often performed in hemorrhagic strokes. This condition is also called stroke infarction. In stroke infarction does not occur bleeding. Changes in brain blood vessel walls can be primary due to congenital or degenerative abnormalities and secondary processes caused by other processes such as inflammation, arteriosclerosis, hypertension, diabetes mellitus and many other processes, so the cause of stroke is very multifactorial. The final sign of cell damage due to ischemia is marked by the nucleus which becomes picnotic and fragmented. Research shows that stroke infarction can occur at the age of 15-55 years. To diagnose the presence or absence of cerebral infarction in the brain is not enough just to use a CT scan, therefore machine learning will also be used to diagnose the presence or absence of cerebral infarction in the brain. For this reason, the authors propose the Naïve Bayes Classifier method as a classification method that has good accuracy, good precision, good memory, and a good F1-score in calcifying a patient whose brain has cerebral infarction or not. In this proposed method, Naïve Bayes Classifier is a probabilistic machine learning model used to classify. Naïve Bayes Classifier is a simple probability technique based on the Bayes theorem with the assumption of independence among predictors. In simple terms, Naïve Bayes Classifier assumes that the presence of certain features in a class is not related to the presence of other features. This method can achieve an accuracy value of up to 92.43%, so this method can be an efficient classification tool.

All-Spin Bayesian Neural Networks

Probabilistic machine learning enabled by the Bayesian formulation has recently gained significant attention in the domain of automated reasoning and decision-making. While impressive strides have been made recently to scale up the performance of deep Bayesian neural networks, they have been primarily standalone software efforts without any regard to the underlying hardware implementation. In this paper, we propose an "All-Spin" Bayesian Neural Network where the underlying spintronic hardware provides a better match to the Bayesian computing models. To the best of our knowledge, this is the first exploration of a Bayesian neural hardware accelerator enabled by emerging post-CMOS technologies. We develop an experimentally calibrated device-circuit-algorithm co-simulation framework and demonstrate $24\times$ reduction in energy consumption against an iso-network CMOS baseline implementation.

Healthcare pathway discovery and probabilistic machine learning

Background and purposeHealthcare pathways define the execution sequence of clinical activities as patients move through a treatment process, and they are critical for maintaining quality of care. The aim of this study is to combine healthcare pathway discovery with predictive models of individualized recovery times. The pathway discovery has a particular emphasis on producing pathway models that are easy to interpret for clinicians without a sufficient background in process mining. The predictive model takes the stochastic volatility of pathway performance indicators into account. MethodThis study utilizes the business process-mining software ProM to design a process mining pipeline for healthcare pathway discovery and enrichment using hospital records. The efficacy of combining learned healthcare pathways with probabilistic machine learning models is demonstrated via a case study that applies the proposed process mining pipeline to discover appendicitis pathways from hospital records. Machine learning methodologies based on probabilistic programming are utilized to explore pathway features that influence patient recovery time. ResultsThe produced appendicitis pathway models are easy for clinical interpretation and provide an unbiased overview of patient movements through the treatment process. Analysis of the discovered pathway model enables reasons for longer than usual treatment times to be explored and deviations from standard treatment pathways to be identified. A probabilistic regression model that estimates patient recovery time based on the information extracted by the process mining pipeline is developed and has the potential to be very useful for hospital scheduling purposes. ConclusionThis study establishes the application of the business process modelling tool ProM for the improvement of healthcare pathway mining methods. The proposed pipeline for healthcare pathway discovery has the potential to support the development of probabilistic machine learning models to further relate healthcare pathways to performance indicators such as patient recovery time.

A Probabilistic Machine Learning Approach to Scheduling Parallel Loops with Bayesian Optimization

This paper proposes Bayesian optimization augmented factoring self-scheduling (BO FSS), a new parallel loop scheduling strategy. BO FSS is an automatic tuning variant of the factoring self-scheduling (FSS) algorithm and is based on Bayesian optimization (BO), a black-box optimization algorithm. Its core idea is to automatically tune the internal parameter of FSS by solving an optimization problem using BO. The tuning procedure only requires online execution time measurement of the target loop. In order to apply BO, we model the execution time using two Gaussian process (GP) probabilistic machine learning models. Notably, we propose a locality-aware GP model, which assumes that the temporal locality effect resembles an exponentially decreasing function. By accurately modeling the temporal locality effect, our locality-aware GP model accelerates the convergence of BO. We implemented BO FSS on the GCC implementation of the OpenMP standard and evaluated its performance against other scheduling algorithms. Also, to quantify our method's performance variation on different workloads, or workload-robustness in our terms, we measure the minimax regret. According to the minimax regret, BO FSS shows more consistent performance than other algorithms. Within the considered workloads, BO FSS improves the execution time of FSS by as much as 22% and 5% on average.

Information Theory Based Probabilistic Machine Learning And Wireline Inversion: Surat Basin Case Study

We present a probabilistic machine learning approach to determine lithologies for the wireline data from the Springbok Sandstone formation in the Surat Basin. Deterministic inversions of the data a...

Machine learning methods for classification of the green infrastructure in city areas

Reducing green urban infrastructure pose a huge threat to cities sustainability. It is important to monitor and track the health of vegetation. For efficient planning of urban development and maintenance of green urban infrastructure, the key is to have up to date maps. Using satellite imagery is the easiest way to cover large city areas. In order to map vegetation, there are many possible solutions; however, the most effective way is using machine learning methods. Machine learning is divided into supervised and unsupervised classification and each can be divided into several different methods. Many authors have considered different methods and they use them to access accuracy on satellite image information extraction. They have used different satellite images and naturally higher resolution imagery results in better classification. However, there is still lack of comprehensive analysis of more supervised machine learning methods in similar cities. This paper aims to provide analysis of four different machine learning methods: support vector machine, artificial neural network, naïve Bayes and random forest. The objective of the support vector machine algorithm is to find a hyperplane in an N-dimensional space that distinctly classifies the data points where hyperplanes are decision boundaries that help to classify the data points and support vectors data points that are closer to the hyperplane and influence the position and orientation of the hyperplane. Artificial neural networks are brain-inspired systems which are intended to replicate the way that humans learn. Neural networks consist of input and output layers, and hidden layers. They are excellent tools for finding patterns which are far too complex or numerous for a human programmer to extract and teach the machine to recognize. A Naive Bayes classifier is a probabilistic machine learning model that is used for classification task and the crux of the classifier is based on the Bayes theorem. Random Forest creates a forest and makes it random and is an ensemble of Decision Trees, most of the time trained with the bagging method which general idea is that a combination of learning models increases the overall result. All of the mentioned methods will be tested on Sentinel-2A imagery. Sentinel-2A multispectral imager has thirteen sensors which is useful in vegetation extraction. Methods will be compared using error matrix and kappa statistics.

Mutual Information Input Selector and Probabilistic Machine Learning Utilisation for Air Pollution Proxies

An air pollutant proxy is a mathematical model that estimates an unobserved air pollutant using other measured variables. The proxy is advantageous to fill missing data in a research campaign or to substitute a real measurement for minimising the cost as well as the operators involved (i.e., virtual sensor). In this paper, we present a generic concept of pollutant proxy development based on an optimised data-driven approach. We propose a mutual information concept to determine the interdependence of different variables and thus select the most correlated inputs. The most relevant variables are selected to be the best proxy inputs, where several metrics and data loss are also involved for guidance. The input selection method determines the used data for training pollutant proxies based on a probabilistic machine learning method. In particular, we use a Bayesian neural network that naturally prevents overfitting and provides confidence intervals around its output prediction. In this way, the prediction uncertainty could be assessed and evaluated. In order to demonstrate the effectiveness of our approach, we test it on an extensive air pollution database to estimate ozone concentration.

A Physics-Informed Data-Driven Approach to Additive Manufacturing Parameter Optimization

Abstract A novel framework including experimental and model-based techniques saves time and enables the introduction of new alloys for additive manufacturing. This article describes the first phase of the probabilistic machine learning framework that was successfully demonstrated to rapidly define optimum parameter sets for commercial high-temperature nickel superalloys, as well as to guide alloy design and selection for compatibility with laser powder bed fusion additive manufacturing.

Comparing unsupervised probabilistic machine learning methods for market basket analysis

We compare several unsupervised probabilistic machine learning methods for market basket analysis, namely binary factor analysis, two topic models (latent Dirichlet allocation and the correlated topic model), the restricted Boltzmann machine and the deep belief net. After an overview of previous applications of unsupervised probabilistic machine learning methods to market basket analysis we shortly present the methods which we investigate and outline their estimation. Performance is measured by tenfold cross-validated log likelihood values. Binary factor analysis vastly outperforms topic models. The restricted Boltzmann machine attains a similar performance advantage over binary factor analysis. Overall, a deep belief net with 45 variables in the first and 15 variables in the second hidden layers turns out to be the best model. We also compare the investigated machine learning methods with respect to ease of interpretation and runtimes. In addition, we show how to interpret the relationships between hidden variables and observed category purchases. To demonstrate managerial implications we estimate the effect of promoting each category both on purchase probability increases of other product categories and the relative increase of basket size. Finally, we indicate several possibilities to extend restricted Boltzmann machines and deep belief nets for market basket analysis.

A Method to identify anomalies in stock market trading based on Probabilistic Machine Learning

<p>Financial operations involve a significant amount of resources and can directly or indirectly affect the lives of virtually all people. For the efficiency and transparency in this context, it is essential to identify financial crimes and to punish the responsible. However, the large number of operations makes it unfeasible for analyzes made exclusively by humans. Thus, the application of automated data analysis techniques is essential. Within this scenario, this work presents a method that identifies anomalies that may be associated with operations in the stock exchange market prohibited by law. Specifically, we seek to find patterns related to insider trading. These types of operations can generate big losses for investors. In this work, publicly available information by the SEC and CVM, based on real cases on BOVESPA, NYSE and NASDAQ stock exchanges, is used as a training base. The method includes the creation of several candidate variables and the identification of relevant variables. With this definition, classifiers based on decision trees and Bayesian networks are constructed, and, after, evaluated and selected. The computational cost of performing such tasks can be quite significant, and it grows quickly with the amount of analyzed data. For this reason, the method considers the use of machine learning algorithms distributed in a computational cluster. In order to perform such tasks, we use the Weka framework with modules that allows the distribution of the processing load in a Hadoop cluster. The use of a computational cluster to execute learning algorithms in a large amount of data has been an active area of research, and this work contributes to the analysis of data in the specific context of financial operations. The obtained results show the feasibility of the approach, although the quality of the results is limited by the exclusive use of publicly available data.</p>

A physics-aware, probabilistic machine learning framework for coarse-graining high-dimensional systems in the Small Data regime

The automated construction of coarse-grained models represents a pivotal component in computer simulation of physical systems and is a key enabler in various analysis and design tasks related to uncertainty quantification. Pertinent methods are severely inhibited by the high-dimension of the parametric input and the limited number of training input/output pairs that can be generated when computationally demanding forward models are considered. Such cases are frequently encountered in the modeling of random heterogeneous media where the scale of the microstructure necessitates the use of high-dimensional random vectors and very fine discretizations of the governing equations. The present paper proposes a probabilistic Machine Learning framework that is capable of operating in the presence of Small Data by exploiting aspects of the physical structure of the problem as well as contextual knowledge. As a result, it can perform comparably well under extrapolative conditions. It unifies the tasks of dimensionality and model-order reduction through an encoder-decoder scheme that simultaneously identifies a sparse set of salient lower-dimensional microstructural features and calibrates an inexpensive, coarse-grained model which is predictive of the output. Information loss is accounted for and quantified in the form of probabilistic predictive estimates. The learning engine is based on Stochastic Variational Inference. We demonstrate how the variational objectives can be used not only to train the coarse-grained model, but also to suggest refinements that lead to improved predictions.

APPROACH FOR 4-D TRAJECTORY MANAGEMENT BASED ON HMM AND TRAJECTORY SIMILARITY

To improve the accuracy and stability of flight trajectory prediction, a novel four-dimensional (4-D) trajectory management approach is proposed in this paper, which consists of the estimation and updating procedure. Historical flight trajectories are proved to be safe and feasible based on the real-time traffic situation, and serve as the data foundation of 4-D trajectory management in this paper. To achieve the goal of 4-D trajectory management, we firstly apply probabilistic statistical models and machine learning approach to predict the fly-over time and altitude of waypoints along the planning route before the flight takes off. Hidden Markov Models (HMMs) are regarded as the probabilistic model to represent the position and altitude transition patterns of the aircraft during the flight operation. The EM algorithm is applied to optimize model parameters of HMMs to fit the training data (historical trajectory set). Then the models with optimized parameters are used to predict the pre-takeoff 4-D trajectory by inferring an optimal hidden state sequence. Finally, after the flight takes off, we propose an algorithm to correct the pre-takeoff prediction results by considering the trajectory similarity between collected path of current execution and its historical trajectories. Simulations with real data show that the prediction results (fly-over time and altitude) of our proposed algorithm are more accurate than that of other existing methods, and would tend to be more credible after correcting with the proposed algorithm. Moreover, the prediction errors of our approach are stable during the whole flight, which is the bottleneck of existing d\eterministic models.

Optimization of Linkage between North Carolina EMS and ED Data: EMS Naloxone Cases

Optimization of Linkage between North Carolina EMS and ED Data: EMS Naloxone Cases

Probabilistic machine learning approach to bridge fatigue failure analysis due to vehicular overloading

With the rapid development of freight transportation, truck overloading becomes very common and severe, posing a great threat to the safety of bridges, and it can even result in bridge failure. Traditional approaches investigating the overloading-induced fatigue damage on bridges, such as finite element analysis (FEA) and reliability analysis, have proven to be computationally expensive and model dependent. In this study, the prediction of fatigue failure probability of bridges due to traffic overloading was investigated by using the feedforward neural network and the Monte Carlo method. Our results show that based on a finite set of training data for the bridge under consideration, the proposed machine-learning-based approach can assist in providing an instantaneous assessment of the fatigue failure probability with high accuracy.

Multi-stage hybridized online sequential extreme learning machine integrated with Markov Chain Monte Carlo copula-Bat algorithm for rainfall forecasting

To ameliorate agricultural impacts due to persistent drought-risks by promoting sustainable utilization and pre-planning of water resources, accurate rainfall forecasting models, addressing the dynamic nature of drought phenomenon, is crucial. In this paper, a multi-stage probabilistic machine learning model is designed and evaluated for forecasting monthly rainfall. The multi-stage hybrid MCMC-Cop-Bat-OS-ELM model utilizing online-sequential extreme learning machines integrated with Markov Chain Monte Carlo (MCMC) based bivariate-copula and the Bat algorithm is employed to incorporate significant antecedent rainfall (t–1) as the model's predictor in the training phase. After computing the partial autocorrelation function (PACF) at the first stage, twenty-five MCMC based copulas (i.e., Gaussian, t, Clayton, Gumble, Frank and Fischer-Hinzmann etc.) are adopted to determine the dependence of antecedent month's rainfall with the current and future rainfall at the second stage of the model design. Bat algorithm is applied to sort the optimal MCMC-copula model by a feature selection strategy at the third stage. At the fourth stage, PACF's of the optimal MCMC-copula model are computed to couple the output with OS-ELM algorithm to forecast future rainfall values in an independent test dataset. As a benchmarking process, standalone extreme learning machine (ELM) and random forest (RF) is also integrated with MCMC based copulas and the Bat algorithm, yielding a hybrid MCMC-Cop-Bat-ELM and a MCMC-Cop-Bat-RF models. The proposed multi-stage hybrid model is tested in agricultural belt region in Faisalabad, Jhelum and Multan, located in Pakistan. The testing performance of all three hybridized models, according to robust statistical error metrics, is satisfactory in comparison to the standalone counterparts, however the multi-stage, hybridized MCMC-Cop-Bat-OS-ELM model is found to be a superior tool for forecasting monthly rainfall. This multi-stage probabilistic learning model can be explored as a pertinent decision-support tool for agricultural water resources management in arid and semi-arid regions where a statistically significant relationship with antecedent rainfall exists.

Scalable importance sampling estimation of Gaussian mixture posteriors in Bayesian networks

In this paper we propose a scalable importance sampling algorithm for computing Gaussian mixture posteriors in conditional linear Gaussian Bayesian networks. Our contribution is based on using a stochastic gradient ascent procedure taking as input a stream of importance sampling weights, so that a mixture of Gaussians is dynamically updated with no need to store the full sample. The algorithm has been designed following a Map/Reduce approach and is therefore scalable with respect to computing resources. The implementation of the proposed algorithm is available as part of the AMIDST open-source toolbox for scalable probabilistic machine learning (http://www.amidsttoolbox.com).