Confidence In Predictions Research Articles

Wind power interval and point prediction model using neural network based multi-objective optimization

Wind power point and interval prediction plays an important role in dispatching. However, for obtaining both point estimations and prediction intervals (PIs), the existing models like constructing the probability density function are too complicated. This paper proposes a novel multi-objective upper and lower bound and point estimation (MOULPE) model. It constructs a neural network (NN) with double outputs to directly estimate the prediction intervals (PIs) and the median of PIs is calculated as point estimation. Considering wide decision-making space, the problem formulation of MOULPE is defined as three objectives which covers both evaluation indices of PIs and point prediction. Furthermore, based on elite opposition-based learning (EOBL), this paper improves non-dominated fast sort genetic algorithm-III (INSGA-III) to search the optimal front. Two criteria called prediction interval nominal confidence (PINC) and point prediction nominal error (PPNE) are adopted to pick out the best solution. According to the general requirements in literature, four examples of real wind power data are conducted. Compared with some state-of-the-art methods, the coverage probability of PIs constructed by the proposed model not only reaches the preset PINC, but the average width is also the lowest. Similarly, the point estimation error of the proposed method is less than PPNE.

Operational Prognostic Model Evaluation

Prognostic analytic models have become a viable way to reduce operational interruptions when sufficient timely data is available. This work describes a set of evaluation metrics which can characterize model performance as a degradation estimate and as a decision enabler. The model accuracy over time is assessed against a correlation with the remaining useful life. This yields both a prediction accuracy and confidence interval. The decision can be based on the level of confidence around the prediction, which is based on both how far into the future the event is predicted and how well the current health and its deterioration is estimated. With an effective means of evaluating prognostic models, better benchmarks can be established to communicate model effectiveness and appropriately schedule routine service.

A fundus image classification framework for learning with noisy labels.

Fundus images are widely used in the screening and diagnosis of eye diseases. Current classification algorithms for computer-aided diagnosis in fundus images rely on large amounts of data with reliable labels. However, the appearance of noisy labels degrades the performance of data-dependent algorithms, such as supervised deep learning. A noisy label learning framework suitable for the multiclass classification of fundus diseases is presented in this paper, which combines data cleansing (DC), adaptive negative learning (ANL), and sharpness-aware minimization (SAM) modules. Firstly, the DC module filters the noisy labels in the training dataset based on the prediction confidence. Then, the ANL module modifies the loss function by choosing complementary labels, which are neither the given labels nor the labels with the highest confidence. Moreover, for better generalization, the SAM module is applied by simultaneously optimizing the loss and its sharpness. Extensive experiments on both private and public datasets show that our method greatly promotes the performance for classification of multiple fundus diseases with noisy labels.

Similar Graph Face Clustering Based on Graph Convolutional Neural Network

Face clustering is primarily a method for grouping a large number of face images and has important applications in the fields of label-free face image annotation and image management. Traditional machine learning clustering algorithms do not work well on face image data and are unable to learn effectively on complex face image features. Recent research has turned to the use of graph convolutional neural networks (GCNs) to learn contextual information from neighbourhood features between face images for inference, which can significantly improve performance. Unlike the conventional link prediction of face data points and confidence prediction between vertices by using GCNs, for this paper a similarity graph based face clustering algorithm is proposed. Construction of subgraphs by K-nearest neighbour algorithm, Learn the similarity between the k-nearest neighbor subgraphs of each face instance by constructing subgraphs with the k-nearest neighbor algorithm. Using a bottom-up clustering strategy to merge subgraphs to complete the clustering, this algorithm improves the clustering results on complex face features by 20% and 7% compared to traditional machine learning methods such as K-means and DBSCAN respectively, improving the clustering accuracy and reducing the computational complexity.

Reliable assessment of uncertainty for appliance recognition in NILM using conformal prediction

AbstractA primary task of Non‐intrusive Load Monitoring (NILM) is the identification of appliances that are switched on or off. However, state‐of‐the‐art machine learning methods such as deep learning do not express uncertainty of their predictions. Especially in cases where appliances are confused, it is desirable that an NILM system can suggest multiple possible predictions to the end‐user, including its confidence and credibility of any given prediction. This can be achieved using conformal prediction, being an effective way to quantify uncertainty of a given machine learning model. In this work, conformal prediction is introduced for NILM and applied to a neural network. The approach is explained and supported by several examples.

Blandaltman: A command to create variants of Bland–Altman plots

Bland–Altman plots can be useful in paired data settings such as measurement-method comparison studies. A Bland–Altman plot has differences, percentage differences, or ratios on the y axis and a mean of the data pairs on the x axis, with 95% limits of agreement indicating the central 95% range of differences, percentage differences, or ratios. This range can vary with the mean. We introduce the community-contributed blandaltman command, which uniquely in Stata can 1) create Bland–Altman plots featuring ratios in addition to differences and percentage differences, 2) allow the limits of agreement for ratios and percentage differences to vary as a function of the mean, and 3) add confidence intervals, prediction intervals, and tolerance intervals to the plots.

Cheese consumption and multiple health outcomes: an umbrella review and updated meta-analysis of prospective studies.

This umbrella review aims to provide a systematic and comprehensive overview of current evidence from prospective studies on the diverse health effects of cheese consumption. We searched PubMed, Embase, and Cochrane Library to identify meta-analyses/pooled analyses of prospective studies examining the association between cheese consumption and major health outcomes from inception to August 31, 2022. We reanalyzed and updated previous meta-analyses and performed de novo meta-analyses with recently published prospective studies, where appropriate. We calculated the summary effect size, 95% prediction confidence intervals, between-study heterogeneity, small-study effects, and excess significance bias for each health outcome. We identified 54 eligible articles of meta-analyses/pooled analyses. After adding newly published original articles, we performed 35 updated meta-analyses and 4 de novo meta-analyses. Together with 8 previous meta-analyses, we finally included 47 unique health outcomes. Cheese consumption was inversely associated with all-cause mortality (highest compared with lowest category: RR = 0.95; 95% CI: 0.92, 0.99), cardiovascular mortality (RR = 0.93; 95% CI: 0.88, 0.99), incident cardiovascular disease (CVD) (RR = 0.92; 95% CI: 0.89, 0.96), coronary heart disease (CHD) (RR = 0.92; 95% CI: 0.86, 0.98), stroke (RR = 0.93; 95% CI: 0.89, 0.98), estrogen receptor-negative (ER-) breast cancer (RR = 0.89; 95% CI: 0.82, 0.97), type 2 diabetes (RR = 0.93; 95% CI: 0.88, 0.98), total fracture (RR = 0.90; 95% CI: 0.86, 0.95), and dementia (RR = 0.81; 95% CI: 0.66, 0.99). Null associations were found for other outcomes. According to the NutriGrade scoring system, moderate quality of evidence was observed for inverse associations of cheese consumption with all-cause and cardiovascular mortality, incident CVD, CHD, and stroke, and for null associations with cancer mortality, incident hypertension, and prostate cancer. Our findings suggest that cheese consumption has neutral to moderate benefits for human health.

3D-Vision-Transformer Stacking Ensemble for Assessing Prostate Cancer Aggressiveness from T2w Images.

Vision transformers represent the cutting-edge topic in computer vision and are usually employed on two-dimensional data following a transfer learning approach. In this work, we propose a trained-from-scratch stacking ensemble of 3D-vision transformers to assess prostate cancer aggressiveness from T2-weighted images to help radiologists diagnose this disease without performing a biopsy. We trained 18 3D-vision transformers on T2-weighted axial acquisitions and combined them into two- and three-model stacking ensembles. We defined two metrics for measuring model prediction confidence, and we trained all the ensemble combinations according to a five-fold cross-validation, evaluating their accuracy, confidence in predictions, and calibration. In addition, we optimized the 18 base ViTs and compared the best-performing base and ensemble models by re-training them on a 100-sample bootstrapped training set and evaluating each model on the hold-out test set. We compared the two distributions by calculating the median and the 95% confidence interval and performing a Wilcoxon signed-rank test. The best-performing 3D-vision-transformer stacking ensemble provided state-of-the-art results in terms of area under the receiving operating curve (0.89 [0.61-1]) and exceeded the area under the precision-recall curve of the base model of 22% (p < 0.001). However, it resulted to be less confident in classifying the positive class.

Addressing the quantification of meteorological uncertainties in the atmospheric transport simulations of the 133Xe industrial background

The French National Data Center (NDC) uses an automated simulation of the 133Xe worldwide atmospheric background as one of the means to categorize the radionuclide measurements of the Comprehensive Nuclear-Test-Ban Treaty (CTBT) International Monitoring System (IMS). These simulations take into account 133Xe releases from the known or assumed major industrial emitters in the world and global-scale meteorological data. However, a quantification of the simulation uncertainties in this operational set up is yet to be addressed. This work discusses the benefits of meteorological ensemble data as available from National Centers for Environmental Prediction (NCEP) for that purpose. For this study, the daily dispersion of releases from the Institute for Radio Elements (IRE), a medical isotope production facility located in Fleurus (Belgium), was calculated over one year with emissions measured in-site and ensemble meteorological data. The ensemble contains 31 members, which resulted in as many predictions of activity concentration for any given time and place. The resulting distribution statistics (mean, median and spread), and the control run, were confronted to the deterministic run and to measurements at one IMS-like station near Paris (France) and one IMS station in Freiburg (Germany). Overall, the ensemble results have decreased the simulation performance, as expected given the use of meteorological analyses only. However, contrasting patterns were found with a detailed analysis of daily activity concentration over two one-month-and-a-half periods. Noticeably, outlier results were found to carry the best forecast in some significant detections, proving their relevance for the measurement categorization, despite their isolated character. Importantly, the ensemble has allowed the quantification of meteorological uncertainties, which was beneficial in all cases. It either has improved the confidence of IMS data categorization or has pointed to low confidence predictions. A criterion to identify the latter is suggested, based on information provided by the ensemble distributions. In addition, maps of probability of detections and of relative spread are suggested to show additional benefits of ensemble meteorology.

Stochastic soiling loss models for heliostats in Concentrating Solar Power plants

Reflectance losses on solar mirrors due to soiling are a significant challenge for Concentrating Solar Power (CSP) plants. Soiling losses can vary significantly from site to site — with (absolute) reflectance losses varying from fractions of a percentage point up to several percentage points per day (pp/day), a fact that has motivated several studies in soiling predictive modelling. Yet, existing studies have so far neglected the characterization of statistical uncertainty in their parameters and predictions. In this paper, two reflectance loss models are proposed that model uncertainty: an extension of a previously developed physical model and a simplified model. A novel uncertainty characterization enables Maximum Likelihood Estimation techniques for parameter estimation for both models, and permits the estimation of parameter (and prediction) confidence intervals.The models are applied to data from ten soiling campaigns conducted at three Australian sites (Brisbane, Mount Isa, Wodonga). The simplified model produces high-quality predictions of soiling losses on novel data, while the semi-physical model performance is mixed. The statistical distributions of daily losses were estimated for different dust loadings. Under median conditions, the daily soiling losses for Brisbane, Mount Isa, and Wodonga are estimated as 0.53±0.66, 0.08±0.08, and 0.58±0.15 pp/day, respectively. Yet, higher observed dust loadings can drive average losses as high as 2 pp/day.Overall, the results suggest a relatively simple approach characterizing the statistical distributions of soiling losses using airborne dust measurements and short reflectance monitoring campaigns.

The AMG model coupled with Rock-Eval® analysis accurately predicts cropland soil organic carbon dynamics in the Tuojiang River Basin, Southwest China

Accurate soil organic carbon models are key to understand the mechanisms governing carbon sequestration in soil and to help develop targeted management strategies to carbon budget. The accuracy and reliability of soil organic carbon (SOC) models remains strongly limited by incorrect initialization of the conceptual kinetic pools and lack of stringent model evaluation using time-series datasets. Notably, due to legacy effects of management and land use change, the traditional spin-up approach for initial allocation of SOC among kinetic pools can bring substantial uncertainties in predicting the evolution of SOC stocks. The AMG model can fulfill these conditions as it is a parsimonious yet accurate SOC model using widely-available input data. In this study, we first evaluated the performance of AMGv2 before and after optimizing the potential mineralization rate (k0) of SOC stock following a leave-one-site-out cross-validation based on 24 long-term field experiments (LTEs) in the Southwest of China. Then, we used Rock-Eval® thermal analysis results as input variables in the PARTYSOC machine learning model to estimate the initial stable SOC fraction (CS/C0) for the 14 LTEs where soil samples were available. The results showed that initializing the CS/C0 ratio using PARTYSOC combined with the optimized k0 further improved the accuracy of model simulations (R2 = 0.87, RMSE = 0.25, d = 0.90). Combining average measured CS/C0 and k0 optimization across all 24 LTEs also improved the model predictive capability by 25% compared to using default parameterization, thus suggesting promising avenue for upscaling model applications at the regional level where only a few measurement data on SOC stability can be available. In conclusion, the new version of the AMG model developed in the Tuojiang River Basin context exhibits excellent performance. This result paves the way for further calibration and validation of the AMG model in a wider set of contexts, with the potential to significantly improve confidence in SOC predictions in croplands over regional scales.

A novel uncertainty-aware deep learning technique with an application on skin cancer diagnosis

Skin cancer, primarily resulting from the abnormal growth of skin cells, is among the most common cancer types. In recent decades, the incidence of skin cancer cases worldwide has risen significantly (one in every three newly diagnosed cancer cases is a skin cancer). Such an increase can be attributed to changes in our social and lifestyle habits coupled with devastating man-made alterations to the global ecosystem. Despite such a notable increase, diagnosis of skin cancer is still challenging, which becomes critical as its early detection is crucial for increasing the overall survival rate. This calls for advancements of innovative computer-aided systems to assist medical experts with their decision making. In this context, there has been a recent surge of interest in machine learning (ML), in particular, deep neural networks (DNNs), to provide complementary assistance to expert physicians. While DNNs have a high processing capacity far beyond that of human experts, their outputs are deterministic, i.e., providing estimates without prediction confidence. Therefore, it is of paramount importance to develop DNNs with uncertainty-awareness to provide confidence in their predictions. Monte Carlo dropout (MCD) is vastly used for uncertainty quantification; however, MCD suffers from overconfidence and being miss calibrated. In this paper, we use MCD algorithm to develop an uncertainty-aware DNN that assigns high predictive entropy to erroneous predictions and enable the model to optimize the hyper-parameters during training, which leads to more accurate uncertainty quantification. We use two synthetic (two moons and blobs) and a real dataset (skin cancer) to validate our algorithm. Our experiments on these datasets prove effectiveness of our approach in quantifying reliable uncertainty. Our method achieved 85.65 ± 0.18 prediction accuracy, 83.03 ± 0.25 uncertainty accuracy, and 1.93 ± 0.3 expected calibration error outperforming vanilla MCD and MCD with loss enhanced based on predicted entropy.

Understanding the Capability of Future Direct-imaging Observations to Quantify Atmospheric Chemical Effects of Stellar Proton Events

Models developed for Earth are often applied in exoplanet contexts. Validation in extraterrestrial settings can provide an important test of model realism and increase our confidence in model predictions. NASA’s upcoming space-based IROUV telescope will provide unprecedented opportunities to perform such tests. Here, we use the Planetary Spectrum Generator to simulate IROUV reflected-light spectroscopic observations of flare-driven photochemical changes produced by the Whole Atmosphere Community Climate Model, part of the Community Earth System Model framework. We find that NO2 is the most observable gas to target, and integrating the signal for two days following the flare and comparing to a baseline of preflare data would achieve the highest signal-to-noise ratio. The NO2 response is much larger for K-star tidally locked planets than G-star rapidly rotating planets and does not depend strongly on O2 level. The NO2 response should be observable for planets within 3–4 pc independent of the phase angle since the amount of reflected light is larger at smaller phases, but the NO2 concentration is low near the substellar point. This work outlines a methodology for validating and ground-truthing atmospheric chemistry models developed for Earth that could be useful for the numerical exploration of exoplanets.

Structure-based modeling of critical micelle concentration (CMC) of anionic surfactants in brine using intelligent methods

Critical micelle concentration (CMC) is one of the main physico-chemical properties of surface-active agents, also known as surfactants, with diverse theoretical and industrial applications. It is influenced by basic parameters such as temperature, pH, salinity, and the chemical structure of surfactants. Most studies have only estimated CMC at fixed conditions based on the surfactant’s chemical parameters. In the present study, we aimed to develop a set of novel and applicable models for estimating CMC of well-known anionic surfactants by considering both the molecular properties of surfactants and basic affecting factors such as salinity, pH, and temperature as modeling parameters. We employed the quantitative-structural property relationship technique to employ the molecular parameters of surfactant ions. We collected 488 CMC values from literature for 111 sodium-based anionic surfactants, including sulfate types, sulfonate, benzene sulfonate, sulfosuccinate, and polyoxyethylene sulfate. We computed 1410 optimized molecular descriptors for each surfactant using Dragon software to be utilized in the modelling processes. The enhanced replacement method was used for selecting the most effective descriptors for the CMC. A multivariate linear model and two non-linear models are the outputs of the present study. The non-linear models were produced using two robust machine learning approaches, stochastic gradient boosting (SGB) trees and genetic programming (GP). Statistical assessment showed highly applicable and acceptable accuracy of the newly developed models (RSGB2 = 0.999395 and RGP2 = 0.954946). The ultimate results showed the superiority and greater ability of the SGB method for making confident predictions.

Collective Human Opinions in Semantic Textual Similarity

Abstract Despite the subjective nature of semantic textual similarity (STS) and pervasive disagreements in STS annotation, existing benchmarks have used averaged human ratings as gold standard. Averaging masks the true distribution of human opinions on examples of low agreement, and prevents models from capturing the semantic vagueness that the individual ratings represent. In this work, we introduce USTS, the first Uncertainty-aware STS dataset with ∼15,000 Chinese sentence pairs and 150,000 labels, to study collective human opinions in STS. Analysis reveals that neither a scalar nor a single Gaussian fits a set of observed judgments adequately. We further show that current STS models cannot capture the variance caused by human disagreement on individual instances, but rather reflect the predictive confidence over the aggregate dataset.

NMR structure verifies the eponymous zinc finger domain of transcription factor ZNF750

ZNF750 is a nuclear transcription factor that activates skin differentiation and has tumor suppressor roles in several cancers. Unusually, ZNF750 has only a single zinc-finger (ZNF) domain, Z*, with an amino acid sequence that differs markedly from the CCHH family consensus. Because of its sequence differences Z* is classified as degenerate, presumed to have lost the ability to bind the zinc ion required for folding. AlphaFold predicts an irregular structure for Z* with low confidence. Low confidence predictions are often inferred to be intrinsically disordered regions of proteins, which would be the case if Z* did not bind Zn2+. We use NMR and CD spectroscopy to show that a 25–51 segment of ZNF750 corresponding to the Z* domain folds into a well-defined antiparallel ββα tertiary structure with a pM dissociation constant for Zn2+ and a thermal stability >80 °C. Of three alternative Zn2+ ligand sets, Z* uses a CCHC rather than the expected CCHH ligating motif. The switch in the last ligand maintains the folding topology and hydrophobic core of the classical ZNF motif. CCHC ZNFs are typically associated with protein–protein interactions, raising the possibility that ZNF750 interacts with DNA through other proteins rather than directly. The structure of Z* provides context for understanding the function of the domain and its cancer-associated mutations. We expect other ZNFs currently classified as degenerate could be CCHC-type structures like Z*.

Screening the stones of Venice: Mapping social perceptions of cultural significance through graph-based semi-supervised classification

Mapping cultural significance of heritage properties in urban environment from the perspective of the public has become an increasingly relevant process, as highlighted by the 2011 UNESCO Recommendation on the Historic Urban Landscape (HUL). With the ubiquitous use of social media and the prosperous developments in machine and deep learning, it has become feasible to collect and process massive amounts of information produced by online communities about their perceptions of heritage as social constructs. Moreover, such information is usually inter-connected and embedded within specific socioeconomic and spatiotemporal contexts. This paper presents a methodological workflow for using semi-supervised learning with graph neural networks (GNN) to classify, summarize, and map cultural significance categories based on user-generated content on social media. Several GNN models were trained as an ensemble to incorporate the multi-modal (visual and textual) features and the contextual (temporal, spatial, and social) connections of social media data in an attributed multi-graph structure. The classification results with different models were aligned and evaluated with the prediction confidence and agreement. Furthermore, message diffusion methods on graphs were proposed to aggregate the post labels onto their adjacent spatial nodes, which helps to map the cultural significance categories in their geographical contexts. The workflow is tested on data gathered from Venice as a case study, demonstrating the generation of social perception maps for this UNESCO World Heritage property. This research framework could also be applied in other cities worldwide, contributing to more socially inclusive heritage management processes. Furthermore, the proposed methodology holds the potential of diffusing any human-generated location-based information onto spatial networks and temporal timelines, which could be beneficial for measuring the safety, vitality, and/or popularity of urban spaces.

Uncertainty aware training to improve deep learning model calibration for classification of cardiac MR images.

Quantifying uncertainty of predictions has been identified as one way to develop more trustworthy artificial intelligence (AI) models beyond conventional reporting of performance metrics. When considering their role in a clinical decision support setting, AI classification models should ideally avoid confident wrong predictions and maximise the confidence of correct predictions. Models that do this are said to be well calibrated with regard to confidence. However, relatively little attention has been paid to how to improve calibration when training these models, i.e. to make the training strategy uncertainty-aware. In this work we: (i) evaluate three novel uncertainty-aware training strategies with regard to a range of accuracy and calibration performance measures, comparing against two state-of-the-art approaches, (ii) quantify the data (aleatoric) and model (epistemic) uncertainty of all models and (iii) evaluate the impact of using a model calibration measure for model selection in uncertainty-aware training, in contrast to the normal accuracy-based measures. We perform our analysis using two different clinical applications: cardiac resynchronisation therapy (CRT) response prediction and coronary artery disease (CAD) diagnosis from cardiac magnetic resonance (CMR) images. The best-performing model in terms of both classification accuracy and the most common calibration measure, expected calibration error (ECE) was the Confidence Weight method, a novel approach that weights the loss of samples to explicitly penalise confident incorrect predictions. The method reduced the ECE by 17% for CRT response prediction and by 22% for CAD diagnosis when compared to a baseline classifier in which no uncertainty-aware strategy was included. In both applications, as well as reducing the ECE there was a slight increase in accuracy from 69% to 70% and 70% to 72% for CRT response prediction and CAD diagnosis respectively. However, our analysis showed a lack of consistency in terms of optimal models when using different calibration measures. This indicates the need for careful consideration of performance metrics when training and selecting models for complex high risk applications in healthcare.

Dynamic applicability domain (dAD): compound-target binding affinity estimates with local conformal prediction.

Increasing efforts are being made in the field of machine learning to advance the learning of robust and accurate models from experimentally measured data and enable more efficient drug discovery processes. The prediction of binding affinity is one of the most frequent tasks of compound bioactivity modelling. Learned models for binding affinity prediction are assessed by their average performance on unseen samples, but point predictions are typically not provided with a rigorous confidence assessment. Approaches such as the conformal predictor framework equip conventional models with a more rigorous assessment of confidence for individual point predictions. In this paper, we extend the inductive conformal prediction (ICP) framework for interaction data, in particular the compound-target binding affinity prediction task. The new framework is based on dynamically defined calibration sets that are specific for each testing pair and provides prediction assessment in the context of calibration pairs from its compound-target neighbourhood, enabling improved estimates based on the local properties of the prediction model. The effectiveness of the approach is benchmarked on several publicly available datasets and tested in realistic use-case scenarios with increasing levels of difficulty on a complex compound-target binding affinity space. We demonstrate that in such scenarios, novel approach combining applicability domain paradigm with conformal prediction framework, produces superior confidence assessment with valid and more informative prediction regions compared to other state-of-the-art conformal prediction approaches. Dataset and the code are available on GitHub (https://github.com/mlkr-rbi/dAD). Supplementary data are available at Bioinformatics online.

A Parameter Correction method of CFD based on the Approximate Bayesian Computation technique

Numerical simulation and modeling techniques are becoming the primary research tools for aerodynamic analysis and design. However, various uncertainties in physical modeling and numerical simulation seriously affect the credibility of Computational Fluid Dynamics (CFD) simulation results. Therefore, CFD models need to be adjusted and modified with consideration of uncertainties to improve the prediction accuracy and confidence level of CFD numerical simulations. This paper presents a parameter correction method of CFD for aerodynamic analysis by making full use of the advantages of the Approximate Bayesian Computation (ABC) technique in dealing with the analysis and inference of complex statistical models, in which the parameters of turbulence models for CFD are inferenced. The proposed parameter correction method is applied to the aerodynamic prediction of the NACA0012 airfoil. The results show the feasibility and effectiveness of the proposed approach in improving CFD prediction accuracy.