Output Probability Distribution Research Articles

Unveiling the unseen potential of graph learning through MLPs: Effective graph learners using propagation-embracing MLPs

Recent studies attempted to utilize multilayer perceptrons (MLPs) to solve semi-supervised node classification on graphs, by training a student MLP by knowledge distillation (KD) from a teacher graph neural network (GNN). While previous studies have focused mostly on training the student MLP by matching the output probability distributions between the teacher and student models during KD, it has not been systematically studied how to inject the structural information in an explicit and interpretable manner. Inspired by GNNs that separate feature transformation T and propagation Π, we re-frame the KD process as enabling the student MLP to explicitly learn both T and Π. Although this can be achieved by applying the inverse propagation Π−1 before distillation from the teacher GNN, it still comes with a high computational cost from large matrix multiplications during training. To solve this problem, we propose Propagate & Distill (P&D), which propagates the output of the teacher GNN before KD and can be interpreted as an approximate process of the inverse propagation Π−1. Through comprehensive evaluations using real-world benchmark datasets, we demonstrate the effectiveness of P&D by showing further performance boost of the student MLP.

Incorporating recklessness to collaborative filtering based recommender systems

Recommender systems are intrinsically tied to a reliability/coverage dilemma: The more reliable we desire the forecasts, the more conservative the decision will be and thus, the fewer items will be recommended. This causes a detriment to the predictive capability of the system, as it is only able to estimate potential interest in items for which there is a consensus in their evaluation, rather than being able to estimate potential interest in any item. In this paper, we propose the inclusion of a new term in the learning process of matrix factorization-based recommender systems, called recklessness, that takes into account the variance of the output probability distribution of the predicted ratings. In this way, gauging this recklessness measure we can force more spiky output distribution, enabling the control of the risk level desired when making decisions about the reliability of a prediction. Experimental results demonstrate that recklessness not only allows for risk regulation but also improves the quantity and quality of predictions provided by the recommender system.

Temporal Scenarios-based Assessment of Maximum Hosting Capacity of Renewable Generation in Distribution Systems Considering ANM Techniques

Abstract Incorporating large-scale renewable distributed generation (RDG) into the power grid introduces significant challenges associated with the indeterminacy of the power system, significantly impacting the comprehensive integration and efficient utilization of the distribution network. This underscores the imperative to investigate scientific evaluation methods and mechanisms for enhancing the RDG absorptive capacity within the distribution network. In light of the stochastic and temporal characteristics inherent to RDG, a novel approach for assessing the RDG hosting capacity within distribution networks is proposed, taking temporal scenarios into account. This method optimizes the RDG output probability distribution model based on its temporal fluctuation change rule, and constructs RDG temporal scenario model by using scenario reduction technology. Subsequently, a temporal scenarios-based RDG hosting capacity assessment model is established, taking into account active network management (ANM) techniques. The effectiveness of the proposed methodology is substantiated through simulation results conducted on a modified IEEE 33-bus system.

Improving Probability-based Prompt Selection Through Unified Evaluation and Analysis

Abstract Previous work in prompt engineering for large language models has introduced different gradient-free probability-based prompt selection methods that aim to choose the optimal prompt among the candidates for a given task but have failed to provide a comprehensive and fair comparison between each other. In this paper, we propose a unified framework to interpret and evaluate the existing probability-based prompt selection methods by performing extensive experiments on 13 common and diverse NLP tasks. We find that each of the existing methods can be interpreted as some variant of the method that maximizes mutual information between the input and the predicted output (MI). Utilizing this finding, we develop several other combinatorial variants of MI and increase the effectiveness of the oracle prompt selection method from 87.79% to 94.98%, measured as the ratio of the performance of the selected prompt to that of the optimal oracle prompt. Furthermore, considering that all the methods rely on the output probability distribution of the model that might be biased, we propose a novel calibration method called Calibration by Marginalization (CBM) that is orthogonal to the existing methods and helps increase the prompt selection effectiveness of the best method to 96.85%, achieving 99.44% of the oracle prompt F1 without calibration.1

Network partition and distributed voltage coordination control strategy of active distribution network system considering photovoltaic uncertainty

Aiming at the voltage exceeding problem caused by the introduction of Photovoltaic (PV) in Active distribution network (ADN), a comprehensive electrical distance index was established considering active and reactive voltage sensitivity, and the uncertainty of traditional PV probability density function (PDF) was optimized by random simulation method. The electrical distance expectation matrix between nodes was established by discretization of PV output probability distribution characteristics, and the ADN system was partitioned by affinity propagation (AP) algorithm. Taking active power loss as the objective function, the non-convex reactive power optimization control problem is transformed into a convex quadratic programming problem by LinDistFlow equation. An improved alternating direction method of multipliers (ADMM) which adaptively adjusts the penalty parameters is used to convert the finite boundaries of adjacent regions into data. By coordinating on-load tap-changer (OLTC), capacitor banks (CBs), and PV inverters on different time scales, the fast optimal control of global voltage in ADN is realized. The proposed method is tested on IEEE 33-bus and IEEE 123-bus distribution systems. After the optimization control algorithm described in this paper, the system network loss was reduced by 3.88%. Compared with the semi-definite programming relaxation, the calculation speed of LinDistFlow equation is increased by 72.2%. The results fully verify the feasibility and high efficiency of the control strategy described in this paper.

Masked Siamese Prompt Tuning for Few-Shot Natural Language Understanding

Recently, prompt-based learning has shown excellent performance on few-shot scenarios. Using frozen language models to tune trainable continuous prompt embeddings has become a popular and powerful methodology. For few-shot natural language understanding, even if we freeze the parameters of the pre-trained language model, the learned pseudo-prompts might still be overfitted. In this paper, we propose a novel masked siamese prompt tuning (MSP-tuning) to improve few-shot natural language understanding. Concretely, MSP-tuning masks randomly out part of the prompt tokens to get a pair of masked siamese prompts for each sample. Each training sample is then fed to the model twice with the masked siamese prompts. Finally, MSP-tuning minimizes the JS-divergence between the two output probability distributions of the pre-trained language model to regularize the model further. Experiment results on the few-shot GLUE benchmark and SuperGLUE benchmark show that MSP-tuning outperforms previous approaches. Numerically, our MSP-tuning achieves an average improvement of 1.79% (BERT-base) and 1.39% (BERT-large) of the GLUE benchmark and 1.90% (RoBERTa-base) and 1.71% (RoBERTa-large) of the SuperGLUE benchmark compared to state-of-the-art method P-tuning. Our method facilitates applying large pre-trained language models in natural language understanding.

Improving Efficiency of Iso-Surface Extraction on Implicit Neural Representations Using Uncertainty Propagation.

Implicit Neural representations (INRs) are widely used for scientific data reduction and visualization by modeling the function that maps a spatial location to a data value. Without any prior knowledge about the spatial distribution of values, we are forced to sample densely from INRs to perform visualization tasks like iso-surface extraction which can be very computationally expensive. Recently, range analysis has shown promising results in improving the efficiency of geometric queries, such as ray casting and hierarchical mesh extraction, on INRs for 3D geometries by using arithmetic rules to bound the output range of the network within a spatial region. However, the analysis bounds are often too conservative for complex scientific data. In this paper, we present an improved technique for range analysis by revisiting the arithmetic rules and analyzing the probability distribution of the network output within a spatial region. We model this distribution efficiently as a Gaussian distribution by applying the central limit theorem. Excluding low probability values, we are able to tighten the output bounds, resulting in a more accurate estimation of the value range, and hence more accurate identification of iso-surface cells and more efficient iso-surface extraction on INRs. Our approach demonstrates superior performance in terms of the iso-surface extraction time on four datasets compared to the original range analysis method and can also be generalized to other geometric query tasks.

Entropic property of randomized QAOA circuits

Quantum approximate optimization algorithm (QAOA) aims to solve discrete optimization problems by sampling bitstrings using a parameterized quantum circuit. The circuit parameters (angles) are optimized in the way that minimizes the cost Hamiltonian expectation value. Recently, general statistical properties of QAOA output probability distributions have begun to be studied. In contrast to the conventional approach, we analyse QAOA circuits with random angles. We provide analytical equations for probabilities and the numerical evidence that for unweighted Max-Cut problems on connected graphs such sampling always gives higher entropy of energy distribution than uniform random sampling of bitstrings. We also analyse the probability to obtain the global optima, which appears to be higher on average than for random sampling.

Distributionally robust optimization for dispatching the integrated operation of rural multi-energy system and waste disposal facilities under uncertainties

With the increasing global attention to environmental protection and sustainable development, governments around the world attach great importance to the low-carbon reform of rural energy and waste management. The interaction between waste-to-energy technology (W2E) and rural multi-energy system has emerged as one of the important issues that must be considered. Based on this, this paper combines two W2E technologies, biomass biogas power generation and garbage incineration power generation, with photovoltaic and energy storage systems. A distributionally robust optimization (DRO) model for the waste-disposal-facilities-involved rural multi-energy system (W-RMES) is proposed. An ambiguity set of the photovoltaic output probability distribution is constructed by the Wasserstein metric. The minimum operation cost of W-RMES is solved under the condition that the prediction error of photovoltaic output obeys the worst probability distribution of the ambiguity set. Finally, through simulation verification, the rational utilization of waste can effectively realize the coordination and optimization between the waste disposal and multi-energy supply of electricity, heat, and biogas. Compared with the conventional rural energy system, W-RMES is more economical and eco-friendlier, which is reflected in the reduction of operation cost by >14.87 %, the removal of flue gas production by >12.89 % and the improvement of waste disposal capacity by >21.77 %. Besides, the model also realizes systemic coordination of robustness and economic efficiency.

FTG: Score-based black-box watermarking by fragile trigger generation for deep model integrity verification

Deep neural networks (DNNs) are widely used in real-world applications, thanks to their exceptional performance in image recognition. However, their vulnerability to attacks, such as Trojan and data poison, can compromise the integrity and stability of DNN applications. Therefore, it is crucial to verify the integrity of DNN models to ensure their security. Previous research on model watermarking for integrity detection has encountered the issue of overexposure of model parameters during embedding and extraction of the watermark. To address this problem, we propose a novel score-based black-box DNN fragile watermarking framework called fragile trigger generation (FTG). The FTG framework only requires the prediction probability distribution of the final output of the classifier during the watermarking process. It generates different fragile samples as the trigger, based on the classification prediction probability of the target classifier and a specified prediction probability mask to watermark it. Different prediction probability masks can promote the generation of fragile samples in corresponding distribution types. The whole watermarking process does not affect the performance of the target classifier. When verifying the watermarking information, the FTG only needs to compare the prediction results of the model on the samples with the previous label. As a result, the required model parameter information is reduced, and the FTG only needs a few samples to detect slight modifications in the model. Experimental results demonstrate the effectiveness of our proposed method and show its superiority over related work. The FTG framework provides a robust solution for verifying the integrity of DNN models, and its effectiveness in detecting slight modifications makes it a valuable tool for ensuring the security and stability of DNN applications.

Flow-Based Spatio-Temporal Structured Prediction of Motion Dynamics.

Conditional Normalizing Flows (CNFs) are flexible generative models capable of representing complicated distributions with high dimensionality and large interdimensional correlations, making them appealing for structured output learning. Their effectiveness in modelling multivariates spatio-temporal structured data has yet to be completely investigated. We propose MotionFlow as a novel normalizing flows approach that autoregressively conditions the output distributions on the spatio-temporal input features. It combines deterministic and stochastic representations with CNFs to create a probabilistic neural generative approach that can model the variability seen in high-dimensional structured spatio-temporal data. We specifically propose to use conditional priors to factorize the latent space for the time dependent modeling. We also exploit the use of masked convolutions as autoregressive conditionals in CNFs. As a result, our method is able to define arbitrarily expressive output probability distributions under temporal dynamics in multivariate prediction tasks. We apply our method to different tasks, including trajectory prediction, motion prediction, time series forecasting, and binary segmentation, and demonstrate that our model is able to leverage normalizing flows to learn complicated time dependent conditional distributions.

Double Consistency Regularization for Transformer Networks

The large-scale and deep-layer deep neural network based on the Transformer model is very powerful in sequence tasks, but it is prone to overfitting for small-scale training data. Moreover, the prediction result of the model with a small disturbance input is significantly lower than that without disturbance. In this work, we propose a double consistency regularization (DOCR) method for the end-to-end model structure, which separately constrains the output of the encoder and decoder during the training process to alleviate the above problems. Specifically, on the basis of the cross-entropy loss function, we build the mean model by integrating the model parameters of the previous rounds and measure the consistency between the models by calculating the KL divergence between the features of the encoder output and the probability distribution of the decoder output of the mean model and the base model so as to impose regularization constraints on the solution space of the model. We conducted extensive experiments on machine translation tasks, and the results show that the BLEU score increased by 2.60 on average, demonstrating the effectiveness of DOCR in improving model performance and its complementary impacts with other regularization techniques.

A Hybrid Probabilistic Risk Analytical Approach to Ship Pilotage Risk Resonance with FRAM

Collision risk in ship pilotage process has complex characteristics that are dynamic, uncertain, and emergent. To reveal collision risk resonance during ship pilotage process, a hybrid probabilistic risk analysis approach is proposed, which integrates the Functional Resonance Analysis Method (FRAM), Dempster–Shafer (D–S) evidence theory, and Monte Carlo (MC) simulation. First, FRAM is used to qualitatively describe the coupling relationship and operation mechanism among the functions of the pilotage operation system. Then, the D–S evidence theory is used to determine the probability distribution of the function output in the specified pilotage scenario after quantitatively expressing the function variability, coupling effect, and the influence of operation conditions through rating scales. Finally, MC simulation is used to calculate the aggregated coupling variability between functions, and the critical couplings and risk resonance paths under different scenarios are identified by setting the threshold and confidence level. The results show that ship collision risk transmission is caused by function resonance in the pilotage system, and the function resonance paths vary with pilotage scenarios. The critical coupling ‘F2-F7(I)’ emerges as a consistent factor in both scenarios, emphasizing the significance of maintaining a proper lookout. The hybrid probabilistic risk analytical approach to ship pilotage risk resonance with FRAM can be a useful method for analysing the causative mechanism of ship operational risk.

Sampling‐Based Methods for Uncertainty Propagation in Flood Modeling Under Multiple Uncertain Inputs: Finding Out the Most Efficient Choice

AbstractIn probabilistic flood modeling, uncertainty manifests in frequency of occurrence, or histograms, for quantities of interest, including the Flood Extent and hazard rating (HR). Such modeling at the field‐scale requires the identification of a more efficient alternative to the Standard Monte Carlo (SMC) method that can reproduce comparable output probability distributions with a relatively reduced sample size, including detailed histograms of quantities of interest. Latin hypercube sampling (LHS) is the most evaluated alternative for fluvial floods but yields no considerable sample size reduction. Potentially better alternatives include adaptive stratified sampling (ASS), Quasi Monte Carlo (QMC) and Haar‐wavelet expansion (HWE), which are yet unevaluated for probabilistic flood modeling. To fulfill this gap, LHS, ASS, QMC, and HWE are compared to quantify sample size reduction to reproduce output detailed histograms—for Flood Extent, and average and maximum HR—while keeping the difference below 10% to the reference SMC prediction. The comparison is done for two test cases with two (i.e., inflow discharge and Manning's coefficient) and three (i.e., further including the ground elevation) input random variables, and a real case with five input random variables. With two input random variables, all four alternatives yield sample size reductions, with QMC and HWE considerably outperforming the others; with three and more input random variables, HWE becomes inflexible and LHS underperforms. Still, QMC is a better choice than ASS to boost sample size reduction for the real case and shall be preferred in probabilistic flood modeling. Accompanying research codes are openly available online.

Online Probabilistic Static Security Assessment for Power Systems Considering High Renewable Penetration

With the rapid development of renewable energy, a large number of renewable energy stations are connected to the power system, which leads to a decrease in the inertia of the power system and an increase in safety risks suffered. Thus, online static security assessment (SSA) is increasingly necessary. However, because of the uncertainty of renewable energy, it is not feasible to check all possible scenarios in online SSA. To reduce the number of calculations and achieve SSA in a short time, a new online SSA method based on scenario clustering for future ultra-short-term security assessment is proposed in this paper. In the offline stage, a key scenario set is constructed by Markov Chain Monte Carlo and K-means with historical data. In the online application, the initial probability distribution of renewable energy outputs is calculated by joint distribution with the output of the previous interval and corrected by weather data. Then load flow calculation with N-1 criteria is executed, and the probability for the safe operation of the system is calculated. The effectiveness of the proposed online SSA scheme has been verified in the IEEE-300 system, where one of the generators is replaced by a renewable energy station.

Dynamic conditioning of two particle discrete-time quantum walks

In real photonic quantum systems losses are an unavoidable factor limiting the scalability to many modes and particles, restraining their application in fields as quantum information and communication. For this reason, a considerable amount of engineering effort has been taken in order to improve the quality of particle sources and system components. At the same time, data analysis and collection methods based on post-selection have been used to mitigate the effect of particle losses. This has allowed for investigating experimentally multi-particle evolutions where the observer lacks knowledge about the system's intermediate propagation states. Nonetheless, the fundamental question how losses affect the behaviour of the surviving subset of a multi-particle system has not been investigated so far. For this reason, here we study the impact of particle losses in a quantum walk of two photons reconstructing the output probability distributions for one photon conditioned on the loss of the other in a known mode and temporal step of our evolution network. We present the underlying theoretical scheme that we have devised in order to model controlled particle losses, we describe an experimental platform capable of implementing our theory in a time multiplexing encoding. In the end we show how localized particle losses change the output distributions without altering their asymptotic spreading properties. Finally we devise a quantum civilization problem, a two walker generalisation of single particle recurrence processes.

Econometric Modeling of Input-Driven Output Risk through a Versatile CES Production Function

The conventional functional form of the Constant-Elasticity-of-Substitution (CES) production function is a general production function nesting a number of other forms of production functions. Examples of such functions include Leontief, Cobb–Douglas, and linear production functions. Nevertheless, the conventional form of the CES production specification is still restrictive in multiple aspects. One example is the fact that the marginal effect of increasing input use always has to be to increase the variability of output quantity by the conventional construction of this function. This paper proposes a generalized variant of the CES production function that allows for various input effects on the probability distribution of output. Failure to allow for this possible input–output risk structure is indeed one of the limitations of the conventional form of the CES production function. This limitation may result in false inferences about input-driven output risk. In light of this, the present paper proposes a solution to this problem. First, it is shown that the familiar CES formulation suffers from very restrictive structural assumptions regarding risk considerations, and that such restrictions may lead to biased and inefficient estimates of production quantity and production risk. Following the general theme of Just and Pope’s approach, a CES-based production-function specification that overcomes this shortcoming of the original CES production function is introduced, and a three-stage Nonlinear Least-Squares (NLS) estimation procedure for the estimation of the proposed functional form is presented. To illustrate the proposed approaches in this paper, two empirical applications in irrigation and fertilizer response using the famous Hexem–Heady experimental dataset are provided. Finally, implications for modeling input-driven production risks are discussed.

Distribution Matching for Dimming Control in Visible-Light Region-of-Interest Signaling

We propose a two-level dimmer based on binary distribution matching where a low-rate signal controls the output probability distribution of a high-rate bit sequence, which can be used in region-of-interest (RoI) signaling applications. To reduce the rate loss of the dimmer, we propose the extended multiset-partition distribution matching (EMPDM) algorithm with a novel binary-tree-structure implementation. In addition, we introduce 4p-EMPDM, a compact version of EMPDM, which has a typical composition (TC) and four leading composition pairs (CPs). The codebook of 4p-EMPDM includes only run-length-aware codewords, which reduces the maximum run-length of the transmitted bit sequence by 4.27 times. Hence, it guarantees flicker mitigation for visible-light RoI signaling systems at 6 kHz without using any run-length limited code (non-RLL). Using experimental data collected from the low-rate RoI signaling prototype, we introduced a threshold range where both intensity and area information of the received training symbols can be exploited to optimize the shaping ratios of the 4p-EMPDM dimmer. Because of the non-RLL feature, the proposed system can support soft-decision forward-error-correction (FEC) decoding to improve reliability. Our system outperforms related systems based on hybrid modulation schemes in terms of spectral efficiency, bit rate, and minimum required optical clock rate.

Probabilistic power curve estimation based on meteorological factors and density LSTM

Wind power curve describes the relationship between wind speed and wind power output, which is useful for wind farm design and wind turbine condition monitoring. However, most research on power curves modeling neglected the effect of historical meteorological factors. Also, the research on probabilistic power curve is still limited. Therefore, this paper proposes a novel probabilistic power curve modeling approach. The effect of both current and historical meteorological variables, such as wind speed, wind direction, ambient temperature and turbulence intensity, on the power curve is studied by binning method and kernel density estimation. Next, instead of forecasting the wind power or its quantiles, a new probabilistic power curve model, Density LSTM with Negative Log-Likelihood Loss, is proposed to forecast the parameters of the probability density function of wind power directly. Finally, different probability distribution of wind power output is studied. Gamma, Laplace and Weibull distributions prove to be more suitable than Gaussian distribution. Based on the datasets of inland and offshore wind farms, it is verified that adding useful historical meteorological variables can improve the forecasting performance of the test dataset. Besides, the proposed probabilistic power curve can effectively improve the prediction performance in the probabilistic prediction task and can improve the prediction precision of the annual energy production.

Modelling parametric uncertainty in large-scale stratigraphic simulations

We combine forward stratigraphic models with a suite of uncertainty quantification and stochastic model calibration algorithms for the characterization of sedimentary successions in large scale systems. The analysis focuses on the information value provided by a probabilistic approach in the modelling of large-scale sedimentary basins. Stratigraphic forward models (SFMs) require a large number of input parameters usually affected by uncertainty. Thus, model calibration requires considerable time both in terms of human and computational resources, an issue currently limiting the applications of SFMs. Our work tackles this issue through the combination of sensitivity analysis, model reduction techniques and machine learning-based optimization algorithms. We first employ a two-step parameter screening procedure to identify relevant parameters and their assumed probability distributions. After selecting a restricted set of important parameters these are calibrated against available information, i.e., the depth of interpreted stratigraphic surfaces. Because of the large costs associated with SFM simulations, probability distributions of model parameters and outputs are obtained through a data driven reduced complexity model. Our study demonstrates the numerical approaches by considering a portion of the Porcupine Basin, Ireland. Results of the analysis are postprocessed to assess (i) the uncertainty and practical identifiability of model parameters given a set of observations, (ii) spatial distribution of lithologies. We analyse here the occurrences of sand bodies pinching against the continental slope, these systems likely resulting from gravity driven processes in deep sea environment.