General Loss Function Research Articles

Beyond Neyman–Pearson: E-values enable hypothesis testing with a data-driven alpha

A standard practice in statistical hypothesis testing is to mention the P-value alongside the accept/reject decision. We show the advantages of mentioning an e-value instead. With P-values, it is not clear how to use an extreme observation (e.g. [Formula: see text]) for getting better frequentist decisions. With e-values it is straightforward, since they provide Type-I risk control in a generalized Neyman-Pearson setting with the decision task (a general loss function) determined post hoc, after observation of the data-thereby providing a handle on "roving [Formula: see text]'s." When Type-II risks are taken into consideration, the only admissible decision rules in the post hoc setting turn out to be e-value-based. Similarly, if the loss incurred when specifying a faulty confidence interval is not fixed in advance, standard confidence intervals and distributions may fail, whereas e-confidence sets and e-posteriors still provide valid risk guarantees. Sufficiently powerful e-values have by now been developed for a range of classical testing problems. We discuss the main challenges for wider development and deployment.

Tomographic projection optimization for volumetric additive manufacturing with general band constraint Lp-norm minimization

Tomographic volumetric additive manufacturing is a rapidly growing fabrication technology that enables rapid production of 3D objects through a single build step. In this process, the design of projections directly impacts geometric resolution, material properties, and manufacturing yield of the final printed part. Herein, we identify the hidden equivalent operations of three major existing projection optimization schemes and reformulate them into a general loss function where the optimization behavior can be systematically studied, and unique capabilities of the individual schemes can coalesce. The loss function formulation proposed in this study unified the optimization for binary and greyscale targets and generalized problem relaxation strategies with local tolerancing and weighting. Additionally, this formulation offers control on error sparsity and consistent dose response mapping throughout initialization, optimization, and evaluation. A parameter-sweep analysis in this study guides users in tuning optimization parameters for application-specific goals.

Integration of prediction and optimization for smart stock portfolio selection

Machine learning (ML) algorithms pose significant challenges in predicting unknown parameters for optimization models in decision-making scenarios. Conventionally, prediction models are optimized independently in decision-making processes, whereas ML algorithms primarily focus on minimizing prediction errors, neglecting the role of decision-making in downstream optimization tasks. The pursuit of high prediction accuracy may not always align with the goal of reducing decision errors. The idea of reducing decision errors has been proposed to address this limitation. This paper introduces an optimization process that integrates predictive regression models within a mean–variance optimization setting. This innovative technique introduces a general loss function to capture decision errors. Consequently, the predictive model not only focuses on forecasting unknown optimization parameters but also emphasizes the predicted values that minimize decision errors. This approach prioritizes decision accuracy over the potential accuracy of unknown parameter prediction. In contrast to traditional ML approaches that minimize standard loss functions such as mean squared error, our proposed model seeks to minimize the objective value derived directly from the decision-making problem. Furthermore, this strategy is validated by developing an optimization-based regression tree model for predicting stock returns and reducing decision errors. Empirical evaluations of our framework reveal its superiority over conventional regression tree methods, demonstrating enhanced decision quality. The computational experiments are conducted on a stock market dataset to compare the effectiveness of the proposed framework with the conventional regression tree-based approach. Remarkably, the results confirm the strengths inherent in this holistic approach.

A Novel Aircraft Trajectory Generation Method Embedded with Data Mining

Data mining has achieved great success in air traffic management as a technology for learning knowledge from historical data that benefits people. However, data mining can rarely be embedded into the trajectory optimization process since regular optimization algorithms cannot utilize the functional and implicit knowledge extracted from historical data in a general paradigm. To tackle this issue, this research proposes a novel data mining-based trajectory generation method that is compatible with existing optimization algorithms. Firstly, the proposed method generates trajectories by combining various maneuvers learned from operation data instead of reconstructing trajectories with generative models. In such a manner, data mining-based trajectory optimization can be achieved by solving a combinatorial optimization problem. Secondly, the proposed method introduces a majorization–minimization-based adversarial training paradigm to train the generation model with more general loss functions, including non-differentiable flight performance constraints. A case study on Guangzhou Baiyun International Airport was conducted to validate the proposed method. The results illustrate that the trajectory generation model can generate trajectories with high fidelity, diversity, and flyability.

Group inference of high-dimensional single-index models

For the supervised and semi-supervised settings, a group inference method is proposed for regression parameters in high-dimensional semi-parametric single-index models with an unknown random link function. The inference procedure is based on least squares, which can be extended to other general convex loss functions. The proposed test statistics are weighted quadratic forms of the regression parameter estimates, in which the weight could be a non-random matrix or the sample covariance matrix of the covariates. The proposed method could detect dense but weak signals and deal with high correlation of covariates inside the group. A ‘contaminated test statistic’ is established in the semi-supervised regime to decrease the variance. The asymptotic properties of the resulting estimators are established. The finite-sample behaviour of the proposed method is evaluated through extensive simulation studies. Applications to two genomic datasets are provided.

BAYES ESTIMATION OF EXPONENTIALLY DISTRIBUTED SURVIVAL DATA UNDER SYMMETRIC AND ASYMMETRIC LOSS FUNCTIONS

In a research study, population data are often not available, so the population parameter is unknown. Meanwhile, knowledge about the population parameter is needed to know the characteristics of the studied population. Therefore, it is needed to estimate the parameter of the population which can be estimated by sample data. There are several methods of parameter estimation which are generally classified into classical and Bayesian method. This research studied the Bayesian parameter estimation method to determine the parameters of the exponentially distributed survival data associated with the reliability measure of the estimates under symmetric and asymmetric loss functions for complete sample data in a closed form. The symmetric loss functions used in this research are Squared Error Loss Function (SELF) and Minimum Expected Loss Function (MELF). The asymmetric loss functions used are the General Entropy Loss Function (GELF) and Linex Loss Function (LLF). Performance of some loss functions used in this research are then compared through numerical simulation to select the best loss function in determining the parameter estimation of the exponentially distributed survival data. We also studied which loss function is best for underestimation and overestimation modeling. Based on simulation results, the Bayes estimates using MELF is the best method to estimate population parameters of the exponentially distributed survival data for the overestimation modeling, while LLF is the best for the underestimation modeling. We provided direct application in a case study of fluorescence lamp survival data. The results show that the best method to estimate the parameter of the standard fluorescence life data is using LLF for underestimation with and MELF for overestimation with .

Worst-case Conditional Value at Risk for asset liability management: A framework for general loss functions

Asset–liability management (ALM) is a challenging task faced by pension funds due to the uncertain nature of future asset returns, employees’ wages, and interest rates. To address this challenge, this paper presents a new mathematical model that uses a Worst-case Conditional Value-at-Risk (WCVaR) constraint to ensure that, with high probability, the funding ratio remains above a regulator-mandated threshold under the worst-case density function that plausibly explains historical sample data. A tractable reformulation of this WCVaR constraint is developed based on the definition of the Worst-case Lower Partial Moment (WLPM) for a general loss function. Additionally, a data-driven moment-based ambiguity set is constructed to capture uncertainty in the moments of the density functions of random variables in the ALM problem. The proposed approach is evaluated using real-world data from the Canada Pension Plan (CPP) and is shown to outperform classical ALM models, based on either CVaR or WCVaR with fixed moments, on out-of-sample data. The proposed framework for handling correlated uncertainty using WCVaR with nonlinear loss functions can be used in other application areas.

Mixture correntropy-based robust distance metric learning for classification

Metric learning is a branch of machine learning that aims to learn from the given training data a valid distance metric, with which the similarity between samples can be more effectively evaluated for classification. Metric learning has attracted significant attention, and a large number of models have been proposed in the past few years. However, the traditional methods adopt hinge loss which easily leads to noise sensitivity and instability. In this paper, to improve the robustness performance, we develop a mixture correntropy criterion where two Laplacian kernel functions are combined as the kernel function and induce a more general nonconvex robust loss function by the mixture correntropy. The properties related to the loss function are analysed and presented. The induced loss amalgamates the superiors of the state-of-the-art robust loss functions and is more effective. With this induced loss, we establish a robust metric learning model (called MCML) and design an effective iterative algorithm to optimize the nonconvex challenging problem. The computational complexity and convergence of algorithm are discussed in theory. Furthermore, a boosting version of MCML (BMCML) is derived, where the low-rank basis learning is jointly optimized with the metric to better uncover the data structure. Finally, extensive experiments are conducted on artificial datasets, UCI benchmark datasets and image datasets. The experimental results verify the robustness and effectiveness of the proposed methods.

Realizable Learning is All You Need

The equivalence of realizable and agnostic learnability is a fundamental phenomenon in learning theory. With variants ranging from classical settings like PAC learning and regression to recent trends such as adversarially robust learning, it's surprising that we still lack a unified theory; traditional proofs of the equivalence tend to be disparate, and rely on strong model-specific assumptions like uniform convergence and sample compression. In this work, we give the first model-independent framework explaining the equivalence of realizable and agnostic learnability: a three-line blackbox reduction that simplifies, unifies, and extends our understanding across a wide variety of settings. This includes models with no known characterization of learnability such as learning with arbitrary distributional assumptions and more general loss functions, as well as a host of other popular settings such as robust learning, partial learning, fair learning, and the statistical query model. More generally, we argue that the equivalence of realizable and agnostic learning is actually a special case of a broader phenomenon we call property generalization: any desirable property of a learning algorithm (e.g. noise tolerance, privacy, stability) that can be satisfied over finite hypothesis classes extends (possibly in some variation) to any learnable hypothesis class.

The E-Bayesian and hierarchical Bayesian estimations for the reliability analysis of Kumaraswamy generalized distribution based on upper record values

This paper investigates the E-Bayesian and hierarchical Bayesian estimations of shape parameter and reliability function of Kumaraswamy generalized distribution based on upper record values. The classical estimation method is utilized to deduce the maximum likelihood estimation of unknown parameter and reliability function. Bayesian estimates are derived by using conjugate Gamma prior distributions under quadratic and general entropy loss functions. Furthermore, assuming that hyper-hyperparameters obey three prior distributions, the E-Bayesian estimates of unknown parameters and reliability functions are obtained. The hierarchical Bayesian estimates are obtained by using hierarchical prior distributions. We also explore some characteristics and size relationships of E-Bayesian and hierarchical Bayesian estimations. The performance of E-Bayesian, Hierarchical Bayesian, Bayesian, and maximum likelihood estimations is compared based on the minimum mean square error criterion. Finally, the proposed estimation methods are applied to evaluate the reliability of the specimen under ultrasonic fatigue testing, and the results align with their structures and profiles.

Theoretical analysis of divide-and-conquer ERM: From the perspective of multi-view

Theoretical analysis of the divide-and-conquer based distributed learning with least square loss in the reproducing kernel Hilbert space (RKHS) have recently been explored within the framework of learning theory. However, the studies on learning theory for general loss functions and hypothesis spaces remain limited. In order to comprehend the properties and behaviors of distributed ERM, we introduce multi-view learning to explore it at a more profound level. In this work, we adopt an innovative perspective by examining the distributed ERM from multi-view, and study the risk performance of distributed ERM for general loss functions and hypothesis spaces. The main theoretical results are two-fold. First, we derive two tight risk bounds under certain basic assumptions on the hypothesis space, as well as the smoothness, Lipschitz continuity, strong convexity of the loss function. Second, we further develop two more general risk bounds for distributed ERM without the restriction of strong convexity. The present work not only bridges the knowledge deficit in learning theory of distributed ERM for general loss functions and hypothesis spaces, from multi-view sight, it also shows that under certain conditions the number of views can guarantee the performance.

Wind Power Curve Modeling With Large-Scale Generalized Kernel-Based Regression Model

Accurate wind power curves (WPCs) are crucial for wind energy development and utilization, e.g., wind power forecasting and wind turbine condition monitoring. In the era of big data, large-scale datasets make the training of power curve models inefficient, especially for kernel-based models. Furthermore, most models do not take into account the error characteristics of WPC modeling. In this study, a large-scale generalized kernel-based regression model is proposed to solve the above problem. First, a generalized loss function, which can model both symmetric and asymmetric error distributions, is designed for model training. Then, the Nyström technique is employed to get the approximate kernel matrix, based on which an eigenvalue-based kernel regression framework is constructed. Next, a large-scale generalized kernel-based regression model is developed with model parameters tuned using the alternating direction method of multipliers. Before WPC modeling, a three-step data processing method based on isolation forest is designed to process missing data, irrational data, and outliers in the collected data. The WPC modeling results on four large-scale wind datasets demonstrate that the proposed model generates accurate WPCs with high efficiency. Furthermore, the effect of turbulence intensity on WPC modeling and the effectiveness of LSGKRM with multivariate inputs are also verified.

Testing the optimality of USDA's WASDE forecasts under unknown loss

AbstractMotivated by the long‐lasting debate on whether the United States Department of Agriculture's (USDA's) World Agricultural Supply and Demand Estimates (WASDE) forecasts are optimal, we employ an unknown loss method for ex post evaluation which assumes that the USDA forecasters' loss function is unknown. We conduct optimality tests of the WASDE forecasts for corn, soybeans, and wheat published during 1988–2019. Our results suggest that USDA forecasters generally realize optimality during the data‐generating process. Our findings are consistent with previous studies when narrowing down the more general unknown loss function to the symmetric or asymmetric loss function which assumes a specific shape for the loss function. This study provides implications based on the unknown loss function that the USDA forecasters can boost their information set as an alternative way to improve the WASDE forecasts. [EconLit Citations: D84, E37, Q13, Q14].

Designing efficient Bayesian sampling plans for exponential distributions based on samples under a (n, t) simple step-stress test

This article studies a method about how to design an efficient Bayesian sampling plan based on samples collected through a (n, t)-sampling scheme with a simple step-stress accelerated life test (ALT). For brevity, such a step-stress accelerated life test is called (n, t)-SSALT. A Bayesian sampling plan (BSP) derived through the (n, t)-SSALT is called BSPA. First, we derive the BSPA with data collected through a (n, t)-SSALT for a general loss function. Given gamma and Jeffreys prior distributions, an explicit expression of the Bayes decision function under a certain loss function is derived. By applying a curtailment procedure to the preceding Bayes decision function, an on-line new Bayes decision function and Bayesian sampling plan, called an efficient Bayesian sampling plan (EBSPA), are constructed. It is shown that the Bayes risk of EBSPA is less than or equal to that of BSPA. This indicates that the EBSPA is more efficient than the BSPA. Comparisons among some BSPAs and the proposed EBSPA are given. The numerical results indicate that, in terms of Bayes risks, the EBSPA significantly outperforms the BSPA.

Recurrent Neural Network Training With Convex Loss and Regularization Functions by Extended Kalman Filtering

This paper investigates the use of extended Kalman filtering to train recurrent neural networks with rather general convex loss functions and regularization terms on the network parameters, including <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\ell _{1}$</tex-math></inline-formula> -regularization. We show that the learning method is competitive with respect to stochastic gradient descent in a nonlinear system identification benchmark and in training a linear system with binary outputs. We also explore the use of the algorithm in data-driven nonlinear model predictive control and its relation with disturbance models for offset-free closed-loop tracking.

Resource frugal optimizer for quantum machine learning

Abstract Quantum-enhanced data science, also known as quantum machine learning (QML), is of growing interest as an application of near-term quantum computers. Variational QML algorithms have the potential to solve practical problems on real hardware, particularly when involving quantum data. However, training these algorithms can be challenging and calls for tailored optimization procedures. Specifically, QML applications can require a large shot-count overhead due to the large datasets involved. In this work, we advocate for simultaneous random sampling over both the dataset as well as the measurement operators that define the loss function. We consider a highly general loss function that encompasses many QML applications, and we show how to construct an unbiased estimator of its gradient. This allows us to propose a shot-frugal gradient descent optimizer called Refoqus (REsource Frugal Optimizer for QUantum Stochastic gradient descent). Our numerics indicate that Refoqus can save several orders of magnitude in shot cost, even relative to optimizers that sample over measurement operators alone.

A distributed multi-area power system state estimation method based on generalized loss function

For power system state estimation, the measurement noise is usually assumed to follow the Gaussian distribution, and the widely used estimator is the weighted least squares (WLS). However, the Gaussian distribution assumption is not always true, and the performance of WLS becomes bad when the measurement noise is non-Gaussian. In this paper, a new distributed state estimation (SE) method is proposed for multi-area power systems. The proposed distributed method is based on the generalized loss function so that it can reduce the influence of non-Gaussian noise and bad data. Further, thanks to the matrix-splitting technology, the proposed distributed method can be implemented in a distributed way so that the computation time in each local area can be reduced. The simulation results carried out in the IEEE 30-bus and 118-bus systems verify the robustness and effectiveness of the proposed distributed SE method.

Optimal sampling and statistical inferences for Kumaraswamy distribution under progressive Type-II censoring schemes

In this paper, we study non-Bayesian and Bayesian estimation of parameters for the Kumaraswamy distribution based on progressive Type-II censoring. First, the maximum likelihood estimates and maximum product spacings are derived. In addition, we derive the asymptotic distribution of the parameters and the asymptotic confidence intervals. Second, Bayesian estimators under symmetric and asymmetric loss functions (Squared error, linear exponential, and general entropy loss functions) are also obtained. The Lindley approximation and the Markov chain Monte Carlo method are used to derive the Bayesian estimates. Furthermore, we derive the highest posterior density credible intervals of the parameters. We further present an optimal progressive censoring scheme among different competing censoring scheme using three optimality criteria. Simulation studies are conducted to evaluate the performance of the point and interval estimators. Finally, one application of real data sets is provided to illustrate the proposed procedures.

Bayesian Approach for estimating the unknown Scale parameter of Erlang Distribution Based on General Entropy Loss Function

We are used Bayes estimators for unknown scale parameter when shape Parameter is known of Erlang distribution. Assuming different informative priors for unknown scale parameter. We derived The posterior density with posterior mean and posterior variance using different informative priors for unknown scale parameter which are the inverse exponential distribution, the inverse chi-square distribution, the inverse Gamma distribution, and the standard Levy distribution as prior. And we derived Bayes estimators based on the general entropy loss function (GELF) is used the Simulation method to obtain the results. we generated different cases for the parameters of the Erlang model, for different sample sizes. The estimates have been compared in terms of their mean-squared error (MSE). We concluded that the best estimators of the scale parameterof the Erlang distribution, based on GELF for the shape parameter (c=1,2,3) under inverse gamma prior with for all samples sizes(n) where the true cases of the Erlang model are and according to the smallest values of MSE

Tensor Stein-rules in a generalized tensor regression model

In this paper, we consider an estimation problem in a generalized tensor regression model with multi-mode covariates. We generalize the main results in recent literature in four ways. First, we weaken assumptions underlying the main results of the previous works. In particular, the dependence structure of the error and covariates are as weak as an L2− mixingale array, and the error term does not need to be uncorrelated with regressors. Second, we consider a more general constraint than the one considered in previous works. Third, we establish the asymptotic properties of the tensor estimators. Specifically, we derive the joint asymptotic distribution of the unrestricted tensor estimator (UE) and restricted tensor estimator (RE). Fourth, we propose a class of shrinkage-type estimators in the context of tensor regression, and under a general loss function, we derive sufficient conditions for which the shrinkage estimators dominate the UE. In addition to these interesting contributions, we derive a kind of functional central limit theorem for mixingale vector-valued and we establish some identities which are useful in studying the risk dominance of shrinkage-type tensor estimators. Finally, to illustrate the application of the proposed methods, we corroborate the results by some simulation studies of binary, Normal and Poisson data and we analyze a multi-relational network and neuro-imaging datasets.