General Loss Function Research Articles

Convolutional Neural Networks for Very Low-Dimensional LPV Approximations of Incompressible Navier-Stokes Equations

The control of general nonlinear systems is a challenging task in particular for large-scale models as they occur in the semi-discretization of partial differential equations (PDEs) of, say, fluid flow. In order to employ powerful methods from linear numerical algebra and linear control theory, one may embed the nonlinear system in the class of linear parameter varying (LPV) systems. In this work, we show how convolutional neural networks can be used to design LPV approximations of incompressible Navier-Stokes equations. In view of a possibly low-dimensional approximation of the parametrization, we discuss the use of deep neural networks (DNNs) in a semi-discrete PDE context and compare their performance to an approach based on proper orthogonal decomposition (POD). For a streamlined training of DNNs directed to the PDEs in aFinite Element(FEM) framework, we also discuss algorithmical details of implementing the proper norms in general loss functions.

Analysis with Applications of the Generalized Type-II Progressive Hybrid Censoring Sample from Burr Type-XII Model

In this article, based on the generalized Type-II progressive hybrid censoring sample from the Burr Type-XII distribution, maximum likelihood and Bayesian inference are discussed. Point and interval estimates of unknown parameters, reliability, and hazard functions are developed. We employed several loss functions, such as squared error, LINEX, and general entropy, as symmetric and asymmetric loss functions and various prior distributions as informative and non-informative priors for Bayesian inference of unknown parameters. Under a generalized Type-II progressive hybrid censoring sample, we also propose a Bayesian one-sample prediction for unobserved failures. We conduct simulation study using the MCMC algorithm for the Bayesian approach based on several prior distributions. Finally, we apply the results of the theoretical research to real data.

A General Loss-Based Nonnegative Matrix Factorization for Hyperspectral Unmixing

Nonnegative matrix factorization (NMF) is a widely used hyperspectral unmixing model which decomposes a known hyperspectral data matrix into two unknown matrices, i.e., endmember matrix and abundance matrix. Due to the use of least-squares loss, the NMF model is usually sensitive to noise or outliers. To improve its robustness, we introduce a general robust loss function to replace the traditional least-squares loss and propose a general loss-based NMF (GLNMF) model for hyperspectral unmixing in this letter. The general loss function is a superset of many common robust loss functions and is suitable for handling different types of noise. Experimental results on simulated and real hyperspectral data sets demonstrate that our GLNMF model is more accurate and robust than existing NMF methods.

A Stable High-Order Tuner for General Convex Functions

Iterative gradient-based algorithms have been increasingly applied for the training of a broad variety of machine learning models including large neural-nets. In particular, momentum-based methods, with accelerated learning guarantees, have received a lot of attention due to their provable guarantees of fast learning in certain classes of problems and multiple algorithms have been derived. However, properties for these methods hold only for constant regressors. When time-varying regressors occur, which is commonplace in dynamic systems, many of these momentum-based methods cannot guarantee stability. Recently, a new High-order Tuner (HT) was developed for linear regression problems and shown to have 1) stability and asymptotic convergence for time-varying regressors and 2) non-asymptotic accelerated learning guarantees for constant regressors. In this letter, we extend and discuss the results of this same HT for general convex loss functions. Through the exploitation of convexity and smoothness definitions, we establish similar stability and asymptotic convergence guarantees. Finally, we provide numerical simulations supporting the satisfactory behavior of the HT algorithm as well as an accelerated learning property.

Unified Optimal Model Averaging With a General Loss Function Based on Cross-Validation

Unified Optimal Model Averaging With a General Loss Function Based on Cross-Validation

Some results on quadratic credibility premium using the balanced loss function

PurposeThis paper generalizes the quadratic framework introduced by Le Courtois (2016) and Sumpf (2018), to obtain new credibility premiums in the balanced case, i.e. under the balanced squared error loss function. More precisely, the authors construct a quadratic credibility framework under the net quadratic loss function where premiums are estimated based on the values of past observations and of past squared observations under the parametric and the non-parametric approaches, this framework is useful for the practitioner who wants to explicitly take into account higher order (cross) moments of past data.Design/methodology/approachIn the actuarial field, credibility theory is an empirical model used to calculate the premium. One of the crucial tasks of the actuary in the insurance company is to design a tariff structure that will fairly distribute the burden of claims among insureds. In this work, the authors use the weighted balanced loss function (WBLF, henceforth) to obtain new credibility premiums, and WBLF is a generalized loss function introduced by Zellner (1994) (see Gupta and Berger (1994), pp. 371-390) which appears also in Dey et al. (1999) and Farsipour and Asgharzadhe (2004).FindingsThe authors declare that there is no conflict of interest and the funding information is not applicable.Research limitations/implicationsThis work is motivated by the following: quadratic credibility premium under the balanced loss function is useful for the practitioner who wants to explicitly take into account higher order (cross) moments and new effects such as the clustering effect to finding a premium more credible and more precise, which arranges both parts: the insurer and the insured. Also, it is easy to apply for parametric and non-parametric approaches. In addition, the formulas of the parametric (Poisson–gamma case) and the non-parametric approach are simple in form and may be used to find a more flexible premium in many special cases. On the other hand, this work neglects the semi-parametric approach because it is rarely used by practitioners.Practical implicationsThere are several examples of actuarial science (credibility).Originality/valueIn this paper, the authors used the WBLF and a quadratic adjustment to obtain new credibility premiums. More precisely, the authors construct a quadratic credibility framework under the net quadratic loss function where premiums are estimated based on the values of past observations and of past squared observations under the parametric and the non-parametric approaches, this framework is useful for the practitioner who wants to explicitly take into account higher order (cross) moments of past data.

Order restricted classical and Bayesian inference of a multiple step-stress model from two-parameter Rayleigh distribution under Type I censoring

This article considers a multiple step-stress model under two-parameter Rayleigh distribution with Type I censoring. Given that the lifetime of the experiment unit gradually decreases as the stress level grows step by step, the parameters of lifetime distribution with the increasing level have a natural order restriction. The reparametrization is applied to deal with this order limitation. Based on the proportional hazard model, the continuous cumulative distribution function and the corresponding likelihood function are inferred. The procedures of computing order restricted maximum likelihood estimations whether at least one failure exists in each stress level or not are introduced. Through transformation, the asymptotic confidence intervals of the original parameters are calculated based on the observed Fisher information matrix. Furthermore, taking the square error loss function, the Linex loss function, and the general entropy loss function into account, Bayesian estimations are discussed. With the importance sampling, the unknown parameters’ estimates are obtained and the associated highest posterior density credible intervals are built up. With simulations under different circumstances, the effectiveness of each method proposed is demonstrated. Finally, an analysis of a real dataset is provided.

Robust linear classification from limited training data

We consider the problem of linear classification under general loss functions in the limited-data setting. Overfitting is a common problem here. The standard approaches to prevent overfitting are dimensionality reduction and regularization. But dimensionality reduction loses information, while regularization requires the user to choose a norm, or a prior, or a distance metric. We propose an algorithm called RoLin that needs no user choice and applies to a large class of loss functions. RoLin combines “reliable” information from the top principal components with a robust optimization to extract any useful information from “unreliable” subspaces. It also includes a new robust cross-validation that is better than existing cross-validation methods in the limited-data setting. Experiments on 25 real-world datasets and three standard loss functions show that RoLin broadly outperforms both dimensionality reduction and regularization. Dimensionality reduction has \(14\%-40\%\) worse test loss on average as compared to RoLin. Against \(L_1\) and \(L_2\) regularization, RoLin can be up to 3x better for logistic loss and 12x better for squared hinge loss. The differences are greatest for small sample sizes, where RoLin achieves the best loss on 2x to 3x more datasets than any competing method. For some datasets, RoLin with 15 training samples is better than the best norm-based regularization with 1500 samples.

On Reliability Estimation of Lomax Distribution under Adaptive Type-I Progressive Hybrid Censoring Scheme

Bayesian estimates involve the selection of hyper-parameters in the prior distribution. To deal with this issue, the empirical Bayesian and E-Bayesian estimates may be used to overcome this problem. The first one uses the maximum likelihood estimate (MLE) procedure to decide the hyper-parameters; while the second one uses the expectation of the Bayesian estimate taken over the joint prior distribution of the hyper-parameters. This study focuses on establishing the E-Bayesian estimates for the Lomax distribution shape parameter functions by utilizing the Gamma prior of the unknown shape parameter along with three distinctive joint priors of Gamma hyper-parameters based on the square error as well as two asymmetric loss functions. These two asymmetric loss functions include a general entropy and LINEX loss functions. To investigate the effect of the hyper-parameters’ selections, mathematical propositions have been derived for the E-Bayesian estimates of the three shape functions that comprise the identity, reliability and hazard rate functions. Monte Carlo simulation has been performed to compare nine E-Bayesian, three empirical Bayesian and Bayesian estimates and MLEs for any aforementioned functions. Additionally, one simulated and two real data sets from industry life test and medical study are applied for the illustrative purpose. Concluding notes are provided at the end.

Autoencoder-Based Unequal Error Protection Codes

We present a novel autoencoder-based approach for designing codes that provide unequal error protection (UEP) capabilities. The proposed approach, based on a generalization of an autoencoder loss function, provides a versatile framework for the design of message-wise and bit-wise UEP codes. Using an associated weight vector, the generalized loss function can be used to trade off error probabilities corresponding to different importance classes and to explore the region of achievable error probabilities. For message-wise UEP, we compare the proposed autoencoder-based UEP codes with a union of random coset codes. For bit-wise UEP, the proposed codes are compared with UEP rateless spinal codes and the superposition of random Gaussian codes. In all cases, the autoencoder-based codes show superior performance while providing design simplicity and flexibility in trading off error protection among different importance classes.

Estimation of smooth functionals in normal models: Bias reduction and asymptotic efficiency

Let X1,…,Xn be i.i.d. random variables sampled from a normal distribution N(μ,Σ) in Rd with unknown parameter θ=(μ,Σ)∈Θ:=Rd×C+d, where C+d is the cone of positively definite covariance operators in Rd. Given a smooth functional f:Θ↦R1, the goal is to estimate f(θ) based on X1,…,Xn. Let Θ(a;d):=Rd×{Σ∈C+d:σ(Σ)⊂[1/a,a]},a≥1, where σ(Σ) is the spectrum of covariance Σ. Let θˆ:=(μˆ,Σˆ), where μˆ is the sample mean and Σˆ is the sample covariance, based on the observations X1,…,Xn. For an arbitrary functional f∈Cs(Θ), s=k+1+ρ,k≥0,ρ∈(0,1], we define a functional fk:Θ↦R such that supθ∈Θ(a;d)‖fk(θˆ)−f(θ)‖L2(Pθ)≲s,β‖f‖Cs(Θ)[(a n∨aβs( d n)s)∧1], where β=1 for k=0 and β>s−1 is arbitrary for k≥1. This error rate is minimax optimal and similar bounds hold for more general loss functions. If d=dn≤nα for some α∈(0,1) and s≥11−α, the rate becomes O(n−1/2). Moreover, for s>11−α, the estimator fk(θˆ) is shown to be asymptotically efficient. The crucial part of the construction of estimator fk(θˆ) is a bias reduction method studied in the paper for more general statistical models than normal.

Adversarial Risk via Optimal Transport and Optimal Couplings

Modern machine learning algorithms perform poorly on adversarially manipulated data. Adversarial risk quantifies the error of classifiers in adversarial settings; adversarial classifiers minimize adversarial risk. In this paper, we analyze adversarial risk and adversarial classifiers from an optimal transport perspective. We show that the optimal adversarial risk for binary classification with 0-1 loss is determined by an optimal transport cost between the probability distributions of the two classes. We develop optimal transport plans (probabilistic couplings) for univariate distributions such as the normal, the uniform, and the triangular distribution. We also derive optimal adversarial classifiers in these settings. Our analysis leads to algorithm-independent fundamental limits on adversarial risk, which we calculate for several real-world datasets. We extend our results to general loss functions under convexity and smoothness assumptions.

Designing Bayesian sampling plans for simple step-stress of accelerated life test on censored data

ABSTRACT This paper studies the problem about how to design a Bayesian sampling plan (BSP) for two exponential distributions linked by the cumulative exposure model through a simple step-stress accelerated life test (ALT). Such a Bayesian sampling plan through the ALT by a simple step-stress procedure is called BSPA. The BSPA with Type-II censoring in a general loss function is derived. Given joint gamma and uniform prior distributions, an explicit Bayes decision function under a certain loss function is derived. Illustrative examples are given to demonstrate how to find the Bayes decision function. A Monte Carlo simulation study is performed for searching the optimal BSPA. Comparison between the proposed BSPA and the conventional BSP is carried out to study the performance of BSPA. The numerical results indicate that the risk reduction of BSPA by applying the accelerated procedure is more significant if the experimental time cost is more expensive.

Least kth-Order and Rényi Generative Adversarial Networks.

We investigate the use of parameterized families of information-theoretic measures to generalize the loss functions of generative adversarial networks (GANs) with the objective of improving performance. A new generator loss function, least kth-order GAN (LkGAN), is introduced, generalizing the least squares GANs (LSGANs) by using a kth-order absolute error distortion measure with k≥1 (which recovers the LSGAN loss function when k=2). It is shown that minimizing this generalized loss function under an (unconstrained) optimal discriminator is equivalent to minimizing the kth-order Pearson-Vajda divergence. Another novel GAN generator loss function is next proposed in terms of Rényi cross-entropy functionals with order α>0, α≠1. It is demonstrated that this Rényi-centric generalized loss function, which provably reduces to the original GAN loss function as α→1, preserves the equilibrium point satisfied by the original GAN based on the Jensen-Rényi divergence, a natural extension of the Jensen-Shannon divergence. Experimental results indicate that the proposed loss functions, applied to the MNIST and CelebA data sets, under both DCGAN and StyleGAN architectures, confer performance benefits by virtue of the extra degrees of freedom provided by the parameters k and α, respectively. More specifically, experiments show improvements with regard to the quality of the generated images as measured by the Fréchet inception distance score and training stability. While it was applied to GANs in this study, the proposed approach is generic and can be used in other applications of information theory to deep learning, for example, the issues of fairness or privacy in artificial intelligence.

Learning with smooth Hinge losses

Due to the non-smoothness of the Hinge loss in SVM, it is difficult to obtain a faster convergence rate with modern optimization algorithms. In this paper, we introduce two smooth Hinge losses ψG(α;σ) and ψM(α;σ) which are infinitely differentiable and converge to the Hinge loss uniformly in α as σ tends to 0. By replacing the Hinge loss with these two smooth Hinge losses, we obtain two smooth support vector machines (SSVMs), respectively. Solving the SSVMs with the Trust Region Newton method (TRON) leads to two quadratically convergent algorithms. Experiments in text classification tasks show that the proposed SSVMs are effective in real-world applications. We also introduce a general smooth convex loss function to unify several commonly-used convex loss functions in machine learning. The general framework provides smooth approximation functions to non-smooth convex loss functions, which can be used to obtain smooth models that can be solved with faster convergent optimization algorithms.

Ridge parameter estimation for the linear regression model under different loss functions using T-K approximation

In multiple linear regression models, the explanatory variables should be uncorrelated within each other but this assumption is violated in most of the cases. Generally, ordinary least square (OLS) estimator produces larger variances when explanatory variables are highly multicollinear. So, in this paper, we propose some new ridge parameters under Bayesian perspective relative to different loss functions, using Tierney and Kadane (T-K) approximation technique to overcome the effect of multicollinearity. We conduct the simulation study to compare the performance of the proposed estimators with OLS estimator and ordinary ridge estimator with some available best ridge parameters using mean squared error as the performance evaluation criterion. A real application is also consider to show the superiority of proposed estimators against competitive estimators. Based on the results of simulation and real application, we conclude that Bayesian ridge parameter estimated under general entropy loss function is better as compared to the OLS estimator and ordinary ridge estimator, when explanatory variables are small. This statement is also true for larger explanatory variables with small sample size. While for larger sample sizes and explanatory variables, the ordinary ridge estimator with best ridge parameter gives the better performance as others.

Robust penalized spline estimation with difference penalties

Penalized spline estimation with discrete difference penalties (P-splines) is a popular estimation method for semiparametric models, but the classical least-squares estimator is highly sensitive to deviations from its ideal model assumptions. To remedy this deficiency, a broad class of P-spline estimators based on general loss functions is introduced and studied. Robust estimators are obtained by well-chosen loss functions, such as the Huber or Tukey loss function. A preliminary scale estimator can also be included in the loss function. It is shown that this class of P-spline estimators enjoys the same optimal asymptotic properties as least-squares P-splines, thereby providing strong theoretical motivation for its use. The proposed estimators may be computed very efficiently through a simple adaptation of well-established iterative least squares algorithms and exhibit excellent performance even in finite samples, as evidenced by a numerical study and a real-data example.

Improving Lasso for model selection and prediction

AbstractIt is known that the Thresholded Lasso (TL), SCAD or MCP correct intrinsic estimation bias of the Lasso. In this paper we propose an alternative method of improving the Lasso for predictive models with general convex loss functions which encompass normal linear models, logistic regression, quantile regression, or support vector machines. For a given penalty we order the absolute values of the Lasso nonzero coefficients and then select the final model from a small nested family by the Generalized Information Criterion. We derive exponential upper bounds on the selection error of the method. These results confirm that, at least for normal linear models, our algorithm seems to be the benchmark for the theory of model selection as it is constructive, computationally efficient and leads to consistent model selection under weak assumptions. Constructivity of the algorithm means that, in contrast to the TL, SCAD or MCP, consistent selection does not rely on the unknown parameters as the cone invertibility factor. Instead, our algorithm only needs the sample size, the number of predictors and an upper bound on the noise parameter. We show in numerical experiments on synthetic and real‐world datasets that an implementation of our algorithm is more accurate than implementations of studied concave regularizations. Our procedure is included in the R package DMRnet and available in the CRAN repository.

Optimal capital allocation principles considering capital shortfall and surplus risks in a hierarchical corporate structure

When allocating a given total capital among main business lines and their sub-business lines, a decision maker will face both capital shortfall risk and capital surplus risk for each business line. There is a trade-off between the two kinds of risks. To balance such two kinds of risks in main business lines and their sub-business lines, we propose a capital allocation model considering both capital shortfall risks and capital surplus risks in a hierarchical corporate structure. We derive optimal allocation principles as solutions that minimize a general loss function that balances the shortfall and surplus risks. As applications of the proposed model, a general proportional allocation principle in the presence of a hierarchical corporate structure is derived. It is illustrated that the general proportional allocation can be viewed as the unique solution minimizing a specified loss function in the proposed model. Connections between this proposed model and some risk factors, such as dependence, extreme events, exogenous risks, and risk measures, are discussed.

Event-based backpropagation can compute exact gradients for spiking neural networks

Spiking neural networks combine analog computation with event-based communication using discrete spikes. While the impressive advances of deep learning are enabled by training non-spiking artificial neural networks using the backpropagation algorithm, applying this algorithm to spiking networks was previously hindered by the existence of discrete spike events and discontinuities. For the first time, this work derives the backpropagation algorithm for a continuous-time spiking neural network and a general loss function by applying the adjoint method together with the proper partial derivative jumps, allowing for backpropagation through discrete spike events without approximations. This algorithm, EventProp, backpropagates errors at spike times in order to compute the exact gradient in an event-based, temporally and spatially sparse fashion. We use gradients computed via EventProp to train networks on the Yin-Yang and MNIST datasets using either a spike time or voltage based loss function and report competitive performance. Our work supports the rigorous study of gradient-based learning algorithms in spiking neural networks and provides insights toward their implementation in novel brain-inspired hardware.