Underlying Loss Function Research Articles

Tight and Efficient Gradient Bounds for Parameterized Quantum Circuits

The training of a parameterized model largely depends on the landscape of the underlying loss function. In particular, vanishing gradients are a central bottleneck in the scalability of variational quantum algorithms (VQAs), and are known to arise in various ways. However, a caveat of most existing gradient bound results is the requirement of t-design circuit assumptions that are typically not satisfied in practice. In this work, we loosen these assumptions altogether and derive tight upper and lower bounds on loss and gradient concentration for a large class of parameterized quantum circuits and arbitrary observables, which are significantly stronger than prior work. Moreover, we show that these bounds, as well as the variance of the loss itself, can be estimated efficiently and classically-providing practical tools to study the loss landscapes of VQA models, including verifying whether or not a circuit/observable induces barren plateaus. In particular, our results can readily be leveraged to rule out barren plateaus for a realistic class of ansätze and mixed observables, namely, observables containing a non-vanishing local term. This insight has direct implications for hybrid Quantum Generative Adversarial Networks (qGANs). We prove that designing the discriminator appropriately leads to 1-local weights that stay constant in the number of qubits, regardless of discriminator depth. This implies that qGANs with appropriately chosen generators do not suffer from barren plateaus even at scale-making them a promising candidate for applications in generative quantum machine learning. We demonstrate this result by training a qGAN to learn a 2D mixture of Gaussian distributions with up to 16 qubits, and provide numerical evidence that global contributions to the gradient, while initially exponentially small, may kick in substantially over the course of training.

Improvised light weight deep CNN based U-Net for the semantic segmentation of lungs from chest X-rays

In the last decade, neural networks and deep learning techniques are widely adopted in the field of medical imaging for image detection, classification, and segmentation tasks and has achieved exceptional results. Deep models have immensely contributed in this sector thus making it easier for quick diagnosis, early and effective treatment. X-Rays are most commonly used radiological medical imaging tool and radiologists. Medical practitioners face huge difficulty to classify and segment different organs by looking at the chest X-Rays for the possible detection of any abnormalities in thoracic cavity which includes lungs, heart, diaphragm, sternum, and clavicles. Specifically, we presented a modified U-Net architecture for the semantic segmentation of lungs from the chest X-Rays images. The proposed model is lighter counterpart of U-Net architecture consisting multiple dropouts in all the deconvolutional layers, this efficiently resolving the problem of overfitting and outperforms the original U-Net model. The light weight proposed architecture outperforms on the combination of three different datasets by producing 92.71% training accuracy and a marvelous generalization accuracy of 93.87% with Adam optimizer and underlying loss function is Dice coefficient. The clubbed dataset comprised of over 900 chest X-Rays images along with corresponding truth segmentation masks provided by the professionals. Conclusively, we offer improved performance, better model generalization with relatively lighter model for the semantic segmentation of lungs from the chest X-Rays images.

Bayesian Risk With Bregman Loss: A Cramér–Rao Type Bound and Linear Estimation

A general class of Bayesian lower bounds when the underlying loss function is a Bregman divergence is demonstrated. This class can be considered as an extension of the Weinstein–Weiss family of bounds for the mean squared error and relies on finding a variational characterization of Bayesian risk. This approach allows for the derivation of a version of the Cramér–Rao bound that is specific to a given Bregman divergence. This new generalization of the Cramér–Rao bound reduces to the classical one when the loss function is taken to be the Euclidean norm. In order to evaluate the effectiveness of the new lower bounds, the paper also develops upper bounds on Bayesian risk, which are based on optimal linear estimators. The effectiveness of the new bound is evaluated in the Poisson noise setting.

Optimal Transport-Based Distributionally Robust Optimization: Structural Properties and Iterative Schemes

We consider optimal transport-based distributionally robust optimization (DRO) problems with locally strongly convex transport cost functions and affine decision rules. Under conventional convexity assumptions on the underlying loss function, we obtain structural results about the value function, the optimal policy, and the worst-case optimal transport adversarial model. These results expose a rich structure embedded in the DRO problem (e.g., strong convexity even if the non-DRO problem is not strongly convex, a suitable scaling of the Lagrangian for the DRO constraint, etc., which are crucial for the design of efficient algorithms). As a consequence of these results, one can develop efficient optimization procedures that have the same sample and iteration complexity as a natural non-DRO benchmark algorithm, such as stochastic gradient descent.

Instance weighting through data imprecisiation

In machine learning, instance weighting is commonly used to control the influence of individual data points in a learning process. The general idea is to improve results (e.g., the accuracy of a predictor) by restricting the influence of training examples that do not appear to be representative and may bias the learner in an undesirable way. The simplest and most common approach is to modulate the influence of each data point through multiplicative scaling. In this paper, we elaborate on the idea of instance weighting through data imprecisiation as a viable alternative to existing methods, and formalize this approach within the framework of superset learning. Roughly speaking, the idea is to reduce the influence of training examples by turning a precise data point into an imprecise observation. Within the framework of optimistic superset learning, a generic approach to superset learning, this effectively comes down to modifying an underlying loss function on a per-instance basis. We illustrate our approach for the case of binary classification with support vector machines, showing that it compares favorably with existing approaches to instance weighting in support vector machines. In a further case study, we demonstrate the usefulness of instance weighting through data imprecisiation for the practical problem of depth estimation in monocular images.

SPLBoost: An Improved Robust Boosting Algorithm Based on Self-Paced Learning

It is known that boosting can be interpreted as an optimization technique to minimize an underlying loss function. Specifically, the underlying loss being minimized by the traditional AdaBoost is the exponential loss, which proves to be very sensitive to random noise/outliers. Therefore, several boosting algorithms, e.g., LogitBoost and SavageBoost, have been proposed to improve the robustness of AdaBoost by replacing the exponential loss with some designed robust loss functions. In this article, we present a new way to robustify AdaBoost, that is, incorporating the robust learning idea of self-paced learning (SPL) into the boosting framework. Specifically, we design a new robust boosting algorithm based on the SPL regime, that is, SPLBoost, which can be easily implemented by slightly modifying off-the-shelf boosting packages. Extensive experiments and a theoretical characterization are also carried out to illustrate the merits of the proposed SPLBoost.

The importance of loss functions: A note on the evolution of the Toxicity Probability Interval design

The Toxicity Probability Interval Design by Ji et al. (2007), which was subsequently modified by the mTPI (Ji et al., 2010), proposed a more efficient approach to early-phase dose-finding than conventional designs like 3 + 3. Subsequent authors reported issues with the method, finding that it tends to stay at a dose level when clinical intuition would suggest the toxicity level warrants decrease. Several iterations of refinement proceeded in an effort to address these issues, including the mTPI-2 and the keyboard method, as well as alternative approaches such as the BOIN. This author suggests the reason for these safety issues involves the underlying loss function. The TPI and mTPI used the identify function defined over wide intervals. As explained in this paper, this function and its domain can be problematic as a model of patients’ loss experience. Later refinements moved the loss function closer to one more consistent with clinical intuition, and this explains their improved safety performance. Greater attention to quality as defined by fitness for use, including early evaluation of patient-experience and clinical-intuition implications of proposed loss functions, may improve future design efforts.

Perturbation analysis of gradient-based adversarial attacks

After the discovery of adversarial examples and their adverse effects on deep learning models, many studies focused on finding more diverse methods to generate these carefully crafted samples. Although empirical results on the effectiveness of adversarial example generation methods against defense mechanisms are discussed in detail in the literature, an in-depth study of the theoretical properties and the perturbation effectiveness of these adversarial attacks has largely been lacking. In this paper, we investigate the objective functions of three popular methods for adversarial example generation: the L-BFGS attack, the Iterative Fast Gradient Sign attack, and Carlini & Wagner’s attack. Specifically, we perform a comparative and formal analysis of the loss functions underlying the aforementioned attacks while laying out large-scale experimental results on the ImageNet dataset. This analysis exposes (1) the faster optimization speed as well as the constrained optimization space of the cross-entropy loss, (2) the detrimental effects of using the signature of the cross-entropy loss on optimization precision as well as optimization space, and (3) the slow optimization speed of the logit loss in the context of adversariality. Our experiments reveal that the Iterative Fast Gradient Sign attack, which is thought to be fast for generating adversarial examples, is the worst attack in terms of the number of iterations required to create adversarial examples in the setting of equal perturbation. Moreover, our experiments show that the underlying loss function of Carlini & Wagner’s attack, which is criticized for being substantially slower than other adversarial attacks, is not that much slower than other loss functions. Finally, we analyze how well neural networks can identify adversarial perturbations generated by the attacks under consideration, hereby revisiting the idea of adversarial retraining on ImageNet.

A Nonparametric Evaluation of the Optimality of German Export and Import Growth Forecasts under Flexible Loss

This study contributes to research on the nonparametric evaluation of German trade forecasts. To this end, I compute random classification and regression forests to analyze the optimality of annual German export and import growth forecasts from 1970 to 2017. A forecast is considered as optimal if a set of predictors, which models the information set of a forecaster at the time of forecast formation, has no explanatory power for the corresponding (sign of the) forecast error. I analyze trade forecasts of four major German economic research institutes, a collaboration of German economic research institutes, and one international forecaster. For trade forecasts with a horizon of half-a-year, I cannot reject forecast optimality for all but one forecaster. In the case of a forecast horizon of one year, forecast optimality is rejected in more cases if the underlying loss function is assumed to be quadratic. Allowing for a flexible loss function results in more favorable assessment of forecast optimality.

A joint quantile and expected shortfall regression framework

We introduce a novel regression framework which simultaneously models the quantile and the Expected Shortfall (ES) of a response variable given a set of covariates. This regression is based on a strictly consistent loss function for the pair quantile and ES, which allows for M- and Z-estimation of the joint regression parameters. We show consistency and asymptotic normality for both estimators under weak regularity conditions. The underlying loss function depends on two specification functions, whose choice affects the properties of the resulting estimators. We find that the Z-estimator is numerically unstable and thus, we rely on M-estimation of the model parameters. Extensive simulations verify the asymptotic properties and analyze the small sample behavior of the M-estimator for different specification functions. This joint regression framework allows for various applications including estimating, forecasting, and backtesting ES, which is particularly relevant in light of the recent introduction of ES into the Basel Accords.

The Lovász Hinge: A Novel Convex Surrogate for Submodular Losses.

Learning with non-modular losses is an important problem when sets of predictions are made simultaneously. The main tools for constructing convex surrogate loss functions for set prediction are margin rescaling and slack rescaling. In this work, we show that these strategies lead to tight convex surrogates iff the underlying loss function is increasing in the number of incorrect predictions. However, gradient or cutting-plane computation for these functions is NP-hard for non-supermodular loss functions. We propose instead a novel surrogate loss function for submodular losses, the Lovász hinge, which leads to O(p logp) complexity with O(p) oracle accesses to the loss function to compute a gradient or cutting-plane. We prove that the Lovász hinge is convex and yields an extension. As a result, we have developed the first tractable convex surrogates in the literature for submodular losses. We demonstrate the utility of this novel convex surrogate through several set prediction tasks, including on the PASCAL VOC and Microsoft COCO datasets.

Computing Jacobian and Hessian of Estimators and Their Application to Risk Approximation

This letter gives formulas to compute the Jacobian and Hessian of an estimator that can be written as the maximum of a given scoring function, which includes the important cases of maximum likelihood (ML) and least squares (LS) estimation. We use the knowledge about these derivatives to compute two approximations of the estimator risk and show that the linear risk approximation of an ML estimator coincides with the Cramer-Rao bound for the case of a Gaussian signal model where the underlying loss function that is used for the risk computation is the squared error loss.

Testing the rationality of DOE's energy price forecasts under asymmetric loss preferences

This paper examines the rationality of the price forecasts for energy commodities of the United States Department of Energy's (DOE), departing from the common assumption in the literature that DOE's forecasts are based on a symmetric underlying loss function with respect to positive vs. negative forecast errors. Instead, we opt for the methodology of Elliott et al. (2005) that allows testing the joint hypothesis of an asymmetric loss function and rationality and reveals the underlying preferences of the forecaster. Results indicate the existence of asymmetries in the shape of the loss function for most energy categories with preferences leaning towards optimism. Moreover, we also examine whether there is a structural break in those preferences over the examined period, 1997–2012.

Revealing asymmetries in the loss function of WTI oil futures market

This paper examines behavioural aspects of the West Texas Intermediate (WTI) oil 1-month futures from 1995 to 2012. We consider that oil futures are formed based on an underlying generalised loss function with an unknown shape parameter that provides information regarding preferences. Even without observing fundamentals of WTI oil futures we can assess whether preferences lean towards a symmetric or asymmetric loss function. Our empirical evidence is robust across information sets and shows that overall loss preferences of WTI 1-month oil futures are rather optimistic and thus the underlying loss function is asymmetric. This implies that if one disregards this asymmetry the WTI oil futures should not be viewed as rational. We further provide statistical tests that allow deviations from a symmetric loss function. As part of a sensitivity analysis, and given the long span of our sample, we perform a novel analysis for detecting breakdowns in our series over time. Based on this analysis we re-examine the shape parameters of the loss function for WTI oil month futures for sub-periods. Interestingly, preferences of WTI 1-month oil futures have shifted towards optimism post 2008 period, marking the collapse of Lehman Brothers.

ARE INVESTMENT EXPECTATIONS RATIONAL, ADAPTIVE OR REGRESSIVE?

I. INTRODUCTION Tests of the rational expectations hypothesis (REH) outside the experimental laboratory typically utilize two types of data: quantitative measures of expectations about publicly observed variables such as inflation, usually elicited from small samples of professional forecasters; and qualitative measures of expectations about privately observed variables such as personal income, elicited from large samples of households. Analyses of both types of data have produced varying levels of support for the REH. For example, Keane and Runkle (1990, 1998) find evidence in favor of the REH using the first type of data, while Bonham and Cohen (2001) in part overturn their results after accounting for the nature of cross-correlations inherent in a group of individuals forecasting the same publicly observed integrated time series. Next, using household survey data, Das and van Soest (1997, 1999) and Das et al. (1999) find evidence against the REH. In contrast, Benitez-Silva et al. (2008) furthered the econometric analysis of such data and presented evidence in favor of the REH. In employing publicly available professional forecaster data several concerns related to pooling, data revisions, and the possibility of signaling on the part of forecasters have been raised. (1) Further, Pesaran and Weale (2006) review a large and growing literature on survey expectations of households and the attendant econometric and substantive issues in analyzing qualitative response expectations data from such surveys. They conclude for myriad reasons that it remains the case that the analysis of individual responses to such surveys, and in particular to those who collect only qualitative information, is underdeveloped. These issues can be addressed in part by data on quantitative expectations about privately observed variables; this paper employs data from two new survey sources that provide such information. The resulting panel of 775 firms in Canada contains quantitative information on expectations about purchases of new machinery and equipment (investment), as well as realized levels of investment, over 7 yr. The data are particularly useful because they avoid some of the issues raised for previous studies and their analysis helps to contribute, on the firm side, to the agenda laid out by Manski (2004) who sees a critical need for basic research on expectations formation. [FIGURE 1 OMITTED] Estimation results suggest that investment expectations are not rational if the assumed underlying loss function is assumed to be symmetric. That is, the usual unbiasedness and efficiency requirements for rational expectations, clearly elucidated for instance by Lovell (1986), do not hold for the collected data. Further, hypothesis tests on statistical models implied by conventional adaptive expectations do not suggest the prevalence of such behavior in the data. However, a particular version of such extrapolative expectations, namely, regressive expectations formation, does find support in the data. The paper is structured as follows. Section II discusses the data on 775 manufacturing plants over 7 yr that was compiled by merging two national surveys in Canada. In Section III, the statistical models (and attendant hypothesis tests) as implied by various expectations formation assumptions are described followed by their estimation results. Section V concludes. II. THE DATA Data were compiled from two confidential, proprietary surveys conducted by Statistics Canada: the Capital Expenditures Survey (CES) and the Annual Survey of Manufactures (ASM) over 7 yr (1986-1992). (2) The CES contains data on one-step ahead expected ([Expect.sub.it]) and realized (Actualit) capital expenditures on machinery and equipment (i.e., investment), and the ASM on shipments ([Shipment.sub.sit]). (3) Owing to the lumpy and long-term nature of construction expenditures, this variable was excluded. Finally, only manufacturing sector records were selected for both surveys. …

Kernel Component Analysis Using an Epsilon-Insensitive Robust Loss Function

Kernel principal component analysis (PCA) is a technique to perform feature extraction in a high-dimensional feature space, which is nonlinearly related to the original input space. The kernel PCA formulation corresponds to an eigendecomposition of the kernel matrix: eigenvectors with large eigenvalues correspond to the principal components in the feature space. Starting from the least squares support vector machine (LS-SVM) formulation to kernel PCA, we extend it to a generalized form of kernel component analysis (KCA) with a general underlying loss function made explicit. For classical kernel PCA, the underlying loss function is L(2) . In this generalized form, one can plug in also other loss functions. In the context of robust statistics, it is known that the L(2) loss function is not robust because its influence function is not bounded. Therefore, outliers can skew the solution from the desired one. Another issue with kernel PCA is the lack of sparseness: the principal components are dense expansions in terms of kernel functions. In this paper, we introduce robustness and sparseness into kernel component analysis by using an epsilon-insensitive robust loss function. We propose two different algorithms. The first method solves a set of nonlinear equations with kernel PCA as starting points. The second method uses a simplified iterative weighting procedure that leads to solving a sequence of generalized eigenvalue problems. Simulations with toy and real-life data show improvements in terms of robustness together with a sparse representation.

Dynamic modeling under linear-exponential loss

We develop a methodology of parametric modeling of time series dynamics when the underlying loss function is linear-exponential (Linex). We propose to directly model the dynamics of the conditional expectation that determines the optimal predictor. The procedure hinges on the exponential quasi-maximum likelihood interpretation of the Linex loss and nicely fits the multiplicative error modeling framework. Many conclusions relating to estimation, inference and forecasting follow from results already available in the econometric literature. The methodology is illustrated using data on United States GNP growth and Treasury bill returns.

Dynamic Modeling Under Linear-Exponential Loss

We develop a methodology of parametric modeling of time series dynamics when the underlying loss function is linear-exponential (Linex). We propose to directly model the dynamics of the conditional expectation that determines the optimal predictor. The procedure hinges on the exponential quasi maximum likelihood interpretation of the Linex loss and nicely fits the multiple error modeling framework. Many conclusions relating to estimation, inference and forecasting follow from results already available in the econometric literature. The methodology is illustrated using data on United States GNP growth and Treasury bill returns.

Finite Sample Evidence Suggesting a Heavy Tail Problem of the Generalized Empirical Likelihood Estimator

Comprehensive Monte Carlo evidence is provided that compares the finite sample properties of generalized empirical likelihood (GEL) estimators to the ones of k-class estimators in the linear instrumental variables (IV) model. We focus on sample median, mean, mean squared error, and on the coverage probability and length of confidence intervals obtained from inverting a t-statistic based on the various estimators. The results indicate that in terms of the above criteria, all the GEL estimators and the limited information maximum likelihood (LIML) estimator behave very similarly. This suggests that GEL estimators might also share the “no-moment” problem of LIML. At sample sizes as in our Monte Carlo study, there is no systematic bias advantage of GEL estimators over k-class estimators. On the other hand, the standard deviation of GEL estimators is pronouncedly higher than for some of the k-class estimators. Therefore, if mean squared error is used as the underlying loss function, our study suggests the use of computationally simple estimators, such as two-stage least squares, in the linear IV model rather than GEL. Based on the properties of confidence intervals, we cannot recommend the use of GEL estimators either in the linear IV model.

The Minimax, the Minimin, and the Hurwicz Adjustment Principle

In this paper the Hurwicz decision rule is applied to an adjustment problem concerning the decision whether a given action should be improved in the light of some knowledge on the states of nature or on other actors' behaviour. In comparison with the minimax and the minimin adjustment principles the general Hurwicz rule reduces to these specific classes whenever the underlying loss function is quadratic and knowledge is given by an ellipsoidal set. In the framework of the adjustment model discussed in this paper Hurwicz's optimism index can be interpreted as a mobility index representing the actor's attitude towards new external information. Examples are given that serve to illustrate the theoretical findings.