Quadratic Loss Function Research Articles

Hebbian learning from first principles

Recently, the original storage prescription for the Hopfield model of neural networks – as well as for its dense generalizations – has been turned into a genuine Hebbian learning rule by postulating the expression of its Hamiltonian for both the supervised and unsupervised protocols. In these notes, first, we obtain these explicit expressions by relying upon maximum entropy extremization à la Jaynes. Beyond providing a formal derivation of these recipes for Hebbian learning, this construction also highlights how Lagrangian constraints within entropy extremization force network’s outcomes on neural correlations: these try to mimic the empirical counterparts hidden in the datasets provided to the network for its training and, the denser the network, the longer the correlations that it is able to capture. Next, we prove that, in the big data limit, whatever the presence of a teacher (or its lacking), not only these Hebbian learning rules converge to the original storage prescription of the Hopfield model but also their related free energies (and, thus, the statistical mechanical picture provided by Amit, Gutfreund and Sompolinsky is fully recovered). As a sideline, we show mathematical equivalence among standard Cost functions (Hamiltonian), preferred in Statistical Mechanical jargon, and quadratic Loss Functions, preferred in Machine Learning terminology. Remarks on the exponential Hopfield model (as the limit of dense networks with diverging density) and semi-supervised protocols are also provided.

Dynamical transition in controllable quantum neural networks with large depth

Understanding the training dynamics of quantum neural networks is a fundamental task in quantum information science with wide impact in physics, chemistry and machine learning. In this work, we show that the late-time training dynamics of quantum neural networks with a quadratic loss function can be described by the generalized Lotka-Volterra equations, leading to a transcritical bifurcation transition in the dynamics. When the targeted value of loss function crosses the minimum achievable value from above to below, the dynamics evolve from a frozen-kernel dynamics to a frozen-error dynamics, showing a duality between the quantum neural tangent kernel and the total error. In both regions, the convergence towards the fixed point is exponential, while at the critical point becomes polynomial. We provide a non-perturbative analytical theory to explain the transition via a restricted Haar ensemble at late time, when the output state approaches the steady state. Via mapping the Hessian to an effective Hamiltonian, we also identify a linearly vanishing gap at the transition point. Compared with the linear loss function, we show that a quadratic loss function within the frozen-error dynamics enables a speedup in the training convergence. The theory findings are verified experimentally on IBM quantum devices.

A robust approach to Gaussian process implementation

Abstract. Gaussian process (GP) regression is a flexible modeling technique used to predict outputs and to capture uncertainty in the predictions. However, the GP regression process becomes computationally intensive when the training spatial dataset has a large number of observations. To address this challenge, we introduce a scalable GP algorithm, termed MuyGPs, which incorporates nearest-neighbor and leave-one-out cross-validation during training. This approach enables the evaluation of large spatial datasets with state-of-the-art accuracy and speed in certain spatial problems. Despite these advantages, conventional quadratic loss functions used in the MuyGPs optimization, such as root mean squared error (RMSE), are highly influenced by outliers. We explore the behavior of MuyGPs in cases involving outlying observations and, subsequently, develop a robust approach to handle and mitigate their impact. Specifically, we introduce a novel leave-one-out loss function based on the pseudo-Huber function (LOOPH) that effectively accounts for outliers in large spatial datasets within the MuyGPs framework. Our simulation study shows that the LOOPH loss method maintains accuracy despite outlying observations, establishing MuyGPs as a powerful tool for mitigating unusual observation impacts in the large data regime. In the analysis of US ozone data, MuyGPs provides accurate predictions and uncertainty quantification, demonstrating its utility in managing data anomalies. Through these efforts, we advance the understanding of GP regression in spatial contexts.

Asymmetries in Federal Reserve Objectives

Abstract Abstract This paper uses evidence from the Federal Open Market Committee’s Summary of Economic Projections to show that US monetary policymakers have objectives over unemployment and inflation outcomes that are not well-approximated through a conventional quadratic loss function. Rather, policymakers derive material costs (benefits) from overshooting (undershooting) their long-run inflation and unemployment goals. The trade-off between the resultant downward tilts in unemployment and inflation played a key role in shaping the evolution of monetary policy choices since the Great Recession.

Batch-sizing and machinability data systems for milling operations: An optimal sustainable cost of quality approach

Nowadays, manufacturers make every effort to achieve a higher quality of their products at an attractive cost. With all the introduced legislation and incentives in the developed world to address global warming, machining shops in the West also strive to cut greenhouse emissions. This article offers an optimal approach to the micro Computer-Aided Process Planning (CAPP) problem to optimize the internal quality cost and buffering effect while keeping the environmental impact low. To optimize the machining parameters, the mathematical model is developed for different milling operations, face, side, and peripheral. cutting speed, feed rate, axial depth of cut, radial depth of cut, nose radius, and batch sizing while maximizing profit and meeting customer demand. A Mixed-Integer Nonlinear Programming (MINLP) model is formulated and solved using Classical Constrained Nonlinear Optimization (CCNO) and Genetic Algorithms (GAs). Surface roughness, used as a metric to evaluate the desired quality level of a finished machined part type, is modeled as a Gaussian random variable to model the surface roughness of the machined part utilizing a cumulative normal distribution. The ratio of rework and scrap is calculated in terms of the surface roughness of the machined part shifting away from the target and exceeding upper and lower specification limits. The internal failure cost model, addressing both scrap and rework, is developed based on Taguchi’s quadratic loss function. CCNO is employed to validate the results obtained by GAs, relaxing the lot-sizing integrality constraint and, thus, the convexity of the produced relaxed model. An iterative method employing a developed multi-regression model is used to solve for the expended power consumption (an inherent highly nonlinear environmental criterion of the developed model) within both GAs and CCNO. This study reveals that the machining parameters substantially impact the cost components of the objective function as well as the scrap and rework quantities. A stringent quality cost target can force the model to optimize the feed rate and nose radius to minimize the internal failure quality cost while improving the environmental impact, including direct and indirect power consumption and CO2 emissions considerations.

Wealth in the Quadratic Loss Function of the Ramsey Malinvaud Cass Koopmans Model of Optimal Savings

En utilisant le developpement limité de Taylor à l’ordre deux du Lagrangien du modèle d’épargne optimale de Ramsey, la richesse est prise en compte dans la fonction de perte quadratique et pas seulement la consommation. La pondération de la richesse est donnée par le degré de concavité de la fonction de production à rendements décroissants multiplié par l’utilité marginale de la consommation. La pondération de la consommation est donnée par le degré de concavité de la fonction d’utilité. Cette fonction de perte quadratique implique que la vitesse de convergence vers l’équilibre dépend de l’arbitrage entre le lissage de la richesse (qui accèlère la convergence) et le lissage de la consommation (qui retarde la convergence). En revanche, le développement limité de Taylor à l’ordre deux de l’utilité (au lieu du Lagrangien) met un poids de la richesse égal à zéro dans la fonction de perte quadratique, ce qui est faux si la fonction de production est à rendements strictement décroissants.

Credibility Theory for Variance Premium Principle

The method of credibility theory plays a critical role in various research areas of actuarial science. Among others, the hypothetical mean and process variance are two quantities that convey crucial information to insurance companies when determining premiums for the insureds. The classical credibility model charges premiums by applying a linear combination of the mean of past claims and the population mean, and the corresponding estimator is proved to be the best estimator of the hypothetical mean under the mean squared loss criterion. Enlightened by the prestigious variance premium principle, we propose a credibility approach to estimate the linear combination of hypothetical mean and process variance under the quadratic loss function. Our proposed estimator consists of the linear form of observations and their quadratic terms, as well as some quantities representing population information. Meanwhile, a spin-off result is found and utilized to compare with the classical credibility model and the q-credibility model. The nonparametric estimators of structural quantities are also provided for ease of practical usage. Several numerical illustrations are carried out to demonstrate the performance of the estimator. A real dataset from a Swiss insurance company is also analyzed for the practical application of our results, where a data-driven procedure is proposed for determining the safety loading.

Quadratic hyper-surface kernel-free large margin distribution machine-based regression and its least-square form

ε-Support vector regression (ε-SVR) is a powerful machine learning approach that focuses on minimizing the margin, which represents the tolerance range between predicted and actual values. However, recent theoretical studies have highlighted that simply minimizing structural risk does not necessarily result in well margin distribution. Instead, it has been shown that the distribution of margins plays a more crucial role in achieving better generalization performance. Furthermore, the kernel-free technique offers a significant advantage as it effectively reduces the overall running time and simplifies the parameter selection process compared to the kernel trick. Based on existing kernel-free regression methods, we present two efficient and robust approaches named quadratic hyper-surface kernel-free large margin distribution machine-based regression (QLDMR) and quadratic hyper-surface kernel-free least squares large margin distribution machine-based regression (QLSLDMR). The QLDMR optimizes the margin distribution by considering both ε-insensitive loss and quadratic loss function similar to the large-margin distribution machine-based regression (LDMR). QLSLDMR aims to reduce the cost of the computing process of QLDMR, which transforms inequality constraints into an equality constraint inspired by least squares support vector machines (LSSVR). Both models combined the spirit of optimal margin distribution with kernel-free technique and after simplification are convex so that they can be solved by some classical methods. Experimental results demonstrate the superiority of the optimal margin distribution combined with the kernel-free technique in robustness, generalization, and efficiency.

TimeSQL: Improving multivariate time series forecasting with multi-scale patching and smooth quadratic loss

Multivariate time series are usually sequences of real-valued variables recorded at regular intervals. Forecasting them poses significant challenges due to their inherent noise and complex temporal dynamics. To effectively address these challenges, we introduce TimeSQL, a novel approach specifically designed for multivariate time series. TimeSQL employs multi-scale patching and a smooth quadratic loss (SQL) function. This enables it to adeptly capture both local and long-term patterns in complex, noisy data environments. The multi-scale patching approach provides a comprehensive view of temporal correlations, while the SQL, rooted in the rational quadratic kernel, strategically adjusts gradients to avoid overfitting. Through both theoretical and empirical analysis, we have demonstrated that TimeSQL exhibits superior noise resilience compared with models based on Mean Squared Error (MSE). Evaluated across eight diverse benchmark datasets, TimeSQL consistently delivers exceptional performance in multivariate time series forecasting. Furthermore, ablation studies underscore the adaptability of TimeSQL's components, enhancing the performance of various other forecasting models and proving their utility as plug-and-play modules.

Dealing with uncertainty: Balancing exploration and exploitation in deep recurrent reinforcement learning

Incomplete knowledge of the environment leads an agent to make decisions under uncertainty. One of the major dilemmas in Reinforcement Learning (RL) where an autonomous agent has to balance two contrasting needs in making its decisions is: exploiting the current knowledge of the environment to maximize the cumulative reward as well as exploring actions that allow improving the knowledge of the environment, hopefully leading to higher reward values (exploration–exploitation trade-off). Concurrently, another relevant issue regards the full observability of the states, which may not be assumed in all applications. For instance, when 2D images are considered as input in an RL approach used for finding the best actions within a 3D simulation environment. In this work, we address these issues by deploying and testing several techniques to balance exploration and exploitation trade-off on partially observable systems for predicting steering wheels in autonomous driving scenarios. More precisely, the final aim is to investigate the effects of using both adaptive and deterministic exploration strategies coupled with a Deep Recurrent Q-Network. Additionally, we adapted and evaluated the impact of a modified quadratic loss function to improve the learning phase of the underlying Convolutional Recurrent Neural Network. We show that adaptive methods better approximate the trade-off between exploration and exploitation and, in general, Softmax and Max-Boltzmann strategies outperform ϵ-greedy techniques.

Comparison between the standard Bayesian method and the hierarchical Bayesian method for estimating the reliability function of the inverse Lomax distribution with type II observational dat

In this paper, the shape and size parameters (without imposing either parameter) and the survival function of the inverse Lomax distribution in the presence of observational data of the second type are estimated using different Bayesian methods, which included the standard Bayesian method and the hierarchical Bayesian method under a symmetric loss function, which is the quadratic loss function, and in order to compare The two methods used Monte Carlo simulation, where different sample sizes and three models of default values for distribution parameters (Ѳ,λ) , were used and the statistical measure, mean integral square error (IMSE), was used. Simulation experiments showed the superiority of the hierarchical Bayesian method because it has the least mean square integral error (IMSE)

Shape restricted additive hazards models: Monotone, unimodal, and U-shape hazard functions.

We consider estimation of the semiparametric additive hazards model with an unspecified baseline hazard function where the effect of a continuous covariate has a specific shape but otherwise unspecified. Such estimation is particularly useful for a unimodal hazard function, where the hazard is monotone increasing and monotone decreasing with an unknown mode. A popular approach of the proportional hazards model is limited in such setting due to the complicated structure of the partial likelihood. Our model defines a quadratic loss function, and its simple structure allows a global Hessian matrix that does not involve parameters. Thus, once the global Hessian matrix is computed, a standard quadratic programming method can be applicable by profiling all possible locations of the mode. However, the quadratic programming method may be inefficient to handle a large global Hessian matrix in the profiling algorithm due to a large dimensionality, where the dimension of the global Hessian matrix and number of hypothetical modes are the same order as the sample size. We propose the quadratic pool adjacent violators algorithm to reduce computational costs. The proposed algorithm is extended to the model with a time-dependent covariate with monotone or U-shape hazard function. In simulation studies, our proposed method improves computational speed compared to the quadratic programming method, with bias and mean square error reductions. We analyze data from a recent cardiovascular study.

Multi-satellite cooperative scheduling method for large-scale tasks based on hybrid graph neural network and metaheuristic algorithm

Traditional heuristic optimization algorithms are no longer applicable to the multiple satellites scheduling with large-scale tasks, as they are unable to provide satisfactory performance in terms of convergence and speed when addressing complex constraint conflicts. Therefore, we propose a multi-satellite cooperative scheduling method for large-scale tasks based on a hybrid graph neural network (GNN) and metaheuristic algorithm. By using the representation and extraction capability of GNN for relations in graph, the features of large-scale tasks and their constraint relations were expressed, and the generalized knowledge was extracted. And combined with metaheuristic algorithm, the optimization framework for large-scale satellite task scheduling based knowledge and data was implemented. The method consists of a GNN pre-solver module for static constraint conflicts and a metaheuristic optimization module for dynamic constraint conflicts of satellite missions. In the GNN pre-solver module, the graph sample and aggregate network was used to learn and extract the common knowledge in the static task-conflict graph, and provided effective prior information for the metaheuristic optimization module. The quadratic unconstrained binary optimization loss function was designed with multiple influencing factors, to convert the discrete optimization function into a continuous function; a greedy threshold was also added to improve the task completion rate. Finally, numerical experimental results showed that the proposed method can achieve an efficient solution to the multi-satellite scheduling problem with tens of thousands tasks. Compared with the commonly used multi-satellite scheduling algorithms (the genetic algorithm (GA) and greedy-GA), the proposed method can obtain higher-quality solutions under the same conditions and greatly improve the computational efficiency of large-scale mission planning.

Subgroup detection based on partially linear additive individualized model with missing data in response

Based on partially linear additive individualized model, a fusion-penalized inverse probability weighted least squares method is proposed to detect the subgroup for missing data in response. Firstly, the B-spline technique is used to approximate the unknown additive individualized functions and then an inverse probability weighted quadratic loss function is established with fusion penalty on the difference of subject-wise B-spline coefficients. Secondly, minimization of such quadratic loss function leads to the estimation of linear regression parameters and individualized B spline coefficients. With a proper tuning parameter, some differences in penalty term are shrunk into zero and thus the corresponding subjects will be clustered into the same subgroup. Thirdly, a clustering method is developed to automatically determine the subgroup membership for the subjects with missing data. Fourthly, large sample properties of resulting estimates are given under some regular conditions. Finally, numerical studies are presented to illustrate the performance of the proposed subgroup detection method.

Discrepancy between structured matrices in the power analysis of a separability test

An important task in the analysis of multivariate data is testing of the covariance matrix structure. In particular, for assessing separability, various tests have been proposed. However, the development of a method of measuring discrepancy between two covariance matrix structures, in relation to the study of the power of the test, remains an open problem. Therefore, a discrepancy measure is proposed such that for two arbitrary alternative hypotheses with the same value of discrepancy, the power of tests remains stable, while for increasing discrepancy the power increases. The basic hypothesis is related to the separable structure of the observation matrix under a doubly multivariate normal model, as assessed by the likelihood ratio and Rao score tests. It is shown that the particular one-parameter method and the Frobenius norm fail in the power analysis of tests, while the entropy and quadratic loss functions can be efficiently used to measure the discrepancy between separable and non-separable covariance structures for a multivariate normal distribution.

The E-Bayesian and hierarchical Bayesian estimations for the reliability analysis of Kumaraswamy generalized distribution based on upper record values

This paper investigates the E-Bayesian and hierarchical Bayesian estimations of shape parameter and reliability function of Kumaraswamy generalized distribution based on upper record values. The classical estimation method is utilized to deduce the maximum likelihood estimation of unknown parameter and reliability function. Bayesian estimates are derived by using conjugate Gamma prior distributions under quadratic and general entropy loss functions. Furthermore, assuming that hyper-hyperparameters obey three prior distributions, the E-Bayesian estimates of unknown parameters and reliability functions are obtained. The hierarchical Bayesian estimates are obtained by using hierarchical prior distributions. We also explore some characteristics and size relationships of E-Bayesian and hierarchical Bayesian estimations. The performance of E-Bayesian, Hierarchical Bayesian, Bayesian, and maximum likelihood estimations is compared based on the minimum mean square error criterion. Finally, the proposed estimation methods are applied to evaluate the reliability of the specimen under ultrasonic fatigue testing, and the results align with their structures and profiles.

Estimation of a Shape Parameter of a Gompertz-lindley Distribution Using Bayesian and Maximum Likelihood Methods

The Gompertz-Lindley distribution is an extension of the Lindley distribution with three parameters. It was found to be more flexible for modeling real life events. The distribution contains two shape parameters and a scale parameter. Despite the necessity of parameter estimation theory in modeling, it has not been shown that a method of estimation method is better for any of these three parameters of the Gompertz-Lindley distribution. This paper identifies the best estimation method for the shape parameter of the Gompertz-Lindley distribution by deriving Bayesian estimators for the shape parameter of the distribution using two non-informative prior distributions (Uniform and Jeffery) and an informative prior (gamma) under squared error loss function (SELF), quadratic loss function (QLF) and precautionary loss function (PLF). These estimators were evaluated and the results compared with the maximum likelihood estimation method using Monte Carlo simulations with the mean square error (MSE) as a criterion for choosing the best estimator.

An economic-statistical design of synthetic Tukey’s control chart with Taguchi’s asymmetric loss functions under log-normal distribution

The synthetic Tukey’s control chart (denoted by Syn-TCC) applies a single observation in the process monitoring and is relatively robust to outliers. For a complex process, the probability distribution of quality characteristic must be carefully identified and the loss of quality characteristic, which is defined as the deviation from the target value, must be accurately estimated so that the quality state of the process may be more comprehensively described. However, the literature rarely studied the economic design of the Syn-TCC with Taguchi’s loss function under non normality. In the present study, it is assumed that the quality characteristic follows a log-normal distribution and Taguchi’s asymmetric linear and quadratic loss functions are involved to develop the economic-statistical design of the Syn-TCC for multiple assignable causes. A case study is presented to illustrate the application of this economic-statistical design of the Syn-TCC to an IC package industry. The sensitivity analysis of this case study reveals that a larger production lot size and larger two coefficients of loss functions generally lead to a higher total expected cost.

Tensor-Based Temporal Control for Partially Observed High-Dimensional Streaming Data

In advanced manufacturing processes, high-dimensional (HD) streaming data (e.g., sequential images or videos) are commonly used to provide online measurements of product quality. Although there exist numerous research studies for monitoring and anomaly detection using HD streaming data, little research is conducted on feedback control based on HD streaming data to improve product quality, especially in the presence of incomplete responses. To address this challenge, this article proposes a novel tensor-based automatic control method for partially observed HD streaming data, which consists of two stages: offline modeling and online control. In the offline modeling stage, we propose a one-step approach integrating parameter estimation of the system model with missing value imputation for the response data. This approach (i) improves the accuracy of parameter estimation, and (ii) maintains a stable and superior imputation performance in a wider range of the rank or missing ratio for the data to be completed, compared to the existing data completion methods. In the online control stage, for each incoming sample, missing observations are imputed by balancing its low-rank information and the one-step-ahead prediction result based on the control action from the last time step. Then, the optimal control action is computed by minimizing a quadratic loss function on the sum of squared deviations from the target. Furthermore, we conduct two sets of simulations and one case study on semiconductor manufacturing to validate the superiority of the proposed framework.

Zero-Norm ELM with Non-convex Quadratic Loss Function for Sparse and Robust Regression

Zero-Norm ELM with Non-convex Quadratic Loss Function for Sparse and Robust Regression