Estimate Of Generalization Error Research Articles

Error bounds of Approximate Weak Rescaled Pure Greedy Algorithms

We study the greedy algorithms for [Formula: see text]-term approximation. We propose a modification of the Weak Rescaled Pure Greedy Algorithm (WRPGA) — Approximate Weak Rescaled Pure Greedy Algorithm (AWRPGA) — with respect to a dictionary of a Banach space [Formula: see text]. By using a geometric property of the unit sphere of [Formula: see text], we obtain a general error estimate in terms of some [Formula: see text]-functional. This estimate implies the convergence condition and convergence rate of the AWRPGA. Furthermore, we obtain the corresponding error estimate for the Vector Approximate Weak Rescaled Pure Greedy Algorithm (VAWRPGA). We show that the AWRPGA (VAWRPGA) performs as well as the WRPGA (VWRPGA) when the noise amplitude changes relatively little. Finally, by using the wavelet bases and trigonometric system of Lebesgue spaces, we show that the convergence rate of the AWRPGA is optimal.

Advanced Ensemble Framework for Diabetes Outcome Forecasting

Abstract: Diabetes is one of the chronic diseases, which is increasing from year to year. Developing an automated system that can detect diabetes patients plays an important role in medical science. A stack classifier model is designed for the detection of diabetes by combining three base estimators such as Random Forest Classifier, LightGBM Classifier, and K-Nearest Neighbors Classifier, and Logistic Regression as meta-classifiers. The data preprocessing includes transforming categorical variables into numerical format. Each base learner is trained on the preprocessed data and predictions are made. In the meta learner stage, Logistic Regression is trained to make predictions based on the predictions of the base learners. The goal of the meta learner is to learn how to combine the predictions of the base learners to make a more accurate final prediction on whether the patient is diabetic or not. The use of a stacking classifier improves prediction accuracy compared to using a single classifier. The developed model gives an accuracy of 98%. The 5-fold cross validation is used to get a more robust estimation of generalization error. Thus, the developed model offers a means to enhance early detection and elevate the quality of care for diabetic patients.

Space-time error estimates for approximations of linear parabolic problems with generalized time boundary conditions

Abstract We first give a general error estimate for the nonconforming approximation of a problem for which a Banach–Nečas–Babuška (BNB) inequality holds. This framework covers parabolic problems with general conditions in time (initial value problems as well as periodic problems) under minimal regularity assumptions. We consider approximations by two types of space-time discretizations, both based on a conforming Galerkin method in space. The first one is the Euler $\theta -$scheme. In this case, we show that the BNB inequality is always satisfied, and may require an extra condition on the time step for $\theta \le \frac 1 2$. The second one is the time discontinuous Galerkin method, where the BNB condition holds without any additional condition.

Error estimates for POD-DL-ROMs: a deep learning framework for reduced order modeling of nonlinear parametrized PDEs enhanced by proper orthogonal decomposition

POD-DL-ROMs have been recently proposed as an extremely versatile strategy to build accurate and reliable reduced order models (ROMs) for nonlinear parametrized partial differential equations, combining (i) a preliminary dimensionality reduction obtained through proper orthogonal decomposition (POD) for the sake of efficiency, (ii) an autoencoder architecture that further reduces the dimensionality of the POD space to a handful of latent coordinates, and (iii) a dense neural network to learn the map that describes the dynamics of the latent coordinates as a function of the input parameters and the time variable. Within this work, we aim at justifying the outstanding approximation capabilities of POD-DL-ROMs by means of a thorough error analysis, showing how the sampling required to generate training data, the dimension of the POD space, and the complexity of the underlying neural networks, impact on the solutions us to formulate practical criteria to control the relative error in the approximation of the solution field of interest, and derive general error estimates. Furthermore, we show that, from a theoretical point of view, POD-DL-ROMs outperform several deep learning-based techniques in terms of model complexity. Finally, we validate our findings by means of suitable numerical experiments, ranging from parameter-dependent operators analytically defined to several parametrized PDEs.

Bellman Neural Networks for the Class of Optimal Control Problems With Integral Quadratic Cost

This work introduces Bellman Neural Networks (BeNNs) and employs them to learn the optimal control actions for the class of optimal control problems (OCPs) with integral quadratic cost. BeNNs represent a particular family of Physics-Informed Neural Networks (PINNs) specifically designed and trained to tackle OCPs via applying the Bellman Principle of Optimality (BPO). The BPO provides necessary and sufficient optimality conditions, which result in a nonlinear partial differential equation known as the Hamilton-Jacobi-Bellman (HJB) equation. BeNNs learn the optimal control actions from the unknown solution of the arising HJB equation (i.e., the value function), where the unknown solution is modeled using a Neural Network. Additionally, the paper shows how to estimate the upper bounds on the generalization error of BeNNs while learning the solutions for the OCP class under consideration. The generalization error estimate is provided in terms of the choice and number of the training points as well as the training error. Numerical studies show that BeNNs can be successfully applied to learn the feedback control actions for the class of optimal control problems considered and, after the training is completed, deployed to control the system in a closed-loop fashion. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Impact Statement</i> —The proposed research improves our understanding of how to solve optimal control problems with closed-loop solutions and has potentially a countless number of applications in several different areas. The study is at the intersection between optimal control theory and artificial intelligence connected with mathematical tools for functional interpolation. This advances the ability to implement a higher level of autonomy in decision-making for practical applications with a beneficial impact on our society.

Applications of the duality theory of convex analysis to the complete electrode model of electrical impedance tomography

The duality theory of convex analysis is applied to the complete electrode model (CEM), which is a standard model in electrical impedance tomography (EIT). This results in a dual formulation of the CEM and a general error estimate. This new formulation of the CEM is written in terms of current fields and is shown to have a unique solution. Using this formulation, the general error estimate is proved, from which two a posteriori error estimates and a well known asymptotic result on CEM solutions are obtained. The first a posteriori error estimate assesses the accuracy of solutions to approximate problems, and the second one assesses the accuracy of approximate solutions. Numerical tests to apply this second estimate are performed, employing the finite element method to obtain approximate solutions.

Evaluating neural network models in site-specific solar PV forecasting using numerical weather prediction data and weather observations

The effective use of solar photovoltaic (PV) installations implies the integration of solar PV output into overall energy consumption planning, optimization, and control. Moreover, day-ahead trading of electricity in Europe makes day-ahead solar PV forecasting utterly important, and thus its accuracy becomes of particular interest. Data-driven PV forecasting models are typically trained using numerical weather prediction (NWP) data, the availability of which represents one of the main obstacles in modeling. In this study, we investigate an alternative scenario, in which an artificial neural network (ANN) is trained on weather observations and then tested on NWP data to simulate the model's use in operational PV forecasting. In the experiments, solar PV output data, historical weather observations, and historical NWP data were collected from three sites in eastern Finland. The results showed that, although training ANN on observational data leads to a slight decrease in its performance compared to ANN trained on NWP data, it still outperforms a physical model. In practice, this alternative scenario means that if historical NWP data are not available for model training, observational data allow effective model selection and parameter tuning, and then generalization error estimates are gradually updated using online NWP data.

Invasive or More Direct Measurements Can Provide an Objective Early-Stopping Ceiling for Training Deep Neural Networks on Non-invasive or Less-Direct Biomedical Data

Early stopping is an extremely common tool to minimize overfitting, which would otherwise be a cause of poor generalization of the model to novel data. However, early stopping is a heuristic that, while effective, primarily relies on ad hoc parameters and metrics. Optimizing when to stop remains a challenge. In this paper, we suggest that for some biomedical applications, a natural dichotomy of invasive/non-invasive measurements, or more generally proximal vs distal measurements of a biological system can be exploited to provide objective advice on early stopping. We discuss the conditions where invasive measurements of a biological process should provide better predictions than non-invasive measurements, or at best offer parity. Hence, if data from an invasive measurement are available locally, or from the literature, that information can be leveraged to know with high certainty whether a model of non-invasive data is overfitted. We present paired invasive/non-invasive cardiac and coronary artery measurements from two mouse strains, one of which spontaneously develops type 2 diabetes, posed as a classification problem. Examination of the various stopping rules shows that generalization is reduced with more training epochs and commonly applied stopping rules give widely different generalization error estimates. The use of an empirically derived training ceiling is demonstrated to be helpful as added information to leverage early stopping in order to reduce overfitting.

On stability and regularization for data-driven solution of parabolic inverse source problems

The diffusion process from some internal source arising in engineering situations can be mathematically described by a parabolic system. We consider an inverse source problem for parabolic system using parametric approximations, where deep neural networks (DNNs) are used to approximate the solution of the inverse problem. First, we prove generalization error estimates depending on training errors and data noise levels by establishing conditional stability of the inverse problem. Following our analysis, we propose a new loss function involving the derivative of the residuals for PDE and measurement data. These extra terms effectively induce higher regularity in solution to deal with the ill-posedness of the inverse problem. Using these regularization terms, we develop reconstruction schemes and demonstrate the effectiveness of our proposed methodology on a number of test problems.

Do ideas have shape? Idea registration as the continuous limit of artificial neural networks

We introduce a Gaussian Process (GP) generalization of ResNets (with unknown functions of the network replaced by GPs and identified via MAP estimation), which includes ResNets (trained with L2 regularization on weights and biases) as a particular case (when employing particular kernels). We show that ResNets (and their warping GP regression extension) converge, in the infinite depth limit, to a generalization of image registration variational algorithms. In this generalization, images are replaced by functions mapping input/output spaces to a space of unexpressed abstractions (ideas), and material points are replaced by data points. Whereas computational anatomy aligns images via warping of the material space, this generalization aligns ideas (or abstract shapes as in Plato’s theory of forms) via the warping of the Reproducing Kernel Hilbert Space (RKHS) of functions mapping the input space to the output space. While the Hamiltonian interpretation of ResNets is not new, it was based on an Ansatz. We do not rely on this Ansatz and present the first rigorous proof of convergence of ResNets with trained weights and biases towards a Hamiltonian dynamics driven flow. Since our proof is constructive and based on discrete and continuous mechanics, it reveals several remarkable properties of ResNets and their GP generalization. ResNets regressors are kernel regressors with data-dependent warping kernels. Minimizers of L2 regularized ResNets satisfy a discrete least action principle implying the near preservation of the norm of weights and biases across layers. The trained weights of ResNets with scaled/strong L2 regularization can be identified by solving an autonomous Hamiltonian system. The trained ResNet parameters are unique up to (a function of) the initial momentum, and the initial momentum representation of those parameters is generally sparse. The kernel (nugget) regularization strategy provides a provably robust alternative to Dropout for ANNs. We introduce a functional generalization of GPs and show that pointwise GP/RKHS error estimates lead to probabilistic and deterministic generalization error estimates for ResNets. When performed with feature maps, the proposed analysis identifies the (EPDiff) mean fields limit of trained ResNet parameters as the number of data points goes to infinity. The search for good architectures can be reduced to that of good kernels, and we show that the composition of warping regression blocks with reduced equivariant multichannel kernels (introduced here) recovers and generalizes CNNs to arbitrary spaces and groups of transformations.

Confidence intervals for the random forest generalization error

We show that the byproducts of the standard training process of a random forest yield not only the well known and almost computationally free out-of-bag point estimate of the model generalization error, but also open a direct path to compute confidence intervals for the generalization error which avoids processes of data splitting and model retraining. Besides the low computational cost involved in their construction, these confidence intervals are shown through simulations to have good coverage and appropriate shrinking rate of their width in terms of the training sample size.

A Cluster-then-label Approach for Few-shot Learning with Application to Automatic Image Data Labeling

Few-shot learning (FSL) aims at learning to generalize from only a small number of labeled examples for a given target task. Most current state-of-the-art FSL methods typically have two limitations. First, they usually require access to a source dataset (in a similar domain) with abundant labeled examples, which may not always be possible due to privacy concerns and copyright issues. Second, they typically do not offer any estimation of the generalization error on the target FSL task, because the handful of labeled examples must be used for training and cannot spare a validation subset. In this article, we propose a cluster-then-label approach to perform few-shot learning. Our approach does not require access to the labeled source dataset and provides an estimation of generalization error. We show empirically, on four benchmark datasets, that our approach provides competitive predictive performance to state-of-the-art FSL approaches and our generalization error estimation is accurate. Finally, we explore the application of our proposed method to automatic image data labeling. We compare our method with existing automatic data labeling systems. The end-to-end performance of our method outperforms the state-of-the-art automatic data labeling system Snuba by 26% and is only 7% away from the fully supervised upper bound.

Neural network guided adjoint computations in dual weighted residual error estimation

In this work, we are concerned with neural network guided goal-oriented a posteriori error estimation and adaptivity using the dual weighted residual method. The primal problem is solved using classical Galerkin finite elements. The adjoint problem is solved in strong form with a feedforward neural network using two or three hidden layers. The main objective of our approach is to explore alternatives for solving the adjoint problem with greater potential of a numerical cost reduction. The proposed algorithm is based on the general goal-oriented error estimation theorem including both linear and nonlinear stationary partial differential equations and goal functionals. Our developments are substantiated with some numerical experiments that include comparisons of neural network computed adjoints and classical finite element solutions of the adjoints. In the programming software, the open-source library deal.II is successfully coupled with LibTorch, the PyTorch C++ application programming interface.Article HighlightsAdjoint approximation with feedforward neural network in dual-weighted residual error estimation.Side-by-side comparisons for accuracy and computational cost with classical finite element computations.Numerical experiments for linear and nonlinear problems yielding excellent effectivity indices.

A Hybrid High-Order method for incompressible flows of non-Newtonian fluids with power-like convective behaviour

Abstract In this work, we design and analyse a Hybrid High-Order (HHO) discretization method for incompressible flows of non-Newtonian fluids with power-like convective behaviour. We work under general assumptions on the viscosity and convection laws, which are associated with possibly different Sobolev exponents $r\in (1,\infty )$ and $s\in (1,\infty )$. After providing a novel weak formulation of the continuous problem, we study its well-posedness highlighting how a subtle interplay between the exponents $r$ and $s$ determines the existence and uniqueness of a solution. We next design an HHO scheme based on this weak formulation, and perform a comprehensive stability and convergence analysis, including convergence for general data and error estimates for shear-thinning fluids and small data. The HHO scheme is validated on a complete panel of model problems.

A General Error Estimate For Parabolic Variational Inequalities

Abstract The gradient discretisation method (GDM) is a generic framework designed recently, as a discretisation in spatial space, to partial differential equations. This paper aims to use the GDM to establish a first general error estimate for numerical approximations of parabolic obstacle problems. This gives the convergence rates of several well-known conforming and non-conforming numerical methods. Numerical experiments based on the hybrid finite volume method are provided to verify the theoretical results.

Assessment of Characteristics and Conditions before the End of Lockdown

After months of blockades and restriction, the decision of the best time to end the lockdown after the first wave of the COVID-19 pandemic is the big question for health rectors. This study aimed to evaluate the characteristics and conditions for ending the blockade after the first wave of COVID-19. Data on the variables of interest were subjected to linear and non-linear regression studies to determine the curve that best explains the data. The coefficient of determination, the standard deviation of y in x, and the observed curve of the confidence interval were estimated. Regression which was estimated subsequently revealed the trend curve. The study found that all dependent variables tend to decrease over time in a quadratic fashion, except for the variable for new cases. In general, the R2 and mean absolute percentage error (MAPE) estimates were satisfactory: gradual and cautious steps should be taken before ending the lockdown. The results suggested that a surveillance of crucial indicators (e.g., incidence, prevalence, and PCR test positivity) should be maintained before lockdown is terminated. Moreover, the findings indicated that long-term preparations should be made to contain future waves of new cases.

Committee neural network potentials control generalization errors and enable active learning.

It is well known in the field of machine learning that committee models improve accuracy, provide generalization error estimates, and enable active learning strategies. In this work, we adapt these concepts to interatomic potentials based on artificial neural networks. Instead of a single model, multiple models that share the same atomic environment descriptors yield an average that outperforms its individual members as well as a measure of the generalization error in the form of the committee disagreement. We not only use this disagreement to identify the most relevant configurations to build up the model's training set in an active learning procedure but also monitor and bias it during simulations to control the generalization error. This facilitates the adaptive development of committee neural network potentials and their training sets while keeping the number of ab initio calculations to a minimum. To illustrate the benefits of this methodology, we apply it to the development of a committee model for water in the condensed phase. Starting from a single reference ab initio simulation, we use active learning to expand into new state points and to describe the quantum nature of the nuclei. The final model, trained on 814 reference calculations, yields excellent results under a range of conditions, from liquid water at ambient and elevated temperatures and pressures to different phases of ice, and the air-water interface-all including nuclear quantum effects. This approach to committee models will enable the systematic development of robust machine learning models for a broad range of systems.

General least product relative error estimation for multiplicative regression models with or without multiplicative distortion measurement errors

We consider the parameter estimation for multiplicative linear regression models with or without multiplicative distortion measurement errors. For the latter, both the response variable and the covariates are are unobserved and distorted by unknown functions of a commonly observable confounding variable. With or without distortion measurement errors, we propose the general least product relative error estimator, and we discuss the estimation efficiency with the least squares estimators by taking logarithmic transformation. Asymptotic properties for the estimators are established. Simulation studies are conducted to demonstrate the performance of the proposed estimation procedures.

Well-Posedness and Finite Element Approximations for Elliptic SPDEs with Gaussian Noises

The paper studies the well-posedness and optimal error estimates of spectral finite element approximations for the boundary value problems of semi-linear elliptic SPDEs driven by white or colored Gaussian noises. The noise term is approximated through the spectral projection of the covariance operator, which is not required to be commutative with the Laplacian operator. Through the convergence analysis of SPDEs with the noise terms replaced by the projected noises, the well-posedness of the SPDE is established under certain covariance operator-dependent conditions. These SPDEs with projected noises are then numerically approximated with the finite element method. A general error estimate framework is established for the finite element approximations. Based on this framework, optimal error estimates of finite element approximations for elliptic SPDEs driven by power-law noises are obtained. It is shown that with the proposed approach, convergence order of white noise driven SPDEs is improved by half for one-dimensional problems, and by an infinitesimal factor for higher-dimensional problems.

Kernel Stability for Model Selection in Kernel-Based Algorithms.

Model selection is one of the fundamental problems in kernel-based algorithms, which is commonly done by minimizing an estimation of generalization error. The notion of stability and cross-validation (CV) error of learning machines consists of two widely used tools for analyzing the generalization performance. However, there are some disadvantages to both tools when applied for model selection: 1) the stability of learning machines is not practical due to the difficulty of the estimation of its specific value and 2) the CV-based estimate of generalization error usually has a relatively high variance, so it is prone to overfitting. To overcome these two limitations, we present a novel notion of kernel stability (KS) for deriving the generalization error bounds and variance bounds of CV and provide an effective approach to the application of KS for practical model selection. Unlike the existing notions of stability of the learning machine, KS is defined on the kernel matrix; hence, it can avoid the difficulty of the estimation of its value. We manifest the relationship between the KS and the popular uniform stability of the learning algorithm, and further propose several KS-based generalization error bounds and variance bounds of CV. By minimizing the proposed bounds, we present two novel KS-based criteria that can ensure good performance. Finally, we empirically analyze the performance of the proposed criteria on many benchmark data, which demonstrates that our KS-based criteria are sound and effective.