General Error Analysis Research Articles

Feature learning and generalization error analysis of two-layer linear neural networks for high-dimensional inputs

AbstractIt is well known that a model can generalize even when it completely interpolates the training data, which is known as the benign overfitting. Indeed, several work have theoretically revealed that the minimum-norm interpolator can exhibit the benign overfitting. On the other hand, deep learning models such as two-layer neural networks have been reported to outperform “shallow” learning models such as kernel methods under appropriate model sizes by adaptively learning the basis functions to the data. This mechanism is called feature learning, and it is known empirically to be beneficial even when the model size is large. However, it is generally difficult to show that benign overfitting occurs in learning models with feature learning especially for regression problems. In this study, we then analyze the predictive error of the estimator after one step feature learning in a two-layer linear neural network optimized by gradient descent methods and study the effect of feature learning on benign overfitting. The results show that feature learning reduces bias compared to a one-layer linear regression model without feature learning, especially when the eigenvalues of the covariance of input decay slowly. On the other hand, we clarify that the variance is hardly changed by feature learning. This differs significantly from the results for benign overfitting in the situation without feature learning and indicates the usefulness of feature learning.

General error analysis of matrix-operation-mode decomposition technique in few-mode fiber laser

The mode decomposition based on matrix operation (MDMO) is one of the fastest mode decomposition (MD) techniques, which is important to the few-mode fiber laser characterization and its applications. In this paper, the general error of the MDMO technique was analyzed, where different influencing factors, such as position deviation of the optical imaging system, coordinate deviation of the image acquisition system, aberrations, and mode distortion were considered. It is found that the MDMO technique based on far-field intensity distribution is less affected by optical imaging system position deviation, coordinate deviation of the image acquisition system, and mode distortion than those based on direct near-field decomposition. But far-field decomposition is more affected by aberration than those based on near-field decomposition. In particular, the numerical results show that the deviation of the coordinate axis direction is an important factor limiting the accuracy of MD. In addition, replacing the ideal eigenmode basis with a distorted eigenmode basis can effectively suppress the decrease in mode decomposition accuracy caused by fiber bending. Moreover, based on detailed numerical analysis results, fitting formulas for estimating the accuracy of the MDMO technique with imperfections are also provided, which provides a comprehensive method for evaluating the accuracy of the MDMO technique in practical engineering operations.

A Survey on Scenario Theory, Complexity, and Compression-Based Learning and Generalization.

This work investigates formal generalization error bounds that apply to support vector machines (SVMs) in realizable and agnostic learning problems. We focus on recently observed parallels between probably approximately correct (PAC)-learning bounds, such as compression and complexity-based bounds, and novel error guarantees derived within scenario theory. Scenario theory provides nonasymptotic and distributional-free error bounds for models trained by solving data-driven decision-making problems. Relevant theorems and assumptions are reviewed and discussed. We propose a numerical comparison of the tightness and effectiveness of theoretical error bounds for support vector classifiers trained on several randomized experiments from 13 real-life problems. This analysis allows for a fair comparison of different approaches from both conceptual and experimental standpoints. Based on the numerical results, we argue that the error guarantees derived from scenario theory are often tighter for realizable problems and always yield informative results, i.e., probability bounds tighter than a vacuous [0,1] interval. This work promotes scenario theory as an alternative tool for model selection, structural-risk minimization, and generalization error analysis of SVMs. In this way, we hope to bring the communities of scenario and statistical learning theory closer, so that they can benefit from each other's insights.

An Optimal Transport Analysis on Generalization in Deep Learning.

Deep neural networks (DNNs) have achieved state-of-the-art performance in various learning tasks, such as computer vision, natural language processing, and speech recognition. However, the fundamental theory of generalization still remains obscure in deep learning-why DNN models can generalize well, despite that they are heavily overparametrized in both depth and width? Recently, some work shows that traditional theory of analyzing the generalization error of learning models fails to explain the generalization of DNNs. The failure is mainly because of one simple fact that the worse case analysis of generalization error for learning models would be too loose for models with large parameter space, such as DNNs. In this work, we propose a new analysis of generalization in DNNs from an optimal transport perspective. Unlike traditional worse-case uniform convergence analysis in learning theory, our analysis of generalization error is dependent on both the learning algorithm and the data distribution and is the average-case analysis. Thus, our theory can be more practical and accurate to describe the generalization behavior of DNNs. More specifically, in this article, we try to answer a fundamental yet unsolved question in deep learning-why deeper models can generalize well than shallow models? The main contribution of this article can be summarized in four aspects. First, under a general learning framework, we derive upper bounds on the generalization error of learning algorithms by their algorithmic transport cost: the expected Wasserstein distance between the output hypothesis and the output hypothesis conditioned on an input example. We further provide several upper bounds on the algorithmic transport cost in terms of total variation distance, relative entropy, and Vapnik-Chervonenkis (VC) dimension. Moreover, we also study different conditions for loss functions under which the generalization error of a learning algorithm can be upper bounded by different probability metrics between distributions relating to the output hypothesis and/or the input data. Finally, under our established framework, we obtain our main results, showing that the generalization error in DNNs decreases exponentially to zero as the number of layers increases.

Precision Machining Technology of Jewelry on CNC Machine Tool Based on Mathematical Modeling

Abstract This article establishes the actual movement mathematical model of CNC machine tools for the precision processing of jewelry. Through analyzing the general geometric error analysis model of CNC machine tools with less than five axes and the method of solving precision CNC instructions, the operating principle of the CNC machine tools is studied. At the same time, we use a transformation matrix to express the relationship between the various moving bodies. The article abstracts the complex motion relationship between entities as the relationship between mathematical matrices. The experimental results show that the theoretical method proposed in this paper can increase the machining accuracy of the machine tool by more than 50%.

A priori generalization error analysis of two-layer neural networks for solving high dimensional Schrödinger eigenvalue problems

This paper analyzes the generalization error of two-layer neural networks for computing the ground state of the Schrödinger operator on add-dimensional hypercube with Neumann boundary condition. We prove that the convergence rate of the generalization error is independent of dimensiondd, under the a priori assumption that the ground state lies in a spectral Barron space. We verify such assumption by proving a new regularity estimate for the ground state in the spectral Barron space. The latter is achieved by a fixed point argument based on the Krein-Rutman theorem.

Generalization Error Analysis of Neural Networks with Gradient Based Regularization

We study gradient-based regularization methods for neural networks. We mainly focus on two regularization methods: the total variation and the Tikhonov regularization. Applying these methods is equivalent to using neural networks to solve some partial differential equations, mostly in high dimensions in practical applications. In this work, we introduce a general framework to analyze the generalization error of regularized networks. The error estimate relies on two assumptions on the approximation error and the quadrature error. Moreover, we conduct some experiments on the image classification tasks to show that gradient-based methods can significantly improve the generalization ability and adversarial robustness of neural networks. A graphical extension of the gradient-based methods are also considered in the experiments.

Performance Analysis of Propagation in VHF Military Tactical Communication System

The main challenge of military tactical communication systems is the accessibility of relevant information on the particular operating environment required for the determination of the waveform's ideal use. The existing propagation model focuses mainly on broadcasting and commercial wireless communication with a highs transceiver antenna that is not suitable for numerous military tactical communication systems. This paper presents a study of the path loss model related to radio propagation profile within the suburban in Kuala Lumpur. The experimental path loss modeling for VHF propagation was collected from various suburban settings for the 30-88 MHz frequency range. This experiment was highly affected by ecological factors and existing wave propagation effects such as reflection, diffraction, scattering, and Doppler effect. Radio propagation performance is evaluated by collecting received power at the allocated substation and comparing it against existing propagation models. The existing propagation model also will be tuned close to the measurement value by identifying the best path loss exponent to perform a suitable model for a suburban area. Theoretical assessments and analysis of the initial measurement stage for radio propagation show the extensive contribution of radio field from potential obstacles at lower VHF frequencies for both short and medium ranges around there. The explanation indicates the standard radio propagation prediction models that are generally reasonable for the suburban area. From the general error analysis, it is seen that, the performance of the LDPL with adjusting path loss exponent is the suitable model since it has least value of error metrics.

Optimal data collection design in machine learning: the case of the fixed effects generalized least squares panel data model

This work belongs to the strand of literature that combines machine learning, optimization, and econometrics. The aim is to optimize the data collection process in a specific statistical model, commonly used in econometrics, employing an optimization criterion inspired by machine learning, namely, the generalization error conditioned on the training input data. More specifically, the paper is focused on the analysis of the conditional generalization error of the Fixed Effects Generalized Least Squares (FEGLS) panel data model, i.e., a linear regression model with applications in several fields, able to represent unobserved heterogeneity in the data associated with different units, for which distinct observations related to the same unit are corrupted by correlated measurement errors. The framework considered in this work differs from the classical FEGLS model for the additional possibility of controlling the conditional variance of the output variable given the associated unit and input variables, by changing the cost per supervision of each training example. Assuming an upper bound on the total supervision cost, i.e., the cost associated with the whole training set, the trade-off between the training set size and the precision of supervision (i.e., the reciprocal of the conditional variance of the output variable) is analyzed and optimized. This is achieved by formulating and solving in closed form suitable optimization problems, based on large-sample approximations of the generalization error associated with the FEGLS estimates of the model parameters, conditioned on the training input data. The results of the analysis extend to the FEGLS case and to various large-sample approximations of its conditional generalization error the ones obtained by the authors in recent works for simpler linear regression models. They highlight the importance of how the precision of supervision scales with respect to the cost per training example in determining the optimal trade-off between training set size and precision. Numerical results confirm the validity of the theoretical findings.

Nonlinearizing Two-parameter Eigenvalue Problems

We investigate a technique to transform a linear two-parameter eigenvalue problem, into a nonlinear eigenvalue problem (NEP). The transformation stems from an elimination of one of the equations in the two-parameter eigenvalue problem, by considering it as a (standard) generalized eigenvalue problem. We characterize the equivalence between the original and the nonlinearized problem theoretically and show how to use the transformation computationally. Special cases of the transformation can be interpreted as a reversed companion linearization for polynomial eigenvalue problems, as well as a reversed (less known) linearization technique for certain algebraic eigenvalue problems with square-root terms. Moreover, by exploiting the structure of the NEP we present algorithm specializations for NEP methods, although the technique also allows general solution methods for NEPs to be directly applied. The nonlinearization is illustrated in examples and simulations, with focus on problems where the eliminated equation is of much smaller size than the other two-parameter eigenvalue equation. This situation arises naturally in domain decomposition techniques. A general error analysis is also carried out under the assumption that a backward stable eigenvalue solver method is used to solve the eliminated problem, leading to the conclusion that the error is benign in this situation.

Large scale analysis of generalization error in learning using margin based classification methods

Large-margin classifiers are popular methods for classification. We derive the asymptotic expression for the generalization error of a family of large-margin classifiers in the limit of both sample size n and dimension p going to ∞ with fixed ratio α = n/p. This family covers a broad range of commonly used classifiers including support vector machine, distance weighted discrimination, and penalized logistic regression. Our result can be used to establish the phase transition boundary for the separability of two classes. We assume that the data are generated from a single multivariate Gaussian distribution with arbitrary covariance structure. We explore two special choices for the covariance matrix: spiked population model and two layer neural networks with random first layer weights. The method we used for deriving the closed-form expression is from statistical physics known as the replica method. Our asymptotic results match simulations already when n, p are of the order of a few hundreds. For two layer neural networks, we reproduce the recently observed ‘double descent’ phenomenology for several classification models. We also discuss some statistical insights that can be drawn from these analysis.

High‐order convergent methods for singularly perturbed quasilinear problems with integral boundary conditions

In this work, we develop a numerical scheme for a class of singularly perturbed quasilinear problems with integral boundary conditions. The quasilinear equation is discretized using a hybrid scheme, and the composite trapezoidal rule is used to discretize the boundary condition. We construct a general error analysis framework for the discrete scheme. Within this framework, the discrete scheme is shown to be uniformly convergent of on Shishkin meshes and on Bakhvalov meshes. Further, we propose adaptive generation of meshes based on a suitable monitor function and the mesh equidistribution principle. We prove that on these meshes the discrete scheme is uniformly convergent of Our theoretical findings are supported by numerical results obtained through experiments.

Metric learning-guided k nearest neighbor multilabel classifier

Multilabel classification deals with the problem where each instance belongs to multiple labels simultaneously. The algorithm based on large margin loss with k nearest neighbor constraints (LM-kNN) is one of the most prominent multilabel classification algorithms. However, due to the use of square hinge loss, LM-kNN needs to iteratively solve a constrained quadratic programming at a high computational cost. To address this issue, we propose a novel metric learning-guided k nearest neighbor approach (MLG-kNN) for multilabel classification. Specifically, we first transform the original instance into the label space by least square regression. Then, we learn a metric matrix in the label space, which makes the predictions of an instance in the learned metric space close to its true class values while far away from others. Since our MLG-kNN can be formulated as an unconstrained strictly (geodesically) convex optimization problem and yield a closed-form solution, the computational complexity is reduced. An analysis of generalization error bound indicates that our MLG-kNN converges to the optimal solutions. Experimental results verify that the proposed approach is more effective than the existing ones for multilabel classification across many benchmark datasets.

A Distributed Rough Evidential K-NN Classifier: Integrating Feature Reduction and Classification

The Evidential K-Nearest Neighbor (EK-NN) classification rule provides a global treatment of imperfect knowledge in class labels, but still suffers from the curse of dimensionality as well as runtime and memory restrictions when performing nearest neighbors search, in particular for large and high-dimensional data. To avoid the curse of dimensionality, this article first proposes a rough evidential K-NN (REK-NN) classification rule in the framework of rough set theory. Based on a reformulated K-NN rough set model, REK-NN selects features and thus reduces complexity by minimizing a proposed neighborhood pignistic decision error rate, which considers both Bayes decision error and spatial information among samples in feature space. In contrast to existing rough set-based feature selection methods, REK-NN is a synchronized rule rather than a stepwise one, in the sense that feature selection and learning are performed simultaneously. In order to further handle data with large sample size, we derive a distributed REK-NN method and implement it in the Apache Spark. The theoretical analysis of the classifier generalization error bound is finally presented. It is shown that the distributed REK-NN achieves good performances while drastically reducing the number of features and consuming less runtime and memory. Numerical experiments conducted on real-world datasets validate our conclusions.

Generalization error analysis for deep convolutional neural network with transfer learning in breast cancer diagnosis

Deep convolutional neural network (DCNN), now popularly called artificial intelligence (AI), has shown the potential to improve over previous computer-assisted tools in medical imaging developed in the past decades. A DCNN has millions of free parameters that need to be trained, but the training sample set is limited in size for most medical imaging tasks so that transfer learning is typically used. Automatic data mining may be an efficient way to enlarge the collected data set but the data can be noisy such as incorrect labels or even a wrong type of image. In this work we studied the generalization error of DCNN with transfer learning in medical imaging for the task of classifying malignant and benign masses on mammograms. With a finite available data set, we simulated a training set containing corrupted data or noisy labels. The balance between learning and memorization of the DCNN was manipulated by varying the proportion of corrupted data in the training set. The generalization error of DCNN was analyzed by the area under the receiver operating characteristic curve for the training and test sets and the weight changes after transfer learning. The study demonstrates that the transfer learning strategy of DCNN for such tasks needs to be designed properly, taking into consideration the constraints of the available training set having limited size and quality for the classification task at hand, to minimize memorization and improve generalizability.

Operator-theoretic framework for forecasting nonlinear time series with kernel analog techniques

Kernel analog forecasting (KAF), alternatively known as kernel principal component regression, is a kernel method used for nonparametric statistical forecasting of dynamically generated time series data. This paper synthesizes descriptions of kernel methods and Koopman operator theory in order to provide a single consistent account of KAF. The framework presented here illuminates the property of the KAF method that, under measure-preserving and ergodic dynamics, it consistently approximates the conditional expectation of observables that are acted upon by the Koopman operator of the dynamical system and are conditioned on the observed data at forecast initialization. More precisely, KAF yields optimal predictions, in the sense of minimal root mean square error with respect to the invariant measure, in the asymptotic limit of large data. The presented framework facilitates, moreover, the analysis of generalization error and quantification of uncertainty. Extensions of KAF to the construction of conditional variance and conditional probability functions, as well as to non-symmetric kernels, are also shown. Illustrations of various aspects of KAF are provided with applications to simple examples, namely a periodic flow on the circle and the chaotic Lorenz 63 system.

Enriched two dimensional mixed finite element models for linear elasticity with weak stress symmetry

The purpose of this article is to derive and analyze new discrete mixed approximations for linear elasticity problems with weak stress symmetry. These approximations are based on the application of enriched versions of classic Poisson-compatible spaces, for stress and displacement variables, and/or on enriched Stokes-compatible space configurations, for the choice of rotation spaces used to weakly enforce stress symmetry. Accordingly, the stress space has to be adapted to ensure stability. Such enrichment procedures are done via space increments with extra bubble functions, which have their support on a single element (in the case of H1-conforming approximations) or with vanishing normal components over element edges (in the case of H(div)-conforming spaces). The advantage of using bubbles as stabilization corrections relies on the fact that all extra degrees of freedom can be condensed, in a way that the number of equations to be solved and the matrix structure are not affected. Enhanced approximations are observed when using the resulting enriched space configurations, which may have different orders of accuracy for the different variables. A general error analysis is derived in order to identify the contribution of each kind of bubble increment on the accuracy of the variables, individually. The use of enriched Poisson spaces improves the rates of convergence of stress divergence and displacement variables. Stokes enhancement by bubbles contributes to equilibrate the accuracy of weak stress symmetry enforcement with the stress approximation order, reaching the maximum rate given by the normal traces (which are not affected).

Regularized modal regression with data-dependent hypothesis spaces

Modal regression aims at learning the conditional mode function, which is different from the traditional least-squares for approximating the conditional mean function. Due to its robust to complex noise and outliers, modal regression has attracted increasing attention recently in statistics and machine learning community. However, most of the previous modal regression models are limited to learning framework with data-independent hypothesis spaces. Usually, the data-dependent hypothesis spaces can provide much flexibility and adaptivity for many learning problems. By employing data-dependent hypothesis spaces, we propose a new regularized modal regression and establish its generalization error analysis. Data experiments demonstrate the competitive performance of the proposed model over the related least-squares regression.

Fast switch and spline scheme for accurate inversion of nonlinear functions: The new first choice solution to Kepler’s equation

Numerically obtaining the inverse of a function is a common task for many scientific problems, often solved using a Newton iteration method. Here we describe an alternative scheme, based on switching variables followed by spline interpolation, which can be applied to monotonic functions under very general conditions. To optimize the algorithm, we designed a specific ultra-fast spline routine. We also derive analytically the theoretical errors of the method and test it on examples that are of interest in physics. In particular, we compute the real branch of Lambert’s W(y) function, which is defined as the inverse of xexp (x), and we solve Kepler’s equation. In all cases, our predictions for the theoretical errors are in excellent agreement with our numerical results, and are smaller than what could be expected from the general error analysis of spline interpolation by many orders of magnitude, namely by an astonishing 3×10−22 factor for the computation of W in the range W(y) ∈ [0, 10], and by a factor 2×10−4 for Kepler’s problem. In our tests, this scheme is much faster than Newton-Raphson method, by a factor in the range 10−4–10−3 for the execution time in the examples, when the values of the inverse function over an entire interval or for a large number of points are requested. For Kepler’s equation and tolerance 10−6 rad, the algorithm outperforms Newton’s method for all values of the number of points N ≥ 2.

A multiscale hybrid method for Darcy’s problems using mixed finite element local solvers

Multiscale Hybrid Mixed (MHM) method refers to a numerical technique targeted to approximate systems of differential equations with strongly varying solutions. For fluid flow, normal fluxes (multiplier) over macro element boundaries, and coarse piecewise constant potential approximations in each macro element are computed (upscaling). Then, small details are resolved by local problems, using fine representations inside the macro elements, setting the multiplier as Neumann boundary conditions (downscaling). In this work a variant of the method is developed, denoted by MHM- H(div), adopting mixed finite elements at the downscaling stage, instead of continuous finite elements used in all previous publications of the method. Thus, this alternative MHM method inherits improvements typical of mixed methods, as better flux accuracy, and local mass conservation at the micro scale level inside the macro elements which are important properties for multi-phase flows in rough heterogeneous media. Different two-scale stable space settings are considered. Vector face functions are supposed to have normal components restricted to a given finite dimensional trace space defined over the macro element boundaries. In each macro element, the internal flux components, with vanishing normal traces, and the potential approximations, may be enriched in different extents: with respect to internal mesh size, internal polynomial degree, or both, the choice being determined by the problem at hands. A unified general error analysis of the MHM-H(div) method is presented for all these two-scale space scenarios. Both MHM versions are compared for 2D test problems, with smooth solutions, for convergence rates verification, and for Darcy’s flow in heterogeneous media. MHM-H(div) 3D simulations are presented for a known singular Darcy’s solution, using adaptive macro partitions, and for an oscillatory permeability scenario.