Standard Regularization Research Articles

Drop-Activation: Implicit Parameter Reduction and Harmonious Regularization

Overfitting frequently occurs in deep learning. In this paper, we propose a novel regularization method called drop-activation to reduce overfitting and improve generalization. The key idea is to drop nonlinear activation functions by setting them to be identity functions randomly during training time. During testing, we use a deterministic network with a new activation function to encode the average effect of dropping activations randomly. Our theoretical analyses support the regularization effect of drop-activation as implicit parameter reduction and verify its capability to be used together with batch normalization (Ioffe and Szegedy in Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167 , 2015). The experimental results on CIFAR10, CIFAR100, SVHN, EMNIST, and ImageNet show that drop-activation generally improves the performance of popular neural network architectures for the image classification task. Furthermore, as a regularizer drop-activation can be used in harmony with standard training and regularization techniques such as batch normalization and AutoAugment (Cubuk et al. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 113–123, 2019). The code is available at https://github.com/LeungSamWai/Drop-Activation .

Video and image classification using atomisation spray image patterns and deep learning

Atomisation of agricultural spray solutions through nozzles is a mechanism for delivering pesticides to their target site. Previous research using mechanistic methods has shown that oil-in-water formulations produce more placid liquid sheets and succumb to a breakup mechanism that begins when holes nucleate within the liquid sheet caused by inhomogeneities of the spray system. The relationships between image analysis metrics obtained from spray images and the resulting atomisation droplet size were explored to estimate drop size statistics from image features. Breakup patterns for oil-in-water formulations obtained from high-speed videos were used with deep learning techniques (convolutional neural networks (CNN), Monte Carlo (MC) dropout methodology and recurrent neural networks (RNN)) to predict droplet size. The videos used were part of a statistical design of experiment (DOE) reported by Cryer et al., (2020). A total of 353,694 train images and 174,208 test images of atomisation (83 videos, 44 classes) were used. Prediction time for the CNN model with standard dropout regularisation was rapid and achieved 99.4% accuracy, whilst the CNN with MC dropout took longer depending on how many forward passes were used but achieved 99.8% and had the ability to estimate uncertainty. The 3D convolution model (99.2%) provided a simpler alternative than the CNN + RNN model (99.6%) at reduced CPU intervals. Therefore, any of these deep learning methods can be suitable to address the prediction atomisation spray performance based upon images of oil-in-water emulsion spray patterns at the nozzle.

Augmented Tikhonov Regularization Method for Dynamic Load Identification

We introduce the augmented Tikhonov regularization method motivated by Bayesian principle to improve the load identification accuracy in seriously ill-posed problems. Firstly, the Green kernel function of a structural dynamic response is established; then, the unknown external loads are identified. In order to reduce the identification error, the augmented Tikhonov regularization method is combined with the Green kernel function. It should be also noted that we propose a novel algorithm to determine the initial values of the regularization parameters. The initial value is selected by finding a local minimum value of the slope of the residual norm. To verify the effectiveness and the accuracy of the proposed method, three experiments are performed, and then the proposed algorithm is used to reproduce the experimental results numerically. Numerical comparisons with the standard Tikhonov regularization method show the advantages of the proposed method. Furthermore, the presented results show clear advantages when dealing with ill-posedness of the problem.

Remarks on the extension group for purely infinite corona algebras

ABSTRACT We study BDF theory in the context of (not necessarily simple) purely infinite corona algebras. Let be a nonunital separable simple C*-algebra with standard regularity properties for which is purely infinite (though not necessarily simple). Let be a separable nuclear C*-algebra. We prove a BDF Voiculescu decomposition theorem for maps from to , and use this to prove that is a group. Then, restricting to the case , for some compact metric space X, and has continuous scale, we provide characterizations of the neutral element for .

Hoeffding-Blum-Kiefer-Rosenblatt independence test statistic on partly not identically distributed data

The established Hoeffding-Blum-Kiefer-Rosenblatt independence test statistic is investigated for partly not identically distributed data. Surprisingly, it turns out that the statistic has the well-known distribution-free limiting null distribution of the classical criterion under standard regularity conditions. An application is testing goodness-of-fit for the regression function in a non parametric random effects meta-regression model, where the consistency is obtained as well. Simulations investigate size and power of the approach for small and moderate sample sizes. A real data example based on clinical trials illustrates how the test can be used in applications.

Variable Selection Using Nonlocal Priors in High-Dimensional Generalized Linear Models With Application to fMRI Data Analysis.

High-dimensional variable selection is an important research topic in modern statistics. While methods using nonlocal priors have been thoroughly studied for variable selection in linear regression, the crucial high-dimensional model selection properties for nonlocal priors in generalized linear models have not been investigated. In this paper, we consider a hierarchical generalized linear regression model with the product moment nonlocal prior over coefficients and examine its properties. Under standard regularity assumptions, we establish strong model selection consistency in a high-dimensional setting, where the number of covariates is allowed to increase at a sub-exponential rate with the sample size. The Laplace approximation is implemented for computing the posterior probabilities and the shotgun stochastic search procedure is suggested for exploring the posterior space. The proposed method is validated through simulation studies and illustrated by a real data example on functional activity analysis in fMRI study for predicting Parkinson’s disease.

Identification-Robust Subvector Inference

This paper introduces identification-robust subvector tests and confidence sets (CS’s) that have asymptotic size equal to their nominal size and are asymptotically efficient under strong identification. Hence, inference is as good asymptotically as standard methods under standard regularity conditions, but also is identification robust. The results do not require special structure on the models under consideration, or strong identification of the nuisance parameters, as many existing methods do. We provide general results under high-level conditions that can be applied to moment condition, likelihood, and minimum distance models, among others. We verify these conditions under primitive conditions for moment condition models. In another paper, we do so for likelihood models. The results build on the approach of Chaudhuri and Zivot (2011), who introduce a C(α)-type Lagrange multiplier test and employ it in a Bonferroni subvector test. Here we consider two-step tests and CS’s that employ a C(α)-type test in the second step. The two-step tests are closely related to Bonferroni tests, but are not asymptotically conservative and achieve asymptotic efficiency under strong identification.

Mesh-free error integration in arbitrary dimensions: A numerical study of discrepancy functions

We are interested in mesh-free formulas based on the Monte-Carlo methodology for the approximation of multi-dimensional integrals, and we investigate their accuracy when the functions belong to a reproducing-kernel space. A kernel typically captures regularity and qualitative properties of functions “beyond” the standard Sobolev regularity class. We are interested in the issue whether quantitative error bounds can be a priori guaranteed in applications (including mathematical finance, scientific computing, and machine learning). Our main contribution is a numerical study of the error discrepancy function based on a comparison between several numerical strategies, when one varies the choice of the kernel, the number of approximation points, and the dimension of the problem. We consider two strategies in order to localize to a bounded set the standard kernels defined in the whole Euclidian space (exponential, multiquadric, Gaussian, truncated), namely, on one hand the class of periodic kernels defined via a discrete Fourier transform on a lattice and, on the other hand, a class of transport-based kernels. Relying on the Poisson formula on a lattice together with heuristic arguments, we study the derivation of theoretical bounds for the discrepancy function of periodic kernels. Then, for four kernels of particular interest we perform extensive numerical experiments and generate the optimal distributions of points and the discrepancy error functions. Our numerical results validate our theoretical observations and, importantly, provide us with quantitative estimates for the error made with a kernel-based strategy as opposed to a purely random strategy.

On universal stabilization property of Interconnection and Damping Assignment Control

It is well-known that the widely popular Interconnection and Damping Assignment Passivity-based Control (IDA-PBC) is “universally stabilizing”, in the sense that it generates all asymptotically stabilizing controllers for general nonlinear systems of the form ẋ=f(x,u). Unfortunately, the proof of this fact relies in the construction of a control law that is not globally defined. Although a standard regularization procedure may be used to enforce the required smoothness properties to the control signal, this procedure is quite technical and not constructive. To overcome this problem we provide an alternative construction that yields a continuous IDA-PBC law.

A new regularization approach for numerical differentiation

It is well known that the problem of numerical differentiation is an ill-posed problem and one requires regularization methods to approximate the solution. The commonly practiced regularization methods are (external) parameter-based like Tikhonov regularization, which has certain inherent difficulties associated with them. In such scenarios, iterative regularization methods serve as an attractive alternative. In this paper, we propose a novel iterative regularization method where the minimizing functional does not contain the noisy data directly, but rather a smoothed or integrated version of it. The advantage, in addition to circumventing the use of noisy data directly, is that the sequence of functions constructed during the descent process tends to avoid overfitting, and hence, does not corrupt the recovery significantly. To demonstrate the effectiveness of our method we compare the numerical results obtained from our method with the numerical results obtained from certain standard regularization methods such as Tikhonov regularization, Total-variation, etc.

Identification of stochastic nonlinear models using optimal estimating functions

The first part of the paper examines the asymptotic properties of linear prediction error method estimators, which were recently suggested for the identification of nonlinear stochastic dynamical models. It is shown that their accuracy depends not only on how the model is parameterized, but also on the shape of the unknown distribution of the data. Therefore, it is not obvious in general which linear prediction error method should be preferred. In the second part, the estimating functions approach is introduced and used to construct estimators that are asymptotically optimal with respect to a specific class of estimators. These estimators rely on partial probabilistic parametric models, and therefore neither require the computations of the likelihood function nor any marginalization integrals. The convergence and consistency of the proposed estimators are established under standard regularity and identifiability assumptions akin to those of prediction error methods. The paper is concluded by several numerical simulation examples.

Deep neural networks based binary classification for single channel speaker independent multi-talker speech separation

Speech separation is an important task of separating a target speech from the mixture signals. Speaker-independent multi-talker speech separation is a challenging task due to unpredictability of the target and interfering speech in the target-interference mixtures. Conventionally, speech separation is used as a signal processing problem, but recently it is formulated as a deep learning problem and discriminative patterns of the speech are learned from the training data. In this paper, we consider the ideal binary mask (IBM) as a supervised binary classification training-target by using fully connected deep neural networks (DNN) for single-channel speaker-independent multi-talker speech separation. The train DNNs is used to estimate IBM training-target. The mean square error (MSE) is used as an objective cost function. Standard backpropagation and Monte-Carlo dropout regularization approaches are used for better generalization and overfitting during training. The estimated training-target is applied to the mixtures to obtain the separated target speech. We have addressed the over-smoothing problem and performed equalization of spectral variances to match the estimated and clean speech features. Our experimental results in various evaluating conditions report that the proposed method outperformed the competing methods in terms of the Perceptual Evaluation of Speech Quality (PESQ), Segmental SNR (SNRSeg), Short-time objective intelligibility (STOI), normalized Frequency weighted SNRSeg (nFwSNRSeg) and HIT-FA rates.

Diffraction tomography with a deep image prior.

We present a tomographic imaging technique, termed Deep Prior Diffraction Tomography (DP-DT), to reconstruct the 3D refractive index (RI) of thick biological samples at high resolution from a sequence of low-resolution images collected under angularly varying illumination. DP-DT processes the multi-angle data using a phase retrieval algorithm that is extended by a deep image prior (DIP), which reparameterizes the 3D sample reconstruction with an untrained, deep generative 3D convolutional neural network (CNN). We show that DP-DT effectively addresses the missing cone problem, which otherwise degrades the resolution and quality of standard 3D reconstruction algorithms. As DP-DT does not require pre-captured data or pre-training, it is not biased towards any particular dataset. Hence, it is a general technique that can be applied to a wide variety of 3D samples, including scenarios in which large datasets for supervised training would be infeasible or expensive. We applied DP-DT to obtain 3D RI maps of bead phantoms and complex biological specimens, both in simulation and experiment, and show that DP-DT produces higher-quality results than standard regularization techniques. We further demonstrate the generality of DP-DT, using two different scattering models, the first Born and multi-slice models. Our results point to the potential benefits of DP-DT for other 3D imaging modalities, including X-ray computed tomography, magnetic resonance imaging, and electron microscopy.

Absum: Simple Regularization Method for Reducing Structural Sensitivity of Convolutional Neural Networks

We propose Absum, which is a regularization method for improving adversarial robustness of convolutional neural networks (CNNs). Although CNNs can accurately recognize images, recent studies have shown that the convolution operations in CNNs commonly have structural sensitivity to specific noise composed of Fourier basis functions. By exploiting this sensitivity, they proposed a simple black-box adversarial attack: Single Fourier attack. To reduce structural sensitivity, we can use regularization of convolution filter weights since the sensitivity of linear transform can be assessed by the norm of the weights. However, standard regularization methods can prevent minimization of the loss function because they impose a tight constraint for obtaining high robustness. To solve this problem, Absum imposes a loose constraint; it penalizes the absolute values of the summation of the parameters in the convolution layers. Absum can improve robustness against single Fourier attack while being as simple and efficient as standard regularization methods (e.g., weight decay and L1 regularization). Our experiments demonstrate that Absum improves robustness against single Fourier attack more than standard regularization methods. Furthermore, we reveal that robust CNNs with Absum are more robust against transferred attacks due to decreasing the common sensitivity and against high-frequency noise than standard regularization methods. We also reveal that Absum can improve robustness against gradient-based attacks (projected gradient descent) when used with adversarial training.

Non-convex sparse regularization for impact force identification

Many convex regularization methods, such as the classical Tikhonov regularization based on l2-norm penalty and the standard sparse regularization method based on l1-norm penalty, have been widely investigated for impact force identification. However, in many practical applications, these regularization methods commonly underestimate the true solution. In this paper, we propose a non-convex sparse regularization method based on lp-norm (0 < p < 1) penalty, to seek the sufficiently sparse and highly accurate solution of impact force identification. Firstly, a non-convex optimization model based on lp-norm penalty instead of l2-norm penalty or l1-norm penalty is developed for regularizing inverse problems of impact force identification to overcome the mismatch between l0-norm and l1-norm regularizations. Secondly, an iteratively reweighed l1-norm algorithm is introduced to solve such a non-convex model through transforming it into a series of l1-norm regularizations. Finally, numerical simulation and experimental validation are carried out to evaluate the effectiveness and performance of lp-norm regularization when 0 = p ≤ 1. Our numerical results demonstrate that, compared to the state-of-the-art Tikhonov regularization and l1-norm regularization, the solution of lp-norm regularization achieves more stability, sparseness and accuracy when dealing with the heavily noisy response. Additionally, both numerical and experimental results demonstrate that the peak relative error of the identified impact force using lp-norm regularization has a decreasing tendency as p is approaching 0; while the identification of lp-norm regularization with 0 = p ≤ 1/2 bears no significant difference, which always outperforms that the identification with 1/2 = p ≤ 1.

Where did the tumor start? An inverse solver with sparse localization for tumor growth models

We present a numerical scheme for solving an inverse problem for parameter estimation in tumor growth models for glioblastomas, a form of aggressive primary brain tumor. The growth model is a reaction–diffusion partial differential equation (PDE) for the tumor concentration. We use a PDE-constrained optimization formulation for the inverse problem. The unknown parameters are the reaction coefficient (proliferation), the diffusion coefficient (infiltration), and the initial condition field for the tumor PDE. Segmentation of magnetic resonance imaging (MRI) scans drive the inverse problem where segmented tumor regions serve as partial observations of the tumor concentration. Like most cases in clinical practice, we use data from a single time snapshot. Moreover, the precise time relative to the initiation of the tumor is unknown, which poses an additional difficulty for inversion. We perform a frozen-coefficient spectral analysis and show that the inverse problem is severely ill-posed. We introduce a biophysically motivated regularization on the structure and magnitude of the tumor initial condition. In particular, we assume that the tumor starts at a few locations (enforced with a sparsity constraint on the initial condition of the tumor) and that the initial condition magnitude in the maximum norm is equal to one. We solve the resulting optimization problem using an inexact quasi-Newton method combined with a compressive sampling algorithm for the sparsity constraint. Our implementation uses PETSc and AccFFT libraries. We conduct numerical experiments on synthetic and clinical images to highlight the improved performance of our solver over a previously existing solver that uses standard two-norm regularization for the calibration parameters. The existing solver is unable to localize the initial condition. Our new solver can localize the initial condition and recover infiltration and proliferation. In clinical datasets (for which the ground truth is unknown), our solver results in qualitatively different solutions compared to the two-norm regularized solver.

A practical way to regularize unfolding of sharply varying spectra with low data statistics

Unfolding is a well-established tool in particle physics. However, a naive application of the standard regularization techniques to unfold the momentum spectrum of protons ejected in the process of negative muon nuclear capture led to a result exhibiting unphysical artifacts. A finite data sample limited the range in which unfolding can be performed, thus introducing a cutoff. A sharply falling “true” distribution led to low data statistics near the cutoff, which exacerbated the regularization bias and produced an unphysical spike in the resulting spectrum. An improved approach has been developed to address these issues and is illustrated using a toy model. The approach uses full Poisson likelihood of data, and produces a continuous, physically plausible, unfolded distribution. The new technique has a broad applicability since sharply falling spectra are common.

LASSO multi-objective learning algorithm for feature selection

This work proposes a new algorithm for training neural networks to solve the problems of feature selection and function approximation. The algorithm applies different weight constraint functions for the hidden and the output layers of a multilayer perceptron neural network. The LASSO operator is applied to the hidden layer; therefore, the training provides automatic selection of relevant features and the standard norm regularization function is applied to the output layer. Therefore, we propose a multi-objective training algorithm that is able to select the important features while solving the approximation problem.

High‐Resolution Gravity Field Models from GRAIL Data and Implications for Models of the Density Structure of the Moon's Crust

AbstractWe present our latest high‐resolution lunar gravity field model of degree and order 1200 in spherical harmonics using Gravity Recovery and Interior Laboratory (GRAIL) data. In addition to a model with the standard spectral Kaula regularization constraint, we determine models by applying a constraint based on topography called rank‐minus‐one (RM1). The new models using this RM1 constraint have high correlations with topography over the entire degree range by design. The RM1 models allow the determination of apparent crustal densities at all spatial scales (called effective density) covered by the model, whereas the Kaula‐constrained model can only be used globally up to spherical harmonic degree 700. We find that the effective density spectrum has a smaller slope for the high degrees when compared to the medium degrees. We interpret this as indicative of a global average surface density, as opposed to an ever‐decreasing effective density as one approaches the surface. We use the RM1 models to derive maps of lateral and vertical density variations in the lunar crust. These models allow us to increase the resolution of this analysis compared to previous studies, by increasing the degree range over which to fit theoretical models of vertical density variations, and by decreasing the size of the spherical caps used in a localized analysis. Several regions on the Moon, such as South Pole‐Aitken and Mare Orientale, are distinct from their surroundings in terms of surface densities. The RM1 models are especially valuable in (localized) spectral studies of the structure of the lunar crust.

An a-priori error analysis for discontinuous Lagrangian finite elements applied to nonconforming dual-mixed formulations: Poisson and Stokes problems

In this paper, we discuss the well-posedness of a mixed discontinuous Galerkin (DG) scheme for the Poisson and Stokes problems in 2D, considering only piecewise Lagrangian finite elements. The complication here lies in the fact that the classical Babuška-Brezzi theory is difficult to verify for low-order finite elements, so we proceed in a non-standard way. First, we prove uniqueness, and then we apply a discrete version of Fredholm's alternative theorem to ensure existence. The a-priori error analysis is done by introducing suitable projections of the exact solution. As a result, we prove that the method is convergent, and, under standard additional regularity assumptions on the exact solution, the optimal rate of convergence of the method is guaranteed.