Least-squares Loss Research Articles

Regularization by double complementary priors for full waveform inversion

Full-waveform inversion (FWI) represents an advanced geophysical imaging technique focused on intricately depicting subsurface physical properties by iteratively minimizing the differences between the simulated and observed seismograms. Unfortunately, the conventional FWI utilizing a least-squares loss function suffers from various drawbacks, including the challenge of local minima and the necessity for human intervention in parameter fine-tuning. It is particularly problematic when handling noisy data and inadequate initial models. Recent works have exhibited promising performance in two-dimensional FWI by integrating structural sparse representation to procure adaptive dictionaries. Drawing inspiration from the competitiveness of structural sparse representation, we introduce a paradigm of group sparse residuals that integrates two types of complementary prior information by harnessing both the internal and external subsurface media models. The proposed algorithm is based on an alternate minimization algorithm to guarantee workflow flexibility and efficient optimization capabilities. We experimentally validate our method for two baseline geological models, and a comparison of the results demonstrates that the proposed algorithm faithfully recovers the velocity models and consistently outperforms other traditional or learning-based algorithms. A further benefit from the group sparse coding used in this method is that it reduces the sensitivity to data noise.

Robust Tensor Completion via Capped Frobenius Norm.

Tensor completion (TC) refers to restoring the missing entries in a given tensor by making use of the low-rank structure. Most existing algorithms have excellent performance in Gaussian noise or impulsive noise scenarios. Generally speaking, the Frobenius-norm-based methods achieve excellent performance in additive Gaussian noise, while their recovery severely degrades in impulsive noise. Although the algorithms using the lp -norm ( ) or its variants can attain high restoration accuracy in the presence of gross errors, they are inferior to the Frobenius-norm-based methods when the noise is Gaussian-distributed. Therefore, an approach that is able to perform well in both Gaussian noise and impulsive noise is desired. In this work, we use a capped Frobenius norm to restrain outliers, which corresponds to a form of the truncated least-squares loss function. The upper bound of our capped Frobenius norm is automatically updated using normalized median absolute deviation during iterations. Therefore, it achieves better performance than the lp -norm with outlier-contaminated observations and attains comparable accuracy to the Frobenius norm without tuning parameter in Gaussian noise. We then adopt the half-quadratic theory to convert the nonconvex problem into a tractable multivariable problem, that is, convex optimization with respect to (w.r.t.) each individual variable. To address the resultant task, we exploit the proximal block coordinate descent (PBCD) method and then establish the convergence of the suggested algorithm. Specifically, the objective function value is guaranteed to be convergent while the variable sequence has a subsequence converging to a critical point. Experimental results based on real-world images and videos exhibit the superiority of the devised approach over several state-of-the-art algorithms in terms of recovery performance. MATLAB code is available at https://github.com/Li-X-P/Code-of-Robust-Tensor-Completion.

Adaptive Multimodel Knowledge Transfer Matrix Machine for EEG Classification.

The emerging matrix learning methods have achieved promising performances in electroencephalogram (EEG) classification by exploiting the structural information between the columns or rows of feature matrices. Due to the intersubject variability of EEG data, these methods generally need to collect a large amount of labeled individual EEG data, which would cause fatigue and inconvenience to the subjects. Insufficient subject-specific EEG data will weaken the generalization capability of the matrix learning methods in neural pattern decoding. To overcome this dilemma, we propose an adaptive multimodel knowledge transfer matrix machine (AMK-TMM), which can selectively leverage model knowledge from multiple source subjects and capture the structural information of the corresponding EEG feature matrices. Specifically, by incorporating least-squares (LS) loss with spectral elastic net regularization, we first present an LS support matrix machine (LS-SMM) to model the EEG feature matrices. To boost the generalization capability of LS-SMM in scenarios with limited EEG data, we then propose a multimodel adaption method, which can adaptively choose multiple correlated source model knowledge with a leave-one-out cross-validation strategy on the available target training data. We extensively evaluate our method on three independent EEG datasets. Experimental results demonstrate that our method achieves promising performances on EEG classification.

Out-of-sample error estimation for M-estimators with convex penalty

Abstract A generic out-of-sample error estimate is proposed for $M$-estimators regularized with a convex penalty in high-dimensional linear regression where $(\boldsymbol{X},\boldsymbol{y})$ is observed and the dimension $p$ and sample size $n$ are of the same order. The out-of-sample error estimate enjoys a relative error of order $n^{-1/2}$ in a linear model with Gaussian covariates and independent noise, either non-asymptotically when $p/n\le \gamma $ or asymptotically in the high-dimensional asymptotic regime $p/n\to \gamma ^{\prime}\in (0,\infty )$. General differentiable loss functions $\rho $ are allowed provided that the derivative of the loss is 1-Lipschitz; this includes the least-squares loss as well as robust losses such as the Huber loss and its smoothed versions. The validity of the out-of-sample error estimate holds either under a strong convexity assumption, or for the L1-penalized Huber M-estimator and the Lasso under a sparsity assumption and a bound on the number of contaminated observations. For the square loss and in the absence of corruption in the response, the results additionally yield $n^{-1/2}$-consistent estimates of the noise variance and of the generalization error. This generalizes, to arbitrary convex penalty and arbitrary covariance, estimates that were previously known for the Lasso.

Local Geometry of Nonconvex Spike Deconvolution From Low-Pass Measurements

Spike deconvolution is the problem of recovering the point sources from their convolution with a known point spread function, which plays a fundamental role in many sensing and imaging applications. In this paper, we investigate the local geometry of recovering the parameters of point sources$\unicode{x2014}$including both amplitudes and locations$\unicode{x2014}$by minimizing a natural nonconvex least-squares loss function measuring the observation residuals. We propose preconditioned variants of gradient descent (GD), where the search direction is scaled via some carefully designed preconditioning matrices. We begin with a simple fixed preconditioner design, which adjusts the learning rates of the locations at a different scale from those of the amplitudes, and show it achieves a linear rate of convergence$\unicode{x2014}$in terms of entrywise errors$\unicode{x2014}$when initialized close to the ground truth, as long as the separation between the true spikes is sufficiently large. However, the convergence rate slows down significantly when the dynamic range of the source amplitudes is large. To bridge this issue, we introduce an adaptive preconditioner design, which compensates for the learning rates of different sources in an iteration-varying manner based on the current estimate. The adaptive design provably leads to an accelerated convergence rate that is independent of the dynamic range, highlighting the benefit of adaptive preconditioning in nonconvex spike deconvolution. Numerical experiments are provided to corroborate the theoretical findings.

Solving multiscale steady radiative transfer equation using neural networks with uniform stability

This paper concerns solving the steady radiative transfer equation with diffusive scaling, using the physics informed neural networks (PINNs). The idea of PINNs is to minimize a least-square loss function, that consists of the residual from the governing equation, the mismatch from the boundary conditions, and other physical constraints such as conservation. It is advantageous of being flexible and easy to execute, and brings the potential for high dimensional problems. Nevertheless, due the presence of small scales, the vanilla PINNs can be extremely unstable for solving multiscale steady transfer equations. In this paper, we propose a new formulation of the loss based on the macro-micro decomposition. We prove that, the new loss function is uniformly stable with respect to the small Knudsen number in the sense that the \(L^2\)-error of the neural network solution is uniformly controlled by the loss. When the boundary condition is an-isotropic, a boundary layer emerges in the diffusion limit and therefore brings an additional difficulty in training the neural network. To resolve this issue, we include a boundary layer corrector that carries over the sharp transition part of the solution and leaves the rest easy to be approximated. The effectiveness of the new methodology is demonstrated in extensive numerical examples.

Multi-Prior Twin Least-Square Network for Anomaly Detection of Hyperspectral Imagery

Anomaly detection of hyperspectral imagery (HSI) identifies the very few samples that do not conform to an intricate background without priors. Despite the extensive success of hyperspectral interpretation techniques based on generative adversarial networks (GANs), applying trained GAN models to hyperspectral anomaly detection remains promising but challenging. Previous generative models can accurately learn the complex background distribution of HSI and typically convert the high-dimensional data back to the latent space to extract features to detect anomalies. However, both background modeling and feature-extraction methods can be improved to become ideal in terms of the modeling power and reconstruction consistency capability. In this work, we present a multi-prior-based network (MPN) to incorporate the well-trained GANs as effective priors to a general anomaly-detection task. In particular, we introduce multi-scale covariance maps (MCMs) of precise second-order statistics to construct multi-scale priors. The MCM strategy implicitly bridges the spectral- and spatial-specific information and fully represents multi-scale, enhanced information. Thus, we reliably and adaptively estimate the HSI label to alleviate the problem of insufficient priors. Moreover, the twin least-square loss is imposed to improve the generative ability and training stability in feature and image domains, as well as to overcome the gradient vanishing problem. Last but not least, the network, enforced with a new anomaly rejection loss, establishes a pure and discriminative background estimation.

Optimization landscape in the simplest constrained random least-square problem

We analyze statistical features of the ‘optimization landscape’ in a random version of one of the simplest constrained optimization problems of the least-square type: finding the best approximation for the solution of a system of M linear equations in N unknowns: ( a k , x ) = b k , k = 1, …, M on the N-sphere x 2 = N. We treat both the N-component vectors a k and parameters b k as independent mean zero real Gaussian random variables. First, we derive the exact expressions for the mean number of stationary points of the least-square loss function in the overcomplete case M > N in the framework of the Kac-Rice approach combined with the random matrix theory for Wishart ensemble. Then we perform its asymptotic analysis as N → ∞ at a fixed α = M/N > 1 in various regimes. In particular, this analysis allows to extract the large deviation function for the density of the smallest Lagrange multiplier λ min associated with the problem, and in this way to find its most probable value. This can be further used to predict the asymptotic mean minimal value of the loss function as N → ∞. Finally, we develop an alternative approach based on the replica trick to conjecture the form of the large deviation function for the density of at N ≫ 1 and any fixed ratio α = M/N > 0. As a by-product, we find the compatibility threshold α c < 1 which is the value of α beyond which a large random linear system on the N-sphere becomes typically incompatible.

An Improved Cervical Cell Segmentation Method Based on Deep Convolutional Network

The cervical cytology smear test is an effective method for cervical cancer early screening, and segmentation accuracy is essential for computer-aided diagnosis. In this study, an improved cervical nucleus and cytoplasm segmentation method based on a deep convolutional network was proposed. This method consisted of a cellular region proposal and pixel-level segmentation network (CRP-PSN). Data were obtained from the 2014 International Symposium on Biomedical Imaging cervical cell segmentation competition open dataset. In CRP networks, online hard example mining and soft-nonmaximum suppression algorithms were incorporated to solve problems, including background noise, impurities, and other interferences in smear images. For PSN networks, a generative adversarial network-generated adversarial network algorithm and least-squares loss function were used to generate cell region segmentation, thereby improving the cytoplasm segmentation results. Finally, the improved CRP-PSN model was analyzed and compared with other typical cervical cell segmentation methods. The experimental results showed that the proposed model could effectively improve the segmentation accuracy of cytoplasm and nuclei in cervical cytology smear images by 92% and 98.6%, respectively. These findings provided strong support for the application of this method for automated interpretation of cervical cytology smear images and improved diagnostic reliability.

Binary Classification of Gaussian Mixtures: Abundance of Support Vectors, Benign Overfitting, and Regularization

Deep neural networks generalize well despite being exceedingly overparameterized and being trained without explicit regularization. This curious phenomenon has inspired extensive research activity in establishing its statistical principles: Under what conditions is it observed? How do these depend on the data and on the training algorithm? When does regularization benefit generalization? While such questions remain wide open for deep neural nets, recent works have attempted gaining insights by studying simpler, often linear, models. Our paper contributes to this growing line of work by examining binary linear classification under a generative Gaussian mixture model in which the feature vectors take the form ${{\it x}}=\pm{{\eta}}+{{\it q}}$, where for a mean vector $\eta$ and feature noise ${{\it q}} \sim \mathcal{N}(0,{{\Sigma}})$. Motivated by recent results on the implicit bias of gradient descent, we study both max-margin support vector machine (SVM) classifiers (corresponding to logistic loss) and min-norm interpolating classifiers (corresponding to least-squares loss). First, we leverage an idea introduced in [V. Muthukumar et al., arXiv:2005.08054, 2020a] to relate the SVM solution to the min-norm interpolating solution. Second, we derive novel nonasymptotic bounds on the classification error of the latter. Combining the two, we present novel sufficient conditions on the covariance spectrum and on the signal-to-noise ratio (SNR) $SNR={||{{\eta}}||_2^4}/{{\eta}}^T{{\Sigma\eta}}$ under which interpolating estimators achieve asymptotically optimal performance as overparameterization increases. Interestingly, our results extend to a noisy model with constant probability noise flips. Contrary to previously studied discriminative data models, our results emphasize the crucial role of the SNR and its interplay with the data covariance. Finally, via a combination of analytical arguments and numerical demonstrations we identify conditions under which the interpolating estimator performs better than corresponding regularized estimates.

Understanding Implicit Regularization in Over-Parameterized Single Index Model

In this article, we leverage over-parameterization to design regularization-free algorithms for the high-dimensional single index model and provide theoretical guarantees for the induced implicit regularization phenomenon. Specifically, we study both vector and matrix single index models where the link function is nonlinear and unknown, the signal parameter is either a sparse vector or a low-rank symmetric matrix, and the response variable can be heavy-tailed. To gain a better understanding of the role played by implicit regularization without excess technicality, we assume that the distribution of the covariates is known a priori. For both the vector and matrix settings, we construct an over-parameterized least-squares loss function by employing the score function transform and a robust truncation step designed specifically for heavy-tailed data. We propose to estimate the true parameter by applying regularization-free gradient descent to the loss function. When the initialization is close to the origin and the stepsize is sufficiently small, we prove that the obtained solution achieves minimax optimal statistical rates of convergence in both the vector and matrix cases. In addition, our experimental results support our theoretical findings and also demonstrate that our methods empirically outperform classical methods with explicit regularization in terms of both l 2 -statistical rate and variable selection consistency. Supplementary materials for this article are available online.

A General Loss-Based Nonnegative Matrix Factorization for Hyperspectral Unmixing

Nonnegative matrix factorization (NMF) is a widely used hyperspectral unmixing model which decomposes a known hyperspectral data matrix into two unknown matrices, i.e., endmember matrix and abundance matrix. Due to the use of least-squares loss, the NMF model is usually sensitive to noise or outliers. To improve its robustness, we introduce a general robust loss function to replace the traditional least-squares loss and propose a general loss-based NMF (GLNMF) model for hyperspectral unmixing in this letter. The general loss function is a superset of many common robust loss functions and is suitable for handling different types of noise. Experimental results on simulated and real hyperspectral data sets demonstrate that our GLNMF model is more accurate and robust than existing NMF methods.

Guided Nonlocal Patch Regularization and Efficient Filtering-Based Inversion for Multiband Fusion

In multiband fusion, an image with a high spatial and low spectral resolution is combined with an image with a low spatial but high spectral resolution to produce a single multiband image having high spatial and spectral resolutions. This comes up in remote sensing applications such as pansharpening (MS+PAN), hyperspectral sharpening (HS+PAN), and HS-MS fusion (HS+MS). Remote sensing images are textured and have repetitive structures. Motivated by nonlocal patch-based methods for image restoration, we propose a convex regularizer that (i) takes into account long-distance correlations, (ii) penalizes patch variation, which is more effective than pixel variation for capturing texture information, and (iii) uses the higher spatial resolution image as a guide image for weight computation. We come up with an efficient ADMM algorithm for optimizing the regularizer along with a standard least-squares loss function derived from the imaging model. The novelty of our algorithm is that by expressing patch variation as filtering operations and by judiciously splitting the original variables and introducing latent variables, we are able to solve the ADMM subproblems efficiently using FFT-based convolution and soft-thresholding. As far as the reconstruction quality is concerned, our method is shown to outperform state-of-the-art variational and deep learning techniques.

Application of Mean-covariance Regression Methods for Estimation of EDP|IM Distributions for Small Record Sets

ABSTRACT The performance of several regression methods is investigated to estimate the distribution of engineering demand parameters conditioned on intensity measures (EDP|IM) for small record sets. In particular, the performance of the multivariate ordinary least squares (OLS), a simultaneous mean-variance regression (MVR) done by a penalized weighted least-square loss function, and a mean-covariance/variance regression based on expectation maximization method (EM) are assessed. The efficiency of the introduced methods is compared with FEMA-P58 methodology. Performance assessment of EM and MVR methods shows that the overall increase in efficiency is about 25–45% for maximum inter-story drift ratios, and 30–50% for maximum absolute floor acceleration.

Sparse Modal Additive Model.

Sparse additive models have been successfully applied to high-dimensional data analysis due to the flexibility and interpretability of their representation. However, the existing methods are often formulated using the least-squares loss with learning the conditional mean, which is sensitive to data with the non-Gaussian noises, e.g., skewed noise, heavy-tailed noise, and outliers. To tackle this problem, we propose a new robust regression method, called as sparse modal additive model (SpMAM), by integrating the modal regression metric, the data-dependent hypothesis space, and the weighted lq,1 -norm regularizer (q ≥ 1) into the additive models. Specifically, the modal regression metric assures the model robustness to complex noises via learning the conditional mode, the data-dependent hypothesis space offers the model adaptivity via sample-based presentation, and the lq,1 -norm regularizer addresses the algorithmic interpretability via sparse variable selection. In theory, the proposed SpMAM enjoys statistical guarantees on asymptotic consistency for regression estimation and variable selection simultaneously. Experimental results on both synthetic and real-world benchmark data sets validate the effectiveness and robustness of the proposed model.

Nonconvex Matrix Factorization From Rank-One Measurements

We consider the problem of recovering low-rank matrices from random rank-one measurements, which spans numerous applications including covariance sketching, phase retrieval, quantum state tomography, and learning shallow polynomial neural networks, among others. Our approach is to directly estimate the low-rank factor by minimizing a nonconvex least-squares loss function via vanilla gradient descent, following a tailored spectral initialization. When the true rank is bounded by a constant, this algorithm is guaranteed to converge to the ground truth (up to global ambiguity) with near-optimal sample complexity and computational complexity. To the best of our knowledge, this is the first guarantee that achieves near-optimality in both metrics. In particular, the key enabler of near-optimal computational guarantees is an implicit regularization phenomenon: without explicit regularization, both spectral initialization and the gradient descent iterates automatically stay within a region incoherent with the measurement vectors. This feature allows one to employ much more aggressive step sizes compared with the ones suggested in prior literature, without the need of sample splitting.

LCPR-Net: low-count PET image reconstruction using the domain transform and cycle-consistent generative adversarial networks.

Reducing the radiation tracer dose and scanning time during positron emission tomography (PET) imaging can reduce the cost of the tracer, reduce motion artifacts, and increase the efficiency of the scanner. However, the reconstructed images to be noisy. It is very important to reconstruct high-quality images with low-count (LC) data. Therefore, we propose a deep learning method called LCPR-Net, which is used for directly reconstructing full-count (FC) PET images from corresponding LC sinogram data. Based on the framework of a generative adversarial network (GAN), we enforce a cyclic consistency constraint on the least-squares loss to establish a nonlinear end-to-end mapping process from LC sinograms to FC images. In this process, we merge a convolutional neural network (CNN) and a residual network for feature extraction and image reconstruction. In addition, the domain transform (DT) operation sends a priori information to the cycle-consistent GAN (CycleGAN) network, avoiding the need for a large amount of computational resources to learn this transformation. The main advantages of this method are as follows. First, the network can use LC sinogram data as input to directly reconstruct an FC PET image. The reconstruction speed is faster than that provided by model-based iterative reconstruction. Second, reconstruction based on the CycleGAN framework improves the quality of the reconstructed image. Compared with other state-of-the-art methods, the quantitative and qualitative evaluation results show that the proposed method is accurate and effective for FC PET image reconstruction.

Anchor Regression: Heterogeneous Data Meet Causality

AbstractWe consider the problem of predicting a response variable from a set of covariates on a data set that differs in distribution from the training data. Causal parameters are optimal in terms of predictive accuracy if in the new distribution either many variables are affected by interventions or only some variables are affected, but the perturbations are strong. If the training and test distributions differ by a shift, causal parameters might be too conservative to perform well on the above task. This motivates anchor regression, a method that makes use of exogenous variables to solve a relaxation of the ‘causal’ minimax problem by considering a modification of the least-squares loss. The procedure naturally provides an interpolation between the solutions of ordinary least squares (OLS) and two-stage least squares. We prove that the estimator satisfies predictive guarantees in terms of distributional robustness against shifts in a linear class; these guarantees are valid even if the instrumental variable assumptions are violated. If anchor regression and least squares provide the same answer (‘anchor stability’), we establish that OLS parameters are invariant under certain distributional changes. Anchor regression is shown empirically to improve replicability and protect against distributional shifts.

Finding optimal hyperparameters of feedforward neural networks for solving differential equations using a genetic algorithm

In this work, feedforward neural networks are used to solve 2D Laplace equation on rectangular domains. Optimal values of weights and biases of a network are recursively computed during the network training by minimizing a least-squares loss function using data at collocation points to approximate the true solution. The performance of the network largely depends on network architecture and model capability. In this work, an optimal set of hyperparameters is searched on various values of relative error. The genetic algorithm is used to find the optimal activation function, optimization algorithm, and weight initialization. In addition, we also searched for the optimal value of the number of hidden layers for a specific value of total parameters. Numerical results show that we can successfully obtain an optimal set of hyperparameters that is consistent across many values of relative error.

Abstract 148: A Machine Learning-based Dispatch Rule for Drone-delivered Defibrillators

Introduction: Drone-delivered defibrillators may reduce response time for out-of-hospital cardiac arrest (OHCA). However, an optimal dispatch rule is not yet known. Methods: We identified all suspected OHCAs in Peel Region, Ontario, Canada from Jan. 2014 to Dec. 2019. We trained a neural network model to predict emergency medical services (EMS) response times using OHCA location, distance from responding ambulance, day of week and time of day. Instead of least-squares loss, our model optimized a loss function that penalized weighted errors in the dispatch decision (type I/II error). Assuming drones were deployed from three bases in the region, we calculated drone response time to each suspected OHCA using real drone specifications. Our dispatch rule dispatched a drone when its calculated response time was shorter than the predicted EMS response time. Response time was calculated as the minimum of the drone and EMS response times. The performance of our dispatch rule was compared on out-of-sample OHCAs using 5-fold cross validation to the baseline cases of (1) no drones, and (2) drone dispatch to every suspected OHCA. Statistical analysis on the median response times was performed using a right-tailed sign test. Results: We identified 4774 suspected OHCAs with a median historical EMS response time of 6.0 minutes. Using our dispatch rule, median response time was significantly shorter at 3.9 minutes (P&lt;0.001). Drones were dispatched to 3803 cases (79.7%) and of those, drone response was faster than EMS in 3076 cases (80.9%). When the drone was not dispatched, it would have been slower than EMS in 856 cases (88.1%). Sending a drone to every suspected OHCA resulted in an identical median response time of 3.9 minutes (P&lt;0.001), with drones arriving before EMS in 3191 cases (66.8%). Conclusion: A machine learning-based dispatch rule can achieve similar response times as a policy that dispatches a drone to all suspected OHCAs, while dispatching drones less frequently.