Norm Penalty Research Articles

Online regularized matrix regression with streaming data

As extensions of vector data with ultrahigh dimensionality and complex structures, matrix data are fast emerging in a large variety of scientific applications. In this paper, we consider the matrix regression with streaming data and propose two-stage online regularized estimators with nuclear norm (NN) and adaptive nuclear norm (ANN) penalties, respectively. In the first stage, an equivalent form of offline matrix regression loss function using current raw data and summary statistics from historical data is established. In the second stage, gradient descent algorithm and soft thresholding methods are implemented iteratively to obtain the proposed online NN and ANN estimators. We establish the asymptotic properties of the resulting online regularized estimators and show the rank selection consistency for the online ANN estimator. The finite-sample performance of the proposed estimators is studied through simulations and an application to Beijing Air Quality data set.

Detail-aware near infrared and visible fusion with multi-order hyper-Laplacian priors

Heavy haze/noise can cause unpleasant information loss in near infrared (NIR) and visible (VI) image fusion. To generate high-quality fused images, this paper proposes a detail-aware near infrared and visible image fusion method, which establishes a detail injection variational framework based on multi-order hyper-Laplacian priors (MHLP). As the core idea, the MHLP is a l1/2 norm penalty term with first-order and second-order gradient differences, and applies sparsity-promoting and complete-comprehensive injection terms to enhance salient structures and fine-scale details. Besides, a noise map is introduced to suppress the structure inconsistency caused by heavy noise. Combining MHLP and a noise map, a novel near infrared and visible fusion framework is organized to deal with harsh environment imaging problems. The framework recovers visual visibility and preserves the significant scenery structural information and intrinsic colors. The video fusion of near infrared and visible also can be realized by the framework, which introduces the inter-frame coherence of successive frames additionally to retain the information lost in the degraded scene. Experiment results on public datasets and industrial sites demonstrate the advantages of our method from both qualitative and quantitative perspectives. The code is available at: https://github.com/BOYang-pro/MHLP.

Identifying covariate-related subnetworks for whole-brain connectome analysis.

Whole-brain connectome data characterize the connections among distributed neural populations as a set of edges in a large network, and neuroscience research aims to systematically investigate associations between brain connectome and clinical or experimental conditions as covariates. A covariate is often related to a number of edges connecting multiple brain areas in an organized structure. However, in practice, neither the covariate-related edges nor the structure is known. Therefore, the understanding of underlying neural mechanisms relies on statistical methods that are capable of simultaneously identifying covariate-related connections and recognizing their network topological structures. The task can be challenging because of false-positive noise and almost infinite possibilities of edges combining into subnetworks. To address these challenges, we propose a new statistical approach to handle multivariate edge variables as outcomes and output covariate-related subnetworks. We first study the graph properties of covariate-related subnetworks from a graph and combinatorics perspective and accordingly bridge the inference for individual connectome edges and covariate-related subnetworks. Next, we develop efficient algorithms to exact covariate-related subnetworks from the whole-brain connectome data with an $\ell_0$ norm penalty. We validate the proposed methods based on an extensive simulation study, and we benchmark our performance against existing methods. Using our proposed method, we analyze two separate resting-state functional magnetic resonance imaging data sets for schizophrenia research and obtain highly replicable disease-related subnetworks.

On the use of cross-validation for the calibration of the adaptive lasso.

Cross-validation is the standard method for hyperparameter tuning, or calibration, of machine learning algorithms. The adaptive lasso is a popular class of penalized approaches based on weighted L1 -norm penalties, with weights derived from an initial estimate of the model parameter. Although it violates the paramount principle of cross-validation, according to which no information from the hold-out test set should be used when constructing the model on the training set, a "naive" cross-validation scheme is often implemented for the calibration of the adaptive lasso. The unsuitability of this naive cross-validation scheme in this context has not been well documented in the literature. In this work, we recall why the naive scheme is theoretically unsuitable and how proper cross-validation should be implemented in this particular context. Using both synthetic and real-world examples and considering several versions of the adaptive lasso, we illustrate the flaws of the naive scheme in practice. In particular, we show that it can lead to the selection of adaptive lasso estimates that perform substantially worse than those selected via a proper scheme in terms of both support recovery and prediction error. In other words, our results show that the theoretical unsuitability of the naive scheme translates into suboptimality in practice, and call for abandoningit.

Imputed quantile tensor regression for near-sited spatial-temporal data

Modern spatial temporal data are collected from sensor networks. Missing data problems are common for this kind of data. Making robust and accurate imputation is important in many applications. There are complex correlations in both spatial and temporal dimensions. Thus, it is even a challenge to model missing spatial-temporal data. In this article, the imputation of missing values is with the help of related covariates. First, we transform the original sensor × time observational matrix to a high order tensor by adding an extra temporal dimension. Then we integrate quantile tensor regression with tensor completion. The objective function includes check loss and nuclear norm penalty. An alternating update algorithm combined with alternating direction method of multipliers (ADMM) is developed to solve the objective function. Theoretical properties of the proposed estimator are investigated. Simulation studies show our proposed method is more robust and can get more accurate imputation results. Real data analysis about Beijing's PM2.5 concentration level is conducted to verify the efficiency of the estimation procedure.

Imputed mean tensor regression for near-sited spatial temporal data

Modern spatial temporal data are often collected from sensor networks. Missing data problems are common to this kind of data. Making accurate imputation is important for many applications. In the unsupervised setting, one technique is to minimize the rank of a tensor or matrix. If we add related covariates, can we get more accurate imputation results? To address this, we transform the original sensor×time measurements to high order tensors by adding additional temporal dimensions and then integrate tensor regression with tensor completion using nuclear norm penalty. One advantage is we can simultaneously estimate parameters and impute missing values due to clear spatial consistency for near-sited spatial-temporal data. The proposed method doesn't assume missing mechanism of the response. Theoretical properties of the proposed estimator are investigated. Simulation studies and real data analysis are conducted to verify the efficiency of the estimation procedure.

Sparse Reconstruction Using Hyperbolic Tangent as Smooth l1-Norm Approximation

In the Compressed Sensing (CS) framework, the underdetermined system of linear equation (USLE) can have infinitely many possible solutions. However, we intend to find the sparsest possible solution, which is l0-norm minimization. However, finding an l0 norm solution out of infinitely many possible solutions is NP-hard problem that becomes non-convex optimization problem. It has been a practically proven fact that l0 norm penalty can be adequately estimated by l1 norm, which recasts a non-convex minimization problem to a convex problem. However, l1 norm non-differentiable and gradient-based minimization algorithms are not applicable, due to this very reason there is a need to approximate l1 norm by its smooth approximation. Iterative shrinkage algorithms provide an efficient method to numerically minimize l1-regularized least square optimization problem. These algorithms are required to induce sparsity in their solutions to meet the CS recovery requirement. In this research article, we have developed a novel recovery method that uses hyperbolic tangent function to recover undersampled signal/images in CS framework. In our work, l1 norm and soft thresholding are both approximated with the hyperbolic tangent functions. We have also proposed the criteria to tune optimization parameters to get optimal results. The error bounds for the proposed l1 norm approximation are evaluated. To evaluate performance of our proposed method, we have utilized a dataset comprised of 1-D sparse signal, compressively sampled MR image and cardiac cine MRI. The MRI is an important imaging modality for assessing cardiac vascular function. It provides the ejection fraction and cardiac output of the heart. However, this advantage comes at the cost of a slow acquisition process. Hence, it is essential to speed up the acquisition process to take the full benefits of cardiac cine MRI. Numerical results based on performance metrics, such as Structural Similarity (SSIM), Peak Signal to Noise Ratio (PSNR) and Root Mean Square Error (RMSE) show that the proposed tangent hyperbolic based CS recovery offers a much better performance as compared to the traditional Iterative Soft Thresholding (IST) recovery methods.

High‐dimensional regression coefficient estimation by nuclear norm plus l1 norm penalization

We propose a new estimator of the regression coefficients for a high‐dimensional linear regression model, which is derived by replacing the sample predictor covariance matrix in the ordinary least square (OLS) estimator with a different predictor covariance matrix estimate obtained by a nuclear norm plus norm penalization. We call the estimator ALgebraic Covariance Estimator‐regression (ALCE‐reg). We make a direct theoretical comparison of the expected mean square error of ALCE‐reg with OLS and RIDGE. We show in a simulation study that ALCE‐reg is particularly effective when both the dimension and the sample size are large, due to its ability to find a good compromise between the large bias of shrinkage estimators (like RIDGE and least absolute shrinkage and selection operator [LASSO]) and the large variance of estimators conditioned by the sample predictor covariance matrix (like OLS and principal orthogonal complement thresholding [POET]).

Service Function Chaining in Industrial Internet of Things With Edge Intelligence: A Natural Actor-Critic Approach

Owing to network function virtualization (NFV), each industrial application is constructed as a service function chain (SFC), concatenating the ordered service functions, to offer applications more flexibly in industrial Internet of Things (IIoT). When it comes to the emerging edge intelligence, the integration of NFV with edge in IIoT would enable more close-proximity services, yet also posing new challenges owing to more complicated environment. Although some efforts have been made to service function chaining in IIoT, the radio resource dynamics are not fully perceived. In this article, we investigate the radio-aware SFC deployment in the edge-enabled IIoT. First, a radio-aware deployment formulation is exhibited, steering the flow traversing both wireless and wired links. Next, Markov decision process is exhibited to track dynamics in both IIoT and radio resources. Afterwards, the natural gradient-based actor-critic SFC paradigm is introduced to adapt to network variation, by incorporating the curvature of parameter space into gradient information. To resolve the high-dimensionality in action space, we then recur to the norm penalty approach, reducing the space size by two orders of magnitude. Finally, numerical experiments are executed to uncover superiority of presented method, disclosing that the latency performance benefits from both the SFC routing between IIoT servers and elaborated wireless resource orchestration.

Riemannian Low-Rank Model Compression for Federated Learning With Over-the-Air Aggregation

Low-rank model compression is a widely used technique for reducing the computational load when training machine learning models. However, existing methods often rely on relaxing the low-rank constraint of the model weights using a regularized nuclear norm penalty, which requires an appropriate hyperparameter that can be difficult to determine in practice. Furthermore, existing compression techniques are not directly applicable to efficient over-the-air (OTA) aggregation in federated learning (FL) systems for distributed Internet-of-Things (IoT) scenarios. In this paper, we propose a novel manifold optimization formulation for low-rank model compression in FL that does not relax the low-rank constraint. Our optimization is conducted directly over the low-rank manifold, guaranteeing that the model is exactly low-rank. We also introduce a consensus penalty in the optimization formulation to support OTA aggregation. Based on our optimization formulation, we propose an alternating Riemannian optimization algorithm with a precoder that enables efficient OTA aggregation of low-rank local models without sacrificing training performance. Additionally, we provide convergence analysis in terms of key system parameters and conduct extensive experiments with real-world datasets to demonstrate the effectiveness of our proposed Riemannian low-rank model compression scheme compared to various state-of-the-art baselines.

Convex regularization in statistical inverse learning problems

We consider a statistical inverse learning problem, where the task is to estimate a function $ f $ based on noisy point evaluations of $ Af $, where $ A $ is a linear operator. The function $ Af $ is evaluated at i.i.d. random design points $ u_n $, $ n = 1, ..., N $ generated by an unknown general probability distribution. We consider Tikhonov regularization with general convex and $ p $-homogeneous penalty functionals and derive concentration rates of the regularized solution to the ground truth measured in the symmetric Bregman distance induced by the penalty functional. We derive concrete rates for Besov norm penalties and numerically demonstrate the correspondence with the observed rates in the context of X-ray tomography.

Joint antenna selection and waveform design for coexistence of MIMO radar and communications

As the problem of spectral congestion is becoming severe, the coexistence between two primary spectrum users, radar and communications, has spurred extensive research interest. To reduce the mutual interferences between the two functions, MIMO radar waveform design needs to consider the compatibility in both spectral and spatial domains, where the former is achieved by null forming in the frequency domain and the latter is achieved by shaped beampattern synthesis. Additionally, high power efficiency and low system overhead are two desirable characteristics for MIMO radar system design. To this end, we first introduce a new realistic waveform constraint, peak-to-valley-power-ratio (PVPR) constraint per antenna to improve the power efficiency. Then, combined with PVPR constraint, we propose a switchable individual antenna power control scheme to jointly optimize waveforms and antenna locations. We adopt a max–min beampattern matching criterion and impose the ell _{2,1}-norm penalty on the waveform matrix to promote the sparsity of the array. To solve the resultant non-smooth and non-convex problem, we develop a modified alternating directions method of multipliers, where a surrogate subproblem over primal variables is solved instead of the original problem, and its local convergence is analyzed. Finally, numerical experiments demonstrate the effectiveness and superiority of the proposed method over counterparts, especially obtaining the lowest sidelobe level and deeper spectral nulls using much fewer antennas.

High-dimensional estimation of quadratic variation based on penalized realized variance

In this paper, we develop a penalized realized variance (PRV) estimator of the quadratic variation (QV) of a high-dimensional continuous Itô semimartingale. We adapt the principle idea of regularization from linear regression to covariance estimation in a continuous-time high-frequency setting. We show that under a nuclear norm penalization, the PRV is computed by soft-thresholding the eigenvalues of realized variance (RV). It therefore encourages sparsity of singular values or, equivalently, low rank of the solution. We prove our estimator is minimax optimal up to a logarithmic factor. We derive a concentration inequality, which reveals that the rank of PRV is—with a high probability—the number of non-negligible eigenvalues of the QV. Moreover, we also provide the associated non-asymptotic analysis for the spot variance. We suggest an intuitive data-driven subsampling procedure to select the shrinkage parameter. Our theory is supplemented by a simulation study and an empirical application. The PRV detects about three–five factors in the equity market, with a notable rank decrease during times of distress in financial markets. This is consistent with most standard asset pricing models, where a limited amount of systematic factors driving the cross-section of stock returns are perturbed by idiosyncratic errors, rendering the QV—and also RV—of full rank.

Elastic Net Nonparallel Hyperplane Support Vector Machine and Its Geometrical Rationality.

Twin support vector machine (TWSVM), which constructs two nonparallel classifying hyperplanes, is widely applied to various fields. However, TWSVM solves two quadratic programming problems (QPPs) separately such that the final classifiers lack consistency and enough prediction accuracy. Moreover, by reason of only considering the 1-norm penalty for slack variables, TWSVM is not well defined in the geometrical view. In this article, we propose a novel elastic net nonparallel hyperplane support vector machine (ENNHSVM), which adopts elastic net penalty for slack variables and constructs two nonparallel separating hyperplanes simultaneously. We further discuss the properties of ENNHSVM theoretically and derive the violation tolerance upper bound to better demonstrate the relative violations of training samples in the same class. In particular, we design a safe screening rule for ENNHSVM to speed up the calculations. We finally compare the performance of ENNHSVM on both synthetic datasets and benchmark datasets with the Lagrangian SVM, the twin parametric-margin SVM, the elastic net SVM, the TWSVM, and the nonparallel hyperplane SVM.

High-Dimensional Precision Matrix Estimation through GSOS with Application in the Foreign Exchange Market

This article studies the estimation of the precision matrix of a high-dimensional Gaussian network. We investigate the graphical selector operator with shrinkage, GSOS for short, to maximize a penalized likelihood function where the elastic net-type penalty is considered as a combination of a norm-one penalty and a targeted Frobenius norm penalty. Numerical illustrations demonstrate that our proposed methodology is a competitive candidate for high-dimensional precision matrix estimation compared to some existing alternatives. We demonstrate the relevance and efficiency of GSOS using a foreign exchange markets dataset and estimate dependency networks for 32 different currencies from 2018 to 2021.

Consistent connectome landscape mining for cross-site brain disease identification using functional MRI.

Many human brain disorders are associated with characteristic alterations in functional connectivity of the brain. A lot of efforts have been devoted to mining disease-related biomarkers for identifying patients with brain disorders from normal controls. However, previous studies show largely inconsistent findings due to variability across numerous study-specific factors such as heterogeneity across different preprocessing pipelines or the use of multi-site data. Also, existing methods usually employ human-engineered features (e.g., graph-theoretical measures) that may be less discriminate for disease identification. To this end, we propose a novel Connectome Landscape Modeling (CLM) method that can mine cross-site consistent connectome landscape and extract data-driven representation of functional connectivity networks for brain disorder identification. Specifically, with functional connectivity networks as input, the proposed CLM model aims to learn a weight matrix for joint cross-site consistent connectome landscape learning, network feature extraction, and disease identification. We impose the row-column overlap norm penalty on the network-based predictor to capture consistent connectome landscape across multiple sites. To capture site-specific patterns, we introduce an ℓ1-norm penalty in CLM. We develop an efficient algorithm based on the Alternating Direction Method of Multipliers (ADMM) to solve the proposed objective function. Experimental results on three real-world fMRI datasets demonstrate the potential use of our CLM in cross-site brain disorder analysis.

A Comparative Study of Engraved-Digit Data Augmentation by Generative Adversarial Networks

In cases where an efficient information retrieval (IR) system retrieves information from images with engraved digits, as found on medicines, creams, ointments, and gels in squeeze tubes, the system needs to be trained on a large dataset. One of the system applications is to automatically retrieve the expiry date to ascertain the efficacy of the medicine. For expiry dates expressed in engraved digits, it is difficult to collect the digit images. In our study, we evaluated the augmentation performance for a limited, engraved-digit dataset using various generative adversarial networks (GANs). Our study contributes to the choice of an effective GAN for engraved-digit image data augmentation. We conclude that Wasserstein GAN with a gradient norm penalty (WGAN-GP) is a suitable data augmentation technique to address the challenge of producing a large, realistic, but synthetic dataset. Our results show that the stability of WGAN-GP aids in the production of high-quality data with an average Fréchet inception distance (FID) value of 1.5298 across images of 10 digits (0–9) that are nearly indistinguishable from our original dataset.

A Frobenius Norm Regularization Method for Convolutional Kernel Tensors in Neural Networks.

The convolutional neural network is a very important model of deep learning. It can help avoid the exploding/vanishing gradient problem and improve the generalizability of a neural network if the singular values of the Jacobian of a layer are bounded around 1 in the training process. We propose a new Frobenius norm penalty function for a convolutional kernel tensor to let the singular values of the corresponding transformation matrix be bounded around 1. We show how to carry out the gradient-type methods. This provides a potentially useful regularization method for the weights of convolutional layers.

Linearly Constrained Nonsmooth Optimization for Training Autoencoders

A regularized minimization model with $l_1$-norm penalty (RP) is introduced for training the autoencoders that belong to a class of two-layer neural networks. We show that the RP can act as an exact penalty model which shares the same global minimizers, local minimizers, and d(irectional)-stationary points with the original regularized model under mild conditions. We construct a bounded box region that contains at least one global minimizer of the RP, and propose a linearly constrained regularized minimization model with $l_1$-norm penalty (LRP) for training autoencoders. A smoothing proximal gradient algorithm is designed to solve the LRP. Convergence of the algorithm to a generalized d-stationary point of the RP and LRP is delivered. Comprehensive numerical experiments convincingly illustrate the efficiency as well as the robustness of the proposed algorithm.

Variable source depth beneath the Indian Ocean geoid low area: Insights from L1 and L2 norm-based scaling power spectrum inversion

Indian Ocean geoid low, situated at the south of the Indian peninsula, is an intriguing feature. Hitherto, several interpretations have been put forth to explain the genesis and causative depths of the anomaly. In this work, we investigate the scaling spectral inversion approach utilizing gravity data to estimate the causative depth and nature of the source from this region. In practice, crustal or sub-surface heterogeneities follow scaling or fractal characteristics, and comparative studies across various regions suggest that the scaling power spectrum method provides better insights into the sub-surface source distribution. An optimum fitting and comparison between the modeled and observed power spectrum have been attained by utilizing two norm penalties e.g. L1 and L2 norms for various combinations of depth and scaling exponents. Equally important, the L1 norm misfit function yields better results when compared to the L2 norm function. We propose that estimated source depths (̴80 km and 120 km, respectively) are perfect examples of a half-space model with a source following scaling or fractal characteristics (scaling exponent value of ̴1.78) along the upper mantle boundary. However, the scaling spectrum windowing analysis of the studied region reveals a variable depth to long wavelength sources as 1014 km, 431 km, and 94 km, which conform well with the recently published depth range. Further, the weight of low and high wavenumbers indicates that the causative source depths might be due to the above-stated depth range rather than a single model fit. The study has demonstrated a good case for wider use of L1 norm-based power spectrum inversion for source depth estimation from potential field data. We further suggest that the causative source for the Indian Ocean geoid low may extend deeper than upper mantle discontinuities.